Package com.tnsai.llm

Class FallbackLLMClient

java.lang.Object

com.tnsai.llm.FallbackLLMClient

All Implemented Interfaces:

LLMClient

public class FallbackLLMClient
extends Object
implements LLMClient

LLM client with automatic fallback support.

Wraps a primary LLM client with fallback options. If the primary fails, automatically retries with fallback clients.

Inspired by Mastra

Based on Mastra's auto model fallback pattern.

Usage

 LLMClient client = FallbackLLMClient.builder()
     .primary(new OpenAIClient("gpt-4o"))
     .fallback(new AnthropicClient("claude-3-sonnet"))
     .fallback(new OllamaClient("llama3.2"))
     .retryOnPrimary(2)
     .build();

 // If gpt-4o fails after 2 retries, automatically tries claude, then llama
 ChatResponse response = client.chat("Hello");
 

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	FallbackLLMClient.Builder	Builder for FallbackLLMClient.

Method Summary

Modifier and Type	Method	Description
static FallbackLLMClient.Builder	builder()	Creates a new builder for FallbackLLMClient.
ChatResponse	chat(ChatRequest request)	Sends a chat request using the unified ChatRequest model.
ChatResponse	chat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)	Sends a chat message and returns the response.
List<LLMClient>	getFallbacks()	Gets the fallback clients.
Optional<Integer>	getMaxTokens()	Gets the maximum tokens setting.
String	getModel()	Gets the model name being used.
LLMClient	getPrimary()	Gets the primary client.
float	getTemperature()	Gets the temperature setting.
Stream<String>	streamChat(ChatRequest request)	Streams a chat response using the unified ChatRequest model.
Stream<String>	streamChat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)	Sends a chat message and streams the response token by token.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.tnsai.llm.LLMClient

chat, chat, chat, chat, getCapabilities, getProviderName, getTopP, listModels, streamChat, streamChatWithHandler, streamChatWithHandler, streamChatWithSpec, streamChatWithSpec

Method Details

builder

public static FallbackLLMClient.Builder builder()

Creates a new builder for FallbackLLMClient.

getModel

public String getModel()

Description copied from interface: LLMClient

Gets the model name being used.

Specified by:

getModel in interface LLMClient

getTemperature

public float getTemperature()

Description copied from interface: LLMClient

Gets the temperature setting.

Specified by:

getTemperature in interface LLMClient

getMaxTokens

public Optional<Integer> getMaxTokens()

Description copied from interface: LLMClient

Gets the maximum tokens setting.

Specified by:

getMaxTokens in interface LLMClient

chat

public ChatResponse chat(String message,
 Optional<String> systemPrompt,
 Optional<List<Map<String,Object>>> history,
 Optional<List<Map<String,Object>>> tools)

Description copied from interface: LLMClient

Sends a chat message and returns the response.

Specified by:

chat in interface LLMClient

Parameters:

message - The user message

systemPrompt - Optional system prompt

history - Optional conversation history

tools - Optional list of available tools for function calling

Returns:

The LLM response as ChatResponse (contains content and optional tool calls)

chat

public ChatResponse chat(ChatRequest request)

Description copied from interface: LLMClient

Sends a chat request using the unified ChatRequest model.

Dispatches to the appropriate chat method based on request properties:

• Multimodal → LLMClient.chat(List, Optional, Optional, Optional)
• Temperature override → LLMClient.chat(String, Optional, Optional, Optional, float)
• Otherwise → LLMClient.chat(String, Optional, Optional, Optional)

Specified by:

chat in interface LLMClient

Parameters:

request - the chat request

Returns:

the LLM response

streamChat

public Stream<String> streamChat(ChatRequest request)

Description copied from interface: LLMClient

Streams a chat response using the unified ChatRequest model.

Specified by:

streamChat in interface LLMClient

Parameters:

request - the chat request

Returns:

stream of response tokens

streamChat

public Stream<String> streamChat(String message,
 Optional<String> systemPrompt,
 Optional<List<Map<String,Object>>> history,
 Optional<List<Map<String,Object>>> tools)

Description copied from interface: LLMClient

Sends a chat message and streams the response token by token.

Important: The returned Stream holds an HTTP connection open. Callers must close the stream when done, ideally via try-with-resources:

 try (Stream<String> tokens = client.streamChat(msg, sys, hist, tools)) {
     tokens.forEach(System.out::print);
 }
 

Failing to close the stream will leak HTTP connections.

Specified by:

streamChat in interface LLMClient

Parameters:

message - The user message

systemPrompt - Optional system prompt

history - Optional conversation history

tools - Optional list of available tools for function calling

Returns:

Stream of response tokens — caller must close

getPrimary

public LLMClient getPrimary()

Gets the primary client.

getFallbacks

public List<LLMClient> getFallbacks()

Gets the fallback clients.