Package com.tnsai.llm.routing

Class FallbackRouter

java.lang.Object
com.tnsai.llm.routing.FallbackRouter
All Implemented Interfaces:
com.tnsai.llm.LLMClient, LLMRouter
public class FallbackRouter extends Object implements LLMRouter
LLM router with automatic fallback on failure.

FallbackRouter provides resilience by automatically trying alternative LLM providers when the primary fails. This is essential for production systems where reliability is critical.

Fallback Behavior

  1. Try primary client (first in list)
  2. On failure, log error and try next client
  3. Continue until success or all clients exhausted
  4. Throw last exception if all fail

Usage Example


 // Create router with ordered fallback chain
 LLMRouter router = FallbackRouter.of(
     new OpenAIClient("gpt-4o"),           // Primary
     new AnthropicClient("claude-sonnet-4"), // First fallback
     new GroqClient("llama-3.3-70b")       // Last resort
 );

 // Use like a normal LLMClient
 ChatResponse response = router.chat("Hello, world!");

 // Check which provider was actually used
 System.out.println("Used: " + response.getModel());

With Capability Filter


 // Only fallback to vision-capable models
 FallbackRouter router = FallbackRouter.builder()
     .addClient(new OpenAIClient("gpt-4o"))
     .addClient(new GeminiClient("gemini-2.5-flash"))
     .requireCapability(cap -> cap.supportsVision())
     .build();

Monitoring


 RoutingStats stats = router.getStats();
 System.out.println("Total requests: " + stats.totalRequests());
 System.out.println("Failures per provider: " + stats.failuresPerProvider());
See Also:

  • Method Details

    • of

      public static FallbackRouter of(com.tnsai.llm.LLMClient... clients)
      Creates a FallbackRouter with the given clients. The first client is the primary; others are fallbacks in order.

    • of

      public static FallbackRouter of(List<com.tnsai.llm.LLMClient> clients)
      Creates a FallbackRouter with the given clients.

    • builder

      public static FallbackRouter.Builder builder()
      Returns a builder for configuring the router.

    • getClients

      public List<com.tnsai.llm.LLMClient> getClients()
      Description copied from interface: LLMRouter
      Returns all registered LLM clients.
      Specified by:
      getClients in interface LLMRouter

    • getPrimaryClient

      public com.tnsai.llm.LLMClient getPrimaryClient()
      Description copied from interface: LLMRouter
      Returns the currently selected/primary client.
      Specified by:
      getPrimaryClient in interface LLMRouter

    • selectByCapability

      public Optional<com.tnsai.llm.LLMClient> selectByCapability(Predicate<com.tnsai.llm.LLMCapabilities> filter)
      Description copied from interface: LLMRouter
      Selects a client that matches the given capability predicate.
      Specified by:
      selectByCapability in interface LLMRouter
      Parameters:
      filter - Predicate to filter by capabilities
      Returns:
      The first matching client, or empty if none match

    • getModel

      public String getModel()
      Specified by:
      getModel in interface com.tnsai.llm.LLMClient

    • getTemperature

      public float getTemperature()
      Specified by:
      getTemperature in interface com.tnsai.llm.LLMClient

    • getMaxTokens

      public Optional<Integer> getMaxTokens()
      Specified by:
      getMaxTokens in interface com.tnsai.llm.LLMClient

    • getCapabilities

      public com.tnsai.llm.LLMCapabilities getCapabilities()
      Specified by:
      getCapabilities in interface com.tnsai.llm.LLMClient

    • chat

      public com.tnsai.llm.ChatResponse chat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Description copied from interface: LLMRouter
      Routes a chat request to the appropriate client.

      The routing decision depends on the implementation:

      • FallbackRouter: Uses primary, falls back on error
      • CostBasedRouter: Selects cheapest viable option
      • LatencyBasedRouter: Selects fastest option
      Specified by:
      chat in interface com.tnsai.llm.LLMClient
      Specified by:
      chat in interface LLMRouter
      Parameters:
      message - The user message
      systemPrompt - Optional system prompt
      history - Optional conversation history
      tools - Optional tools
      Returns:
      The chat response from the selected client

    • streamChat

      public Stream<String> streamChat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      streamChat in interface com.tnsai.llm.LLMClient

    • streamChatWithSpec

      public Stream<com.tnsai.llm.ChatChunk> streamChatWithSpec(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      streamChatWithSpec in interface com.tnsai.llm.LLMClient

    • getStats

      public LLMRouter.RoutingStats getStats()
      Description copied from interface: LLMRouter
      Returns routing statistics (calls per provider, failures, latencies).
      Specified by:
      getStats in interface LLMRouter

    • resetStats

      public void resetStats()
      Description copied from interface: LLMRouter
      Resets routing statistics.
      Specified by:
      resetStats in interface LLMRouter