Package com.tnsai.llm.routing

Class RoundRobinRouter

java.lang.Object
com.tnsai.llm.routing.RoundRobinRouter
All Implemented Interfaces:
com.tnsai.llm.LLMClient, LLMRouter
public class RoundRobinRouter extends Object implements LLMRouter
Routes LLM requests using round-robin load distribution.

RoundRobinRouter distributes requests evenly across all providers, which is useful for:

  • Staying under per-provider rate limits
  • Distributing load across multiple accounts/endpoints
  • A/B testing different models

Usage Example


 // Distribute load across multiple providers
 RoundRobinRouter router = RoundRobinRouter.of(
     new OpenAIClient("gpt-4o"),
     new AnthropicClient("claude-sonnet-4"),
     new GroqClient("llama-3.3-70b")
 );

 // Requests are distributed 1 → 2 → 3 → 1 → 2 → ...
 router.chat("Query 1"); // GPT-4o
 router.chat("Query 2"); // Claude
 router.chat("Query 3"); // Groq
 router.chat("Query 4"); // GPT-4o again

Rate Limit Avoidance


 // Use multiple OpenAI accounts to avoid rate limits
 RoundRobinRouter router = RoundRobinRouter.of(
     new OpenAIClient("gpt-4o", apiKey1),
     new OpenAIClient("gpt-4o", apiKey2),
     new OpenAIClient("gpt-4o", apiKey3)
 );

 // Requests are distributed across accounts
See Also:

  • Method Details

    • of

      public static RoundRobinRouter of(com.tnsai.llm.LLMClient... clients)
      Creates a RoundRobinRouter with the given clients.

    • of

      public static RoundRobinRouter of(List<com.tnsai.llm.LLMClient> clients)
      Creates a RoundRobinRouter with the given clients.

    • builder

      public static RoundRobinRouter.Builder builder()
      Returns a builder for configuring the router.

    • getClients

      public List<com.tnsai.llm.LLMClient> getClients()
      Description copied from interface: LLMRouter
      Returns all registered LLM clients.
      Specified by:
      getClients in interface LLMRouter

    • getPrimaryClient

      public com.tnsai.llm.LLMClient getPrimaryClient()
      Description copied from interface: LLMRouter
      Returns the currently selected/primary client.
      Specified by:
      getPrimaryClient in interface LLMRouter

    • selectByCapability

      public Optional<com.tnsai.llm.LLMClient> selectByCapability(Predicate<com.tnsai.llm.LLMCapabilities> filter)
      Description copied from interface: LLMRouter
      Selects a client that matches the given capability predicate.
      Specified by:
      selectByCapability in interface LLMRouter
      Parameters:
      filter - Predicate to filter by capabilities
      Returns:
      The first matching client, or empty if none match

    • getModel

      public String getModel()
      Specified by:
      getModel in interface com.tnsai.llm.LLMClient

    • getTemperature

      public float getTemperature()
      Specified by:
      getTemperature in interface com.tnsai.llm.LLMClient

    • getMaxTokens

      public Optional<Integer> getMaxTokens()
      Specified by:
      getMaxTokens in interface com.tnsai.llm.LLMClient

    • getCapabilities

      public com.tnsai.llm.LLMCapabilities getCapabilities()
      Specified by:
      getCapabilities in interface com.tnsai.llm.LLMClient

    • chat

      public com.tnsai.llm.ChatResponse chat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Description copied from interface: LLMRouter
      Routes a chat request to the appropriate client.

      The routing decision depends on the implementation:

      • FallbackRouter: Uses primary, falls back on error
      • CostBasedRouter: Selects cheapest viable option
      • LatencyBasedRouter: Selects fastest option
      Specified by:
      chat in interface com.tnsai.llm.LLMClient
      Specified by:
      chat in interface LLMRouter
      Parameters:
      message - The user message
      systemPrompt - Optional system prompt
      history - Optional conversation history
      tools - Optional tools
      Returns:
      The chat response from the selected client

    • streamChat

      public Stream<String> streamChat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      streamChat in interface com.tnsai.llm.LLMClient

    • streamChatWithSpec

      public Stream<com.tnsai.llm.ChatChunk> streamChatWithSpec(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      streamChatWithSpec in interface com.tnsai.llm.LLMClient

    • getStats

      public LLMRouter.RoutingStats getStats()
      Description copied from interface: LLMRouter
      Returns routing statistics (calls per provider, failures, latencies).
      Specified by:
      getStats in interface LLMRouter

    • resetStats

      public void resetStats()
      Description copied from interface: LLMRouter
      Resets routing statistics.
      Specified by:
      resetStats in interface LLMRouter

    • getCurrentIndex

      public int getCurrentIndex()
      Returns the current rotation index.