Package com.tnsai.llm.routing

Class LatencyBasedRouter

java.lang.Object
com.tnsai.llm.routing.LatencyBasedRouter
All Implemented Interfaces:
com.tnsai.llm.LLMClient, LLMRouter
public class LatencyBasedRouter extends Object implements LLMRouter
Routes LLM requests to the fastest available provider.

LatencyBasedRouter optimizes for response time by selecting the provider with the lowest latency. It learns from actual response times and adapts routing decisions based on measured performance.

Latency Optimization Strategy

  1. Initially uses estimated latency from capabilities
  2. Tracks actual latency per provider
  3. Routes to provider with lowest moving average
  4. Falls back on failure, updates latency data

Usage Example


 // Create router optimizing for speed
 LatencyBasedRouter router = LatencyBasedRouter.builder()
     .addClient(new GroqClient("llama-3.3-70b"))   // ~100ms TTFT
     .addClient(new OpenAIClient("gpt-4o-mini"))   // ~300ms TTFT
     .addClient(new AnthropicClient("claude-sonnet-4")) // ~600ms TTFT
     .build();

 // Routes to Groq (fastest) automatically
 ChatResponse response = router.chat("Quick question");

 // If Groq slows down, router adapts and switches

Adaptive Routing


 // Router learns from actual performance
 for (int i = 0; i < 100; i++) {
     router.chat("Test query " + i);
 }

 // After warmup, routing is based on actual measured latency
 LatencyBasedRouter.LatencyStats stats = router.getLatencyStats();
 System.out.println("Avg latency per provider: " + stats.avgLatencyPerProvider());
 System.out.println("Fastest provider: " + stats.fastestProvider());

With Latency Threshold


 // Only use providers with acceptable latency
 LatencyBasedRouter router = LatencyBasedRouter.builder()
     .addClient(new GroqClient("llama-3.3-70b"))
     .addClient(new OpenAIClient("gpt-4o"))
     .maxLatencyMs(500)  // Exclude providers slower than 500ms
     .build();
See Also:

  • Method Details

    • of

      public static LatencyBasedRouter of(com.tnsai.llm.LLMClient... clients)
      Creates a LatencyBasedRouter with the given clients.

    • builder

      public static LatencyBasedRouter.Builder builder()
      Returns a builder for configuring the router.

    • getClients

      public List<com.tnsai.llm.LLMClient> getClients()
      Description copied from interface: LLMRouter
      Returns all registered LLM clients.
      Specified by:
      getClients in interface LLMRouter

    • getPrimaryClient

      public com.tnsai.llm.LLMClient getPrimaryClient()
      Description copied from interface: LLMRouter
      Returns the currently selected/primary client.
      Specified by:
      getPrimaryClient in interface LLMRouter

    • selectByCapability

      public Optional<com.tnsai.llm.LLMClient> selectByCapability(Predicate<com.tnsai.llm.LLMCapabilities> filter)
      Description copied from interface: LLMRouter
      Selects a client that matches the given capability predicate.
      Specified by:
      selectByCapability in interface LLMRouter
      Parameters:
      filter - Predicate to filter by capabilities
      Returns:
      The first matching client, or empty if none match

    • getModel

      public String getModel()
      Specified by:
      getModel in interface com.tnsai.llm.LLMClient

    • getTemperature

      public float getTemperature()
      Specified by:
      getTemperature in interface com.tnsai.llm.LLMClient

    • getMaxTokens

      public Optional<Integer> getMaxTokens()
      Specified by:
      getMaxTokens in interface com.tnsai.llm.LLMClient

    • getCapabilities

      public com.tnsai.llm.LLMCapabilities getCapabilities()
      Specified by:
      getCapabilities in interface com.tnsai.llm.LLMClient

    • chat

      public com.tnsai.llm.ChatResponse chat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Description copied from interface: LLMRouter
      Routes a chat request to the appropriate client.

      The routing decision depends on the implementation:

      • FallbackRouter: Uses primary, falls back on error
      • CostBasedRouter: Selects cheapest viable option
      • LatencyBasedRouter: Selects fastest option
      Specified by:
      chat in interface com.tnsai.llm.LLMClient
      Specified by:
      chat in interface LLMRouter
      Parameters:
      message - The user message
      systemPrompt - Optional system prompt
      history - Optional conversation history
      tools - Optional tools
      Returns:
      The chat response from the selected client

    • streamChat

      public Stream<String> streamChat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      streamChat in interface com.tnsai.llm.LLMClient

    • streamChatWithSpec

      public Stream<com.tnsai.llm.ChatChunk> streamChatWithSpec(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      streamChatWithSpec in interface com.tnsai.llm.LLMClient

    • getLatencyStats

      public LatencyBasedRouter.LatencyStats getLatencyStats()
      Returns latency statistics.

    • getStats

      public LLMRouter.RoutingStats getStats()
      Description copied from interface: LLMRouter
      Returns routing statistics (calls per provider, failures, latencies).
      Specified by:
      getStats in interface LLMRouter

    • resetStats

      public void resetStats()
      Description copied from interface: LLMRouter
      Resets routing statistics.
      Specified by:
      resetStats in interface LLMRouter