Package com.tnsai.llm.routing

package com.tnsai.llm.routing
LLM routing and load balancing for multi-provider setups.

The routing package provides intelligent request routing across multiple LLM providers, enabling:

  • Resilience: Automatic fallback when providers fail
  • Cost optimization: Route to cheapest viable provider
  • Latency optimization: Route to fastest provider
  • Load distribution: Spread requests across providers
  • Capability-based selection: Auto-select by features needed

Available Routers

RouterStrategyUse Case
FallbackRouter Try primary, fallback on error High availability
CostBasedRouter Cheapest first Budget optimization
LatencyBasedRouter Fastest first (adaptive) Performance-critical apps
CapabilityRouter By required features Multi-modal applications
RoundRobinRouter Even distribution Rate limit avoidance

Quick Start


 // Simple fallback chain
 LLMRouter router = FallbackRouter.of(
     new OpenAIClient("gpt-4o"),
     new AnthropicClient("claude-sonnet-4"),
     new GroqClient("llama-3.3-70b")
 );

 // Use like a normal LLMClient
 ChatResponse response = router.chat("Hello!");

 // Cost-optimized routing
 LLMRouter costRouter = CostBasedRouter.builder()
     .addClient(new OpenAIClient("gpt-4o-mini"))
     .addClient(new OpenAIClient("gpt-4o"))
     .build();

 // Latency-optimized routing
 LLMRouter fastRouter = LatencyBasedRouter.builder()
     .addClient(new GroqClient("llama-3.3-70b"))
     .addClient(new OpenAIClient("gpt-4o"))
     .maxLatencyMs(500)
     .build();

Monitoring

All routers provide statistics: 


 RoutingStats stats = router.getStats();
 System.out.println("Total requests: " + stats.totalRequests());
 System.out.println("Failures per provider: " + stats.failuresPerProvider());
 System.out.println("Avg latency per provider: " + stats.avgLatencyPerProvider());
See Also:
  • LLMClient
  • LLMCapabilities