Package com.tnsai.llm.routing

Class CostBasedRouter

java.lang.Object
com.tnsai.llm.routing.CostBasedRouter
All Implemented Interfaces:
com.tnsai.llm.LLMClient, LLMRouter
public class CostBasedRouter extends Object implements LLMRouter
Routes LLM requests to the cheapest viable provider.

CostBasedRouter optimizes for cost by selecting the cheapest model that meets the capability requirements. This can reduce LLM costs by up to 85% for simple queries by routing them to cheaper models.

Cost Optimization Strategy

  1. Filter clients by capability requirements
  2. Sort by input cost (cheapest first)
  3. Try cheapest client, fallback on failure

Usage Example


 // Create router with multiple price tiers
 CostBasedRouter router = CostBasedRouter.builder()
     .addClient(new OpenAIClient("gpt-4o-mini"))    // $0.15/1M input
     .addClient(new GroqClient("llama-3.3-70b"))   // $0.59/1M input
     .addClient(new OpenAIClient("gpt-4o"))        // $2.50/1M input
     .addClient(new AnthropicClient("claude-sonnet-4")) // $3.00/1M input
     .build();

 // Simple queries go to cheapest model (gpt-4o-mini)
 router.chat("What is 2+2?");

 // Complex queries that fail on cheap models fallback to expensive ones
 router.chat("Analyze this complex code...");

With Capability Filter


 // Find cheapest vision model
 CostBasedRouter router = CostBasedRouter.builder()
     .addClient(new OpenAIClient("gpt-4o-mini"))  // Has vision, cheap
     .addClient(new OpenAIClient("gpt-4o"))      // Has vision
     .addClient(new GroqClient("llama-3.3-70b")) // No vision
     .requireCapability(cap -> cap.supportsVision())
     .build();

 // Automatically filters to vision-capable models, selects cheapest

Cost Tracking


 // After running queries
 CostBasedRouter.CostStats stats = router.getCostStats();
 System.out.println("Estimated total cost: $" + stats.totalEstimatedCost());
 System.out.println("Cost per provider: " + stats.costPerProvider());
See Also:

  • Method Details

    • of

      public static CostBasedRouter of(com.tnsai.llm.LLMClient... clients)
      Creates a CostBasedRouter with the given clients.

    • builder

      public static CostBasedRouter.Builder builder()
      Returns a builder for configuring the router.

    • getClients

      public List<com.tnsai.llm.LLMClient> getClients()
      Description copied from interface: LLMRouter
      Returns all registered LLM clients.
      Specified by:
      getClients in interface LLMRouter

    • getPrimaryClient

      public com.tnsai.llm.LLMClient getPrimaryClient()
      Description copied from interface: LLMRouter
      Returns the currently selected/primary client.
      Specified by:
      getPrimaryClient in interface LLMRouter

    • selectByCapability

      public Optional<com.tnsai.llm.LLMClient> selectByCapability(Predicate<com.tnsai.llm.LLMCapabilities> filter)
      Description copied from interface: LLMRouter
      Selects a client that matches the given capability predicate.
      Specified by:
      selectByCapability in interface LLMRouter
      Parameters:
      filter - Predicate to filter by capabilities
      Returns:
      The first matching client, or empty if none match

    • getModel

      public String getModel()
      Specified by:
      getModel in interface com.tnsai.llm.LLMClient

    • getTemperature

      public float getTemperature()
      Specified by:
      getTemperature in interface com.tnsai.llm.LLMClient

    • getMaxTokens

      public Optional<Integer> getMaxTokens()
      Specified by:
      getMaxTokens in interface com.tnsai.llm.LLMClient

    • getCapabilities

      public com.tnsai.llm.LLMCapabilities getCapabilities()
      Specified by:
      getCapabilities in interface com.tnsai.llm.LLMClient

    • chat

      public com.tnsai.llm.ChatResponse chat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Description copied from interface: LLMRouter
      Routes a chat request to the appropriate client.

      The routing decision depends on the implementation:

      • FallbackRouter: Uses primary, falls back on error
      • CostBasedRouter: Selects cheapest viable option
      • LatencyBasedRouter: Selects fastest option
      Specified by:
      chat in interface com.tnsai.llm.LLMClient
      Specified by:
      chat in interface LLMRouter
      Parameters:
      message - The user message
      systemPrompt - Optional system prompt
      history - Optional conversation history
      tools - Optional tools
      Returns:
      The chat response from the selected client

    • streamChat

      public Stream<String> streamChat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      streamChat in interface com.tnsai.llm.LLMClient

    • streamChatWithSpec

      public Stream<com.tnsai.llm.ChatChunk> streamChatWithSpec(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      streamChatWithSpec in interface com.tnsai.llm.LLMClient

    • getCostStats

      public CostBasedRouter.CostStats getCostStats()
      Returns cost statistics.

    • getStats

      public LLMRouter.RoutingStats getStats()
      Description copied from interface: LLMRouter
      Returns routing statistics (calls per provider, failures, latencies).
      Specified by:
      getStats in interface LLMRouter

    • resetStats

      public void resetStats()
      Description copied from interface: LLMRouter
      Resets routing statistics.
      Specified by:
      resetStats in interface LLMRouter