Package com.tnsai.llm.cache

Class CachedLLMClient

java.lang.Object
com.tnsai.llm.cache.CachedLLMClient
All Implemented Interfaces:
com.tnsai.llm.LLMClient
public class CachedLLMClient extends Object implements com.tnsai.llm.LLMClient
LLM client wrapper that adds semantic caching capabilities.

CachedLLMClient wraps any LLMClient and adds transparent caching. Similar prompts return cached responses instead of calling the LLM, providing significant cost savings and latency reduction.

How It Works

 User Prompt → Check Cache → Cache Hit? → Return Cached Response
                    ↓                            ↑
                 Cache Miss → Call LLM → Store Response

Thresholds

  • High Threshold (0.95): Direct cache hit - return immediately
  • Low Threshold (0.70): Gray zone - may need verification
  • Below Low: Cache miss - call LLM

Usage Example


 // Wrap any LLM client
 LLMClient cached = CachedLLMClient.wrap(new OpenAIClient("gpt-4o"))
     .withCache(InMemorySemanticCache.builder()
         .embeddingProvider(new OllamaEmbeddingProvider())
         .ttlSeconds(3600)
         .build())
     .highThreshold(0.95)
     .lowThreshold(0.70)
     .build();

 // First call - cache miss, calls LLM
 cached.chat("What is Python?");

 // Similar question - cache hit!
 cached.chat("Tell me about Python");  // Returns cached response

 // Check stats
 CachedLLMClient c = (CachedLLMClient) cached;
 System.out.println("Hit rate: " + c.getCacheStats().hitRate());

Cost Savings

 Without Cache: 1000 requests × $0.003/req = $3.00
 With 70% hit rate: 300 requests × $0.003 = $0.90
 Savings: $2.10 (70%)
See Also:

  • Method Details

    • wrap

      public static CachedLLMClient.Builder wrap(com.tnsai.llm.LLMClient client)
      Creates a builder to wrap the given LLM client.

    • chat

      public com.tnsai.llm.ChatResponse chat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      chat in interface com.tnsai.llm.LLMClient

    • streamChat

      public Stream<String> streamChat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      streamChat in interface com.tnsai.llm.LLMClient

    • streamChatWithSpec

      public Stream<com.tnsai.llm.ChatChunk> streamChatWithSpec(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      streamChatWithSpec in interface com.tnsai.llm.LLMClient

    • getModel

      public String getModel()
      Specified by:
      getModel in interface com.tnsai.llm.LLMClient

    • getTemperature

      public float getTemperature()
      Specified by:
      getTemperature in interface com.tnsai.llm.LLMClient

    • getMaxTokens

      public Optional<Integer> getMaxTokens()
      Specified by:
      getMaxTokens in interface com.tnsai.llm.LLMClient

    • getCapabilities

      public com.tnsai.llm.LLMCapabilities getCapabilities()
      Specified by:
      getCapabilities in interface com.tnsai.llm.LLMClient

    • getDelegate

      public com.tnsai.llm.LLMClient getDelegate()
      Returns the underlying LLM client.

    • getCacheStats

      public SemanticCache.CacheStats getCacheStats()
      Returns cache statistics.

    • getCache

      public SemanticCache getCache()
      Returns the cache instance.

    • clearCache

      public void clearCache()
      Clears the cache.

    • warmCache

      public void warmCache(Map<String,String> promptResponses)
      Warms the cache with common prompts and responses.
      Parameters:
      promptResponses - Map of prompts to expected responses