Package com.tnsai.llm.cache

Class PromptCachingClient

java.lang.Object
com.tnsai.llm.cache.PromptCachingClient
All Implemented Interfaces:
com.tnsai.llm.LLMClient
public class PromptCachingClient extends Object implements com.tnsai.llm.LLMClient
LLMClient wrapper that adds prompt caching support.

PromptCachingClient wraps any LLMClient and adds cache control markers to system prompts, tools, and conversation history.

Features

  • Automatic system prompt caching
  • Tool definition caching
  • Conversation history caching (configurable breakpoints)
  • Cache usage tracking and statistics

Usage Example


 // Wrap an Anthropic client
 LLMClient baseClient = new AnthropicClient("claude-sonnet-4-20250514");
 PromptCachingClient client = PromptCachingClient.builder()
     .client(baseClient)
     .cacheSystemPrompt(true)
     .cacheTools(true)
     .cacheHistoryBreakpoints(2)  // Cache last 2 history points
     .build();

 // Use normally - caching is automatic
 ChatResponse response = client.chat("Hello", systemPrompt, history, tools);

 // Check cache usage
 if (response.hasCacheUsage()) {
     System.out.println("Cache read tokens: " + response.getCacheReadInputTokens().get());
 }

Cost Savings

  • Cache reads: 90% cheaper than regular input tokens
  • Cache writes: 25% more expensive (one-time cost)
  • TTL: 5 minutes (refreshed on each use)
See Also:

  • Method Details

    • builder

      public static PromptCachingClient.Builder builder()

    • getModel

      public String getModel()
      Specified by:
      getModel in interface com.tnsai.llm.LLMClient

    • getTemperature

      public float getTemperature()
      Specified by:
      getTemperature in interface com.tnsai.llm.LLMClient

    • getMaxTokens

      public Optional<Integer> getMaxTokens()
      Specified by:
      getMaxTokens in interface com.tnsai.llm.LLMClient

    • getCapabilities

      public com.tnsai.llm.LLMCapabilities getCapabilities()
      Specified by:
      getCapabilities in interface com.tnsai.llm.LLMClient

    • chat

      public com.tnsai.llm.ChatResponse chat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      chat in interface com.tnsai.llm.LLMClient

    • chat

      public com.tnsai.llm.ChatResponse chat(List<com.tnsai.models.chat.ContentPart> messageParts, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      chat in interface com.tnsai.llm.LLMClient

    • streamChat

      public Stream<String> streamChat(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      streamChat in interface com.tnsai.llm.LLMClient

    • streamChatWithSpec

      public Stream<com.tnsai.llm.ChatChunk> streamChatWithSpec(String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Specified by:
      streamChatWithSpec in interface com.tnsai.llm.LLMClient

    • getTotalCacheReadTokens

      public long getTotalCacheReadTokens()
      Gets total cache read tokens across all requests.

    • getTotalCacheCreationTokens

      public long getTotalCacheCreationTokens()
      Gets total cache creation tokens across all requests.

    • getRequestCount

      public long getRequestCount()
      Gets total request count.

    • getCacheHitRate

      public double getCacheHitRate()
      Gets cache hit rate (0.0 to 1.0).

    • getEstimatedSavings

      public double getEstimatedSavings()
      Gets estimated cost savings from caching.
      Returns:
      Savings as a fraction (e.g., 0.85 = 85% savings)

    • resetStats

      public void resetStats()
      Resets statistics.

    • getDelegate

      public com.tnsai.llm.LLMClient getDelegate()
      Gets the underlying delegate client.

    • isCacheSystemPromptEnabled

      public boolean isCacheSystemPromptEnabled()
      Checks if system prompt caching is enabled.

    • isCacheToolsEnabled

      public boolean isCacheToolsEnabled()
      Checks if tool caching is enabled.

    • getCacheHistoryBreakpoints

      public int getCacheHistoryBreakpoints()
      Gets the number of history breakpoints for caching.