Package com.tnsai.llm.cache

Interface SemanticCache

All Known Implementing Classes:
InMemorySemanticCache
public interface SemanticCache
Semantic cache for LLM responses based on prompt similarity.

SemanticCache stores LLM responses and retrieves them based on semantic similarity rather than exact string matching. This enables cache hits for semantically equivalent prompts like:

  • "What is the capital of France?" ≈ "France's capital city?"
  • "Explain Python" ≈ "What is Python programming?"

How It Works

  1. Prompt is converted to embedding vector
  2. Similar embeddings are searched in cache
  3. If similarity > threshold, cached response is returned
  4. Otherwise, LLM is called and response is cached

Usage Example


 SemanticCache cache = InMemorySemanticCache.builder()
     .embeddingProvider(new OllamaEmbeddingProvider())
     .highThreshold(0.95)  // Direct hit
     .lowThreshold(0.70)   // Skip cache
     .ttlSeconds(3600)     // 1 hour TTL
     .maxEntries(10000)
     .build();

 // Check cache
 Optional<CacheEntry> hit = cache.findSimilar("What is Python?", 0.90);
 if (hit.isPresent()) {
     return hit.get().response();  // Cache hit!
 }

 // Cache miss - call LLM and store
 String response = llm.chat("What is Python?");
 cache.put("What is Python?", response);
See Also:

  • Method Details

    • put

      CacheEntry put(String prompt, String response)
      Stores a prompt-response pair in the cache.
      Parameters:
      prompt - The original prompt
      response - The LLM response
      Returns:
      The created cache entry

    • put

      CacheEntry put(String prompt, Optional<String> systemPrompt, String response)
      Stores a prompt-response pair with system prompt.
      Parameters:
      prompt - The user prompt
      systemPrompt - The system prompt
      response - The LLM response
      Returns:
      The created cache entry

    • findSimilar

      Optional<CacheEntry> findSimilar(String prompt, double minSimilarity)
      Finds the most similar cached entry above the threshold.
      Parameters:
      prompt - The prompt to search for
      minSimilarity - Minimum similarity threshold (0-1)
      Returns:
      The most similar entry if found, empty otherwise

    • findSimilar

      Optional<CacheEntry> findSimilar(String prompt, Optional<String> systemPrompt, double minSimilarity)
      Finds the most similar cached entry with system prompt consideration.
      Parameters:
      prompt - The user prompt
      systemPrompt - The system prompt
      minSimilarity - Minimum similarity threshold
      Returns:
      The most similar entry if found

    • findAllSimilar

      List<SemanticCache.SimilarityResult> findAllSimilar(String prompt, double minSimilarity, int maxResults)
      Finds all similar entries above the threshold.
      Parameters:
      prompt - The prompt to search for
      minSimilarity - Minimum similarity threshold
      maxResults - Maximum number of results
      Returns:
      List of similar entries sorted by similarity (descending)

    • getByHash

      Optional<CacheEntry> getByHash(String promptHash)
      Gets an entry by exact prompt hash.
      Parameters:
      promptHash - The prompt hash
      Returns:
      The entry if found

    • remove

      boolean remove(String id)
      Removes an entry from the cache.
      Parameters:
      id - The entry ID
      Returns:
      true if entry was removed

    • clear

      void clear()
      Clears all entries from the cache.

    • size

      int size()
      Returns the current number of entries.

    • getStats

      SemanticCache.CacheStats getStats()
      Returns cache statistics.

    • resetStats

      void resetStats()
      Resets cache statistics.