Package com.tnsai.llm.cache

Class InMemorySemanticCache

java.lang.Object
com.tnsai.llm.cache.InMemorySemanticCache
All Implemented Interfaces:
SemanticCache
public class InMemorySemanticCache extends Object implements SemanticCache
In-memory implementation of SemanticCache.

InMemorySemanticCache stores cached entries in a ConcurrentHashMap and performs linear search for similarity matching. For production with large caches, consider using a vector database.

Features

  • Thread-safe concurrent access
  • Configurable TTL (time-to-live)
  • LRU eviction when max size exceeded
  • Detailed statistics tracking

Usage Example


 SemanticCache cache = InMemorySemanticCache.builder()
     .embeddingProvider(new HashEmbeddingProvider())
     .ttlSeconds(3600)
     .maxEntries(1000)
     .build();

 // Store
 cache.put("What is Java?", "Java is a programming language...");

 // Retrieve
 Optional<CacheEntry> hit = cache.findSimilar("Java programming?", 0.8);

  • Method Details

    • builder

      public static InMemorySemanticCache.Builder builder()

    • put

      public CacheEntry put(String prompt, String response)
      Description copied from interface: SemanticCache
      Stores a prompt-response pair in the cache.
      Specified by:
      put in interface SemanticCache
      Parameters:
      prompt - The original prompt
      response - The LLM response
      Returns:
      The created cache entry

    • put

      public CacheEntry put(String prompt, Optional<String> systemPrompt, String response)
      Description copied from interface: SemanticCache
      Stores a prompt-response pair with system prompt.
      Specified by:
      put in interface SemanticCache
      Parameters:
      prompt - The user prompt
      systemPrompt - The system prompt
      response - The LLM response
      Returns:
      The created cache entry

    • findSimilar

      public Optional<CacheEntry> findSimilar(String prompt, double minSimilarity)
      Description copied from interface: SemanticCache
      Finds the most similar cached entry above the threshold.
      Specified by:
      findSimilar in interface SemanticCache
      Parameters:
      prompt - The prompt to search for
      minSimilarity - Minimum similarity threshold (0-1)
      Returns:
      The most similar entry if found, empty otherwise

    • findSimilar

      public Optional<CacheEntry> findSimilar(String prompt, Optional<String> systemPrompt, double minSimilarity)
      Description copied from interface: SemanticCache
      Finds the most similar cached entry with system prompt consideration.
      Specified by:
      findSimilar in interface SemanticCache
      Parameters:
      prompt - The user prompt
      systemPrompt - The system prompt
      minSimilarity - Minimum similarity threshold
      Returns:
      The most similar entry if found

    • findAllSimilar

      public List<SemanticCache.SimilarityResult> findAllSimilar(String prompt, double minSimilarity, int maxResults)
      Description copied from interface: SemanticCache
      Finds all similar entries above the threshold.
      Specified by:
      findAllSimilar in interface SemanticCache
      Parameters:
      prompt - The prompt to search for
      minSimilarity - Minimum similarity threshold
      maxResults - Maximum number of results
      Returns:
      List of similar entries sorted by similarity (descending)

    • getByHash

      public Optional<CacheEntry> getByHash(String promptHash)
      Description copied from interface: SemanticCache
      Gets an entry by exact prompt hash.
      Specified by:
      getByHash in interface SemanticCache
      Parameters:
      promptHash - The prompt hash
      Returns:
      The entry if found

    • remove

      public boolean remove(String id)
      Description copied from interface: SemanticCache
      Removes an entry from the cache.
      Specified by:
      remove in interface SemanticCache
      Parameters:
      id - The entry ID
      Returns:
      true if entry was removed

    • clear

      public void clear()
      Description copied from interface: SemanticCache
      Clears all entries from the cache.
      Specified by:
      clear in interface SemanticCache

    • size

      public int size()
      Description copied from interface: SemanticCache
      Returns the current number of entries.
      Specified by:
      size in interface SemanticCache

    • getStats

      public SemanticCache.CacheStats getStats()
      Description copied from interface: SemanticCache
      Returns cache statistics.
      Specified by:
      getStats in interface SemanticCache

    • resetStats

      public void resetStats()
      Description copied from interface: SemanticCache
      Resets cache statistics.
      Specified by:
      resetStats in interface SemanticCache