Package com.tnsai.memory.advanced

Class BM25Index

java.lang.Object
com.tnsai.memory.advanced.BM25Index
public final class BM25Index extends Object
BM25 (Okapi BM25) index for text-based memory retrieval.

Implements the standard BM25 scoring formula: 

   score(D, Q) = sum_i IDF(q_i) * (f(q_i, D) * (k1 + 1))
                                  / (f(q_i, D) + k1 * (1 - b + b * |D| / avgdl))
where k1=1.2 and b=0.75 are the standard tuning parameters.

Document frequencies, term frequencies and the total token count are kept up to date by add(com.tnsai.memory.strategy.MemoryEntry) and remove(java.lang.String), so a search reads only the postings of its query terms instead of rescanning every document. Entries with the same score are ordered by ascending id.

Thread-safe: a read/write lock keeps the counters one consistent state; searches share the read lock, changes take the write lock.

  • Constructor Details

    • BM25Index

      public BM25Index()

  • Method Details

    • add

      public void add(MemoryEntry entry)
      Adds an entry to the index, replacing any entry with the same id.
      Parameters:
      entry - the memory entry to index

    • remove

      public boolean remove(String id)
      Removes an entry from the index.
      Parameters:
      id - the entry ID
      Returns:
      true if the entry was present and removed

    • search

      public List<ScoredEntry> search(String query, int topK)
      Searches for the top-K entries matching the query using BM25 scoring.
      Parameters:
      query - the search query
      topK - maximum number of results
      Returns:
      scored entries sorted by descending BM25 score

    • size

      public int size()
      Returns the number of indexed entries.

    • clear

      public void clear()
      Clears the entire index.