Package com.tnsai.intelligence.rag.vector

Class SelectiveReembedIndex

java.lang.Object
com.tnsai.intelligence.rag.vector.SelectiveReembedIndex
public final class SelectiveReembedIndex extends Object
Local vector index that re-embeds only chunks whose content hash changed.

Unique SHA-256 payloads share one stored embedding. A source is a cheap graph of those hashes (consecutive chunks are neighbors). Qdrant and pgvector remain the production VectorIndex backends; this is the laptop/default ingest path.

Storage vs a naive full reindex is the unique-hash set, not a marketing 97%. Recompute cost on a one-file edit is that file's new hashes, not the whole corpus.

Since:
0.14.0

  • Constructor Details

    • SelectiveReembedIndex

      public SelectiveReembedIndex(com.tnsai.memory.advanced.EmbeddingFunction embeddings)

  • Method Details

    • syncSource

      public SelectiveReembedIndex.ReindexStats syncSource(String sourceId, List<String> chunks)
      Replaces one source's chunks. Unchanged hashes keep their vector.
      Parameters:
      sourceId - stable source identity (file path, document id)
      chunks - ordered chunk texts
      Returns:
      reuse / embed / remove counts

    • search

      public List<SelectiveReembedIndex.ScoredChunk> search(String query, int topK)
      Cosine search over unique stored embeddings.
      Parameters:
      query - query text
      topK - maximum hits
      Returns:
      highest-first hits

    • size

      public int size()
      Unique stored embeddings.

    • neighbors

      public Set<String> neighbors(String contentHash)
      Neighbor hashes for one content hash (same-source adjacency).

    • uniqueEmbeddingRatio

      public double uniqueEmbeddingRatio()
      Unique embeddings divided by the naive per-chunk count. Lower is better; 1.0 means no sharing.