Package com.tnsai.server.rag

Class FileIndexer

java.lang.Object

com.tnsai.server.rag.FileIndexer

public final class FileIndexer
extends Object

Server index adapter over the canonical FileIngestionService.

Features:

• Shares discovery, ignore, detection, extraction, and chunking with declarative RAG
• Pipeline fingerprints for incremental re-indexing (content plus extractor, chunker, format, and selector identity)
• Stage-then-commit so a failed replacement never publishes a mixed generation to TF-IDF and BM25
• Reconciles deleted, renamed, previous-root, and now-skipped files against the previous load through FileSourceReconciler. A file that fails to read is not stale: previous chunks stay until a later complete ingest replaces them. A failed walk leaves the index untouched and reports FileIndexer.ProgressStage.FAILED
• Progress reporting via Consumer of FileIndexer.IndexProgress

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	FileIndexer.IndexProgress	Progress snapshot during discovery, normalization, and index mutation.
static final record	FileIndexer.IndexResult	Summary of a completed index run.
static enum	FileIndexer.ProgressStage	Indexing lifecycle stage.

Constructor Summary

Constructors

Constructor	Description
FileIndexer()	Creates an indexer using the shared default admission caps.
FileIndexer(com.tnsai.intelligence.rag.binding.FileIngestionService.Limits limits)	Creates an indexer with the same admission-cap policy as FileIngestionService.

Method Summary

Modifier and Type	Method	Description
void	clearHashes()	Forgets the manifest of previous runs, forcing a full re-index on the next run.
FileIndexer.IndexResult	index(com.tnsai.rag.KnowledgeSourceConfig source, com.tnsai.knowledge.KnowledgeBase knowledgeBase, BM25Stream bm25Stream, Consumer<FileIndexer.IndexProgress> progressCallback)	Indexes one canonical FILE source, including its declared format, selectors, and knowledge-unit slicing policy.
FileIndexer.IndexResult	index(Path rootDir, com.tnsai.knowledge.KnowledgeBase knowledgeBase, BM25Stream bm25Stream, Consumer<FileIndexer.IndexProgress> progressCallback)	Indexes one supported file or all supported files under a directory.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

FileIndexer

public FileIndexer()

Creates an indexer using the shared default admission caps.

FileIndexer

public FileIndexer(com.tnsai.intelligence.rag.binding.FileIngestionService.Limits limits)

Creates an indexer with the same admission-cap policy as FileIngestionService.

Parameters:

limits - max file / depth / corpus policy

Method Details

index

public FileIndexer.IndexResult index(Path rootDir,
 com.tnsai.knowledge.KnowledgeBase knowledgeBase,
 BM25Stream bm25Stream,
 Consumer<FileIndexer.IndexProgress> progressCallback)
                              throws IOException

Indexes one supported file or all supported files under a directory. A relative rootDir resolves against the working directory, as a Path does, whatever base directory ClassConfig.useBaseDirectory(java.nio.file.Path) set.

Parameters:

rootDir - source file or directory to ingest

knowledgeBase - target knowledge base

progressCallback - receives progress updates (may be null)

Returns:

result summary

Throws:

IOException - if rootDir is not a file or directory

index

public FileIndexer.IndexResult index(com.tnsai.rag.KnowledgeSourceConfig source,
 com.tnsai.knowledge.KnowledgeBase knowledgeBase,
 BM25Stream bm25Stream,
 Consumer<FileIndexer.IndexProgress> progressCallback)
                              throws IOException

Indexes one canonical FILE source, including its declared format, selectors, and knowledge-unit slicing policy. A relative source path resolves by KnowledgePaths.resolve(java.lang.String): against the base directory ClassConfig.useBaseDirectory(java.nio.file.Path) set, or the working directory when none is set.

Parameters:

source - canonical FILE source

knowledgeBase - target knowledge base

bm25Stream - target BM25 index, or null

progressCallback - receives progress updates (may be null)

Returns:

result summary

Throws:

IOException - if the configured path is not a file or directory

Since:

0.15.0

clearHashes

public void clearHashes()

Forgets the manifest of previous runs, forcing a full re-index on the next run. Waits for a running index to finish so that its commit cannot restore the manifest afterwards.