Package com.tnsai.intelligence.rag.binding

Class FileIngestionService

java.lang.Object
com.tnsai.intelligence.rag.binding.FileIngestionService
public final class FileIngestionService extends Object
Canonical filesystem ingestion module shared by declarative and Server RAG.

Reads textual files from disk. The Knowledge.path() value may point to a single file or a directory. DocumentFormat.AUTO detects each candidate from a bounded prefix and uses its extension only as a fallback hint. Explicit DocumentFormat.TEXT accepts any file name; every other explicit format validates all extensions before ingestion and validates content before returning a document.

Small files retain their source-local identity and normalized content. Larger files are split through NormalizedDocumentChunker; structural extractor blocks take precedence over generic language, line, and character boundaries. Retrieval limits therefore apply to chunks rather than to the corpus's physical file layout (TAN-5327).

Candidate contract

A directory walk is recursive and deterministic. Symbolic links are never followed. KnowledgeSourceConfig.unit() selects the retrieval-unit slicing policy and is also a fingerprint input, so changing the declared policy invalidates the staged index. AUTO is omitted from the payload and preserves the previous ingest path byte-for-byte; any other pipeline change may still invalidate an undeclared source. KnowledgeSourceConfig.include() and KnowledgeSourceConfig.exclude() run before format validation or reads: empty include admits every regular non-symbolic-link file, empty exclude admits every include match, and exclude always wins. AUTO skips unsupported and binary survivors, but fails when a non-empty source contains no usable file after selection. Explicit structured formats reject an extension mismatch for the complete source before any file is read. Binary formats require an optional extractor and are never decoded as text by this loader. Textual candidates retain a bounded, structurally normalized UTF-8 representation after validation.

Path safety

A relative path is resolved by KnowledgePaths.resolve(java.lang.String): against the base directory ClassConfig.useBaseDirectory set, or the JVM's working directory when none is set; absolute paths are honoured. When the file-system provider supplies secure directory streams, every path component is opened relative to a directory handle with symbolic-link following disabled. The operating-system-owned macOS aliases from /var, /tmp, and /etc into /private are canonicalized; all source-tree symbolic links remain rejected. Other providers use a no-follow traversal that validates the complete path before and after each read and rejects root escapes. That fallback is for application-owned source trees and does not claim race-free ingestion of a corpus concurrently writable by an adversary; such deployments require a provider with secure directory streams. AUTO reads only a bounded prefix before rejecting unsupported entries, so their complete bytes are never retained in the loaded-document list. Accepted textual files are normalized through the discovered ContentExtractor set before stable, provenance-bearing chunks enter the vector store.
Since:
0.13.0

  • Constructor Details

    • FileIngestionService

      public FileIngestionService()
      Creates a service using the framework's bounded source defaults.

    • FileIngestionService

      public FileIngestionService(FileIngestionService.Limits limits)
      Creates a service with an explicit admission-cap policy.
      Parameters:
      limits - max file / depth / corpus policy

  • Method Details

    • ingest

      public List<FileIngestionService.IngestedChunk> ingest(com.tnsai.rag.KnowledgeSourceConfig source)
      Discovers, validates, extracts, and chunks one FILE knowledge source.
      Parameters:
      source - canonical FILE-source configuration
      Returns:
      deterministic chunks with flattened provenance, a content hash, and a pipeline fingerprint

    • ingestWithProgress

      public FileIngestionService.IngestionResult ingestWithProgress(com.tnsai.rag.KnowledgeSourceConfig source, Consumer<FileIngestionService.IngestionProgress> observer)
      Ingests one FILE source while reporting discovery and processing progress.
      Parameters:
      source - canonical FILE-source configuration
      observer - synchronous progress observer
      Returns:
      chunks plus the complete discovered-file snapshot