Package com.tnsai.rag

Class NormalizedDocumentChunker

java.lang.Object
com.tnsai.rag.NormalizedDocumentChunker
public final class NormalizedDocumentChunker extends Object
Splits normalized documents and source files into stable retrieval units.

Small documents retain their original identity and content. Larger normalized documents prefer extractor-provided DocumentBlocks so format-native structural paths and line provenance survive indexing. Raw source files use language boundaries where available and otherwise fall back to bounded line and character groups.

Fixed-size windows overlap by OVERLAP_LINES so a sentence that straddles an arbitrary cut is still present in one chunk. Semantic boundaries — markdown headings, source-code symbols — do not overlap: those split points are already meaningful. @Knowledge.unit = LINE and HEADING emit position-addressed slices rather than overlapping windows, so overlap is suppressed there. DOCUMENT is a single unit. AUTO and CHUNK apply overlap only on the fixed-size fallback.

Since:
0.13.0

  • Field Details

    • MAX_LINES_PER_CHUNK

      public static final int MAX_LINES_PER_CHUNK
      Maximum physical lines retained in one generic chunk.
      See Also:

    • OVERLAP_LINES

      public static final int OVERLAP_LINES
      Lines repeated from the previous fixed-size window. 15% of MAX_LINES_PER_CHUNK; not applied across semantic boundaries or LINE/HEADING units.
      Since:
      0.15.0
      See Also:

    • MAX_CHARS_PER_CHUNK

      public static final int MAX_CHARS_PER_CHUNK
      Maximum characters retained in one generic chunk.
      See Also:

    • MAX_LINES_PER_DOCUMENT

      public static final int MAX_LINES_PER_DOCUMENT
      Maximum physical lines admitted from one source document.

      This is the document-size contract, and it is deliberately unchanged by overlapping windows. Overlap repeats OVERLAP_LINES lines per window, so covering the same document costs proportionally more windows; that cost is paid by a dedicated overlap expansion budget rather than by shrinking what a caller may ingest.

      Since:
      0.15.0
      See Also:

    • MAX_CHUNKS_PER_DOCUMENT

      public static final int MAX_CHUNKS_PER_DOCUMENT
      Maximum retrieval units admitted from one source document.

      This remains the pre-overlap admission ceiling. Overlapping windows consume a separate internal expansion budget, so callers that suppress overlap do not inherit a fourfold global resource allowance.

      See Also:

    • MAX_DOCUMENT_CHARS

      public static final int MAX_DOCUMENT_CHARS
      Maximum characters admitted from one source document.
      See Also:

  • Method Details

    • chunk

      public static List<NormalizedDocumentChunker.Chunk> chunk(ExtractedDocument document)
      Chunks one extractor-normalized document.
      Parameters:
      document - normalized document
      Returns:
      immutable chunks, or an empty list for blank content

    • chunk

      public static List<NormalizedDocumentChunker.Chunk> chunk(ExtractedDocument document, KnowledgeUnit unit)
      Applies an explicit ingest unit to one extractor-normalized document.
      Parameters:
      document - normalized document
      unit - requested ingest unit
      Returns:
      immutable retrieval units, or an empty list for blank content
      Throws:
      IllegalArgumentException - when the unit is incompatible with the document format or would exceed a retrieval-unit limit
      Since:
      0.15.0

    • validateUnit

      public static void validateUnit(KnowledgeUnit unit, DocumentFormat format)
      Validates one explicit ingest unit against a concrete or AUTO format. AUTO defers compatibility until content detection resolves the format.
      Parameters:
      unit - requested ingest unit
      format - declared or detected document format
      Throws:
      IllegalArgumentException - when the pair has no defined slicing semantics
      Since:
      0.15.0

    • chunk

      public static List<NormalizedDocumentChunker.Chunk> chunk(Path filePath, String content, String language)
      Chunks a raw source file using the shared server/declarative fallback.
      Parameters:
      filePath - stable source-relative file path
      content - UTF-8 text
      language - normalized language key
      Returns:
      immutable chunks

    • detectLanguage

      public static String detectLanguage(String fileName)
      Derives the normalized language key from a file extension.