Package com.tnsai.rag.files

Class IncrementalTextFilesRetriever

java.lang.Object
com.tnsai.rag.files.IncrementalTextFilesRetriever
All Implemented Interfaces:
RetrievalEngine.Retriever
public final class IncrementalTextFilesRetriever extends Object implements RetrievalEngine.Retriever
Searches what one agent has written in a run: every text file under its <run>/episode-N/<agent>/ folders, ranked by keyword relevance (BM25).

Each line of a file is one piece of evidence, so a match brings the line it was found on, and a line longer than 2000 characters is cut into pieces. The index follows the files as they grow: each search reads only what was appended since the last one, and reads a file again from its start when it shrank, was replaced or was rewritten where it had been read. A file with a NUL byte in its first 8192 bytes is not text and is left out; in a text file, bytes that are not UTF-8 are read as the replacement character. Files at the run's top level, outside the episode folders, are not searched. The run folder is taken by its real path, so a link to it is followed; below it symbolic links are not followed, and a file whose real path is not under the run folder's is left out. A file that cannot be read is searched as it was last read, with one WARN per path.

A run without any such folder or file yet gives an empty result, which is not a failure: an agent that has written nothing has nothing to find.

Scope and conversation

The request's current episode is the value of its ChatRetrievalContext.CURRENT_EPISODE_FILTER metadata filter: the number N of the episode-N folder, written in decimal digits. Any other value is rejected with an IllegalArgumentException, so a mistyped value cannot switch the filters below off.

  • With IncrementalTextFilesRetriever.Scope.RUN, the default, every episode of the run is searched, not only the current one. With IncrementalTextFilesRetriever.Scope.EPISODE only the current episode is searched, and a request without one finds nothing.
  • A .history or .log record is evidence only for a request that has both a current episode and a conversation; a request without either is given no such record from any episode.
  • A .history or .log record of the current episode is evidence when it was written outside any conversation or in the request's conversation. A record of the current episode without the conversation field, and a line before a file's first record, is not evidence, with one WARN per file. The conversation filter is scoped to the current episode because a host's conversation identity need not carry across episodes: the records of the other episodes are evidence whatever conversation they were written in, with or without the field.
  • Other files are not filtered by conversation.

IncrementalTextFilesRetriever.Scope.RUN therefore suits a single-user host, whose other conversations are its own user's; a host whose run serves several users uses IncrementalTextFilesRetriever.Scope.EPISODE.

Evidence text is passed through the builder's Masker before it is indexed, so what the masker hides is neither returned nor searchable; masking is off by default. A record is masked as a whole before it is cut into lines and pieces, so a secret written over several of its lines, or across where a long line is cut, is hidden whole; in a file without records, the lines read together are masked together. What is read in separate reads, such as a record still being written or a file longer than one read block, is masked separately. Masking that joins lines leaves a record fewer lines. A record's level and conversation are taken from it as written, and the turn message is masked before it is compared with the records.

In a .history or .log file, a record is a line that RecordParser parses and the lines after it that do not; in any other file each line is a record. A log record at DEBUG or TRACE is left out. The builder's RecordInterpreter reads every other record as a note, a message told or said, or something left out; the default, RecordInterpreter.NONE, reads every record as a note. From that:

  • The search is for the request's turn message when it has one, else for its query. With a host interpreter that reads records of being told the turn message as told, those records are not returned, so the evidence holds what the agent knew before the turn; with the default, none is left out this way.
  • Of the records a host interpreter reads as telling or saying the same message, only the best ranked is given; a record that kept only a message's start is a copy of the record whose message starts with it. Of the notes of the same text, without their record's time and writer, only the best ranked is given, with any interpreter.

Retrieval is serialized per instance.

Since:
0.18.6

  • Field Details

    • DEFAULT_SOURCE_NAME

      public static final String DEFAULT_SOURCE_NAME
      The source name the evidence gives unless the builder sets another.
      See Also:

    • MAX_PIECE_CHARS

      public static final int MAX_PIECE_CHARS
      The longest piece of a line one piece of evidence holds.
      See Also:

  • Method Details

    • builder

      public static IncrementalTextFilesRetriever.Builder builder(Path runFolder, String agentName)
      Starts a retriever of one agent's files in one run.
      Parameters:
      runFolder - the run whose episode-N folders are searched
      agentName - the agent whose folders they are
      Returns:
      the builder
      Throws:
      NullPointerException - if an argument is null

    • retrieve

      public RetrievalResult retrieve(RetrievalRequest request)
      Description copied from interface: RetrievalEngine.Retriever
      Executes retrieval for a validated request.

      Implementations that return true from RetrievalEngine.Retriever.ownsDeadlineEnforcement() must evaluate cancellation and deadlines against one consistent clock before, between, and after blocking phases, and apply the request deadline to backend-native I/O timeouts. Other implementations retain the engine wrapper's entry and exit checks against the system clock.

      Specified by:
      retrieve in interface RetrievalEngine.Retriever
      Parameters:
      request - validated retrieval request
      Returns:
      non-null retrieval result