Package com.tnsai.evaluation.evaluators.agent

package com.tnsai.evaluation.evaluators.agent
Agent-tier evaluators (TNS-373).

Where the existing top-level evaluators package scores single-turn text outputs (response quality, JSON correctness, cost efficiency) and evaluators.geval scores them with an LLM judge against caller-supplied criteria, this package scores agent traces — the plan, the steps the agent took, the tools it picked, and the end state.

The four metrics

  • PlanAdherenceEvaluator — LLM judge: did the agent stick to the declared plan?
  • StepEfficiencyEvaluator — deterministic: was the goal reached without burning extra tool calls? Pairs an expected step count (from expectedToolSequence or metadata['expected_steps']) against the actual count.
  • ToolCorrectnessEvaluator — LLM judge: were the chosen tools appropriate for the task? Softer signal than the deterministic ToolSelectionEvaluator, which checks exact-sequence precision/recall.
  • TaskCompletionEvaluator — LLM judge: did the final response actually solve the task? Fixed-contract complement to G-Eval's criteria-driven grading.

How they relate

Adherence asks "did you do the right thing?", efficiency asks "did you do it without waste?", correctness asks "were the right tools picked?", completion asks "did you reach the destination?". An agent can max out adherence on a wrong plan; conversely it can ignore the plan and still nail completion. Run them together to triangulate.

Plug-in style

Each metric implements the existing Evaluator SPI, so existing harnesses (EvalAssert, BenchmarkRunner) accept them without changes.

Since:
0.10.1 (TNS-373)