Package com.tnsai.evaluation.evaluators.agent

Class ToolCorrectnessEvaluator

java.lang.Object

com.tnsai.evaluation.evaluators.AbstractJudgeEvaluator

com.tnsai.evaluation.evaluators.agent.ToolCorrectnessEvaluator

All Implemented Interfaces:

Evaluator

public final class ToolCorrectnessEvaluator
extends AbstractJudgeEvaluator

Agent-tier metric: were the tools the agent invoked semantically appropriate for the task?

Companion to ToolSelectionEvaluator, which already exists in this module: that one is deterministic precision/recall against a fixed expected sequence. This one asks an LLM judge to rate appropriateness when an exact ground-truth trace isn't available — a softer signal that catches cases like "the agent called web_search when knowledge_base_lookup was a better fit" even though both were on the allow-list.

Inputs the judge sees: the user's request, the declared plan (instructions, optional), and the ordered list of tool names the agent actually invoked. The judge produces a 1–5 rating normalised to [0.0, 1.0].

Returns a passing score with a "no tools used" note when the actual sequence is empty — having nothing to evaluate is the caller's signal, not a failure of this metric.

Since:

0.10.1 (TNS-373)

Nested Class Summary

Nested classes/interfaces inherited from interface com.tnsai.evaluation.spi.Evaluator

Evaluator.EvaluationInput

Constructor Summary

Constructors

Constructor	Description
ToolCorrectnessEvaluator(LLMJudge judge)

Method Summary

Methods inherited from class com.tnsai.evaluation.evaluators.AbstractJudgeEvaluator

evaluate, name

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ToolCorrectnessEvaluator

public ToolCorrectnessEvaluator(LLMJudge judge)