All Classes and Interfaces
Class
Description
Base for single-call LLM-as-judge evaluators: build one prompt, send it to the
LLMJudge once, parse the verdict.Measures whether the agent's response actually addresses the user's query.
Orchestrates the self-improving evaluation loop:
mine → cluster → generate → gate.
Result of a single auto-harness loop iteration.
A saved baseline from a benchmark run, used for regression detection.
Persistent store for benchmark baselines.
Aggregated result of running a benchmark dataset.
Result for a single test case within a benchmark run.
Runs benchmark datasets against an agent execution function,
collecting evaluation results from registered evaluators.
Output from agent execution for a test case.
Detects demographic, cultural, or ideological bias in agent responses.
Measures whether retrieved documents are relevant to the query.
Measures whether all relevant information needed for the answer was retrieved.
Measures whether a multi-turn conversation achieved its stated goal.
Evaluates cost efficiency: cost per successful task completion.
Generates BenchmarkRunner-compatible regression test cases from failure clusters.
Thread-safe
FailureMiner that collects failures reported via
DefaultFailureMiner.record(FailureRecord).Fixed TR/EN retrieval fixture shipped with the evaluation module.
Named embedder under comparison.
Per-model scorecard across every slice.
One query and its gold document inside a
EmbeddingBenchSlice.Picks a default candidate and a quality override from measured scores.
Renders a markdown table for the measured model × slice matrix.
Runs the bilingual fixture against one or more
EmbeddingBenchModels.Language slice of the bilingual retrieval fixture.
Retrieval metrics for one language slice.
Fluent assertion API for evaluation tests.
Assertion chain for evaluation results.
A generated regression test case derived from a
FailureCluster.Converts
FailureClusters into regression EvalCases.JUnit 5 extension that powers
EvalTest.Writes evaluation scores back onto
AgentTrace spans for observability dashboards.Marks a method as an evaluation test.
Aggregated evaluation report with drill-down capability.
Result of a single evaluation.
SPI interface for all evaluators.
Input context provided to evaluators.
A group of
FailureRecords sharing a common root cause.Groups
FailureRecords into FailureClusters by root cause.Extracts
FailureRecords from agent execution traces.Captures a single agent failure from an execution trace.
Measures whether the agent's response is grounded in retrieved documents.
G-Eval: LLM-as-judge evaluator using a 2-step chain-of-thought process.
Detects hallucinated content in agent responses by checking factual claims
against provided context.
Evaluates whether the agent followed given instructions.
Evaluates JSON correctness of agent output — parse validity, required fields,
and type conformance.
Measures whether the agent retains information from earlier conversation turns.
Functional interface for LLM-based judging.
EmbeddingFunction over an Ollama /api/embed endpoint.Detects PII leakage in agent responses using regex + checksum validation.
Agent-tier metric: did the agent stick to its declared plan?
Pass/fail gate based on configurable thresholds and baseline regression detection.
Quality gate verdict.
Configuration for quality gate pass/fail thresholds.
Runs an eval suite and gates on a pass rate threshold.
Result of a regression gate evaluation.
Evaluates response quality across three dimensions:
relevance, completeness, and accuracy.
Shared sequence comparison utilities for evaluators.
Groups
FailureRecords by errorType + phase.Statistical analysis utilities for evaluation scores.
Statistical summary.
Agent-tier metric: did the agent reach the goal without burning extra
tool calls?
Agent-tier metric: did the agent's final state actually solve the
task?
A single test case in a benchmark dataset.
A collection of test cases for benchmarking.
Deterministic bag-of-tokens embedding used as an offline bench baseline.
Agent-tier metric: were the tools the agent invoked semantically
appropriate for the task?
Evaluates whether the agent selected the correct tools in the right order.
Detects toxic, harmful, or offensive content in agent responses.
Bridges AgentTrace → Evaluator → score annotation pipeline.
Record of a failed evaluation for downstream processing (e.g., AutoHarness).
Converts completed
AgentTrace spans into Evaluator.EvaluationInput records.Compares evaluation results across multiple runs to detect trends.
Measures whether the last assistant turn is relevant to the preceding conversation context.