Package com.tnsai.evaluation.autoharness
package com.tnsai.evaluation.autoharness
Self-improving agent evaluation loop (auto-harness pattern).
The auto-harness pipeline automatically mines agent failures, clusters them by root cause, generates regression eval cases, and gates changes against the accumulated eval suite.
Core abstractions
FailureRecord— single agent failureFailureMiner— extracts failures from tracesFailureCluster— grouped failures by root causeFailureClusterer— groups failures into clustersEvalCaseGenerator— converts clusters to eval casesRegressionGate— gates on pass rate thresholdAutoHarnessLoop— orchestrates the full loop
- Since:
- 0.1.0
-
ClassDescriptionOrchestrates the self-improving evaluation loop: mine → cluster → generate → gate.Result of a single auto-harness loop iteration.Generates BenchmarkRunner-compatible regression test cases from failure clusters.Thread-safe
FailureMinerthat collects failures reported viaDefaultFailureMiner.record(FailureRecord).A generated regression test case derived from aFailureCluster.ConvertsFailureClusters into regressionEvalCases.A group ofFailureRecords sharing a common root cause.GroupsFailureRecords intoFailureClusters by root cause.ExtractsFailureRecords from agent execution traces.Captures a single agent failure from an execution trace.Runs an eval suite and gates on a pass rate threshold.Result of a regression gate evaluation.GroupsFailureRecords byerrorType + phase.