Package com.tnsai.evaluation.benchmark

Class BenchmarkRunner

java.lang.Object

com.tnsai.evaluation.benchmark.BenchmarkRunner

public final class BenchmarkRunner
extends Object

Runs benchmark datasets against an agent execution function, collecting evaluation results from registered evaluators.

Usage

 BenchmarkRunner runner = BenchmarkRunner.builder()
     .evaluator(new ResponseQualityEvaluator())
     .evaluator(new ToolSelectionEvaluator())
     .agentFunction(input -> agentExecute(input))
     .build();

 BenchmarkResult result = runner.run(dataset);
 

Since:

0.1.0

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	BenchmarkRunner.AgentOutput	Output from agent execution for a test case.
static final class	BenchmarkRunner.Builder

Method Summary

Modifier and Type	Method	Description
static BenchmarkRunner.Builder	builder()
BenchmarkResult	run(TestDataset dataset)	Runs all test cases in the dataset and collects results.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

run

public BenchmarkResult run(TestDataset dataset)

Runs all test cases in the dataset and collects results.

builder

public static BenchmarkRunner.Builder builder()