Package com.tnsai.evaluation.evaluators.geval

Class GEvalEvaluator

java.lang.Object
com.tnsai.evaluation.evaluators.geval.GEvalEvaluator
All Implemented Interfaces:
Evaluator
public final class GEvalEvaluator extends Object implements Evaluator
G-Eval: LLM-as-judge evaluator using a 2-step chain-of-thought process.

Step 1: The LLM generates intermediate evaluation steps (CoT) for the given criteria. Step 2: Using those steps, the LLM scores the agent's response on a 1-5 scale, which is normalized to [0.0, 1.0].

Based on the G-Eval paper (Liu et al., 2023) — "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment".

Usage


 LLMJudge judge = prompt -> myLlmClient.chat(prompt);
 var evaluator = new GEvalEvaluator("coherence",
     "Does the response maintain logical flow and consistency?", judge);
 EvaluationResult result = evaluator.evaluate(input);
Since:
0.2.0

  • Constructor Details

    • GEvalEvaluator

      public GEvalEvaluator(String criteriaName, String criteriaDescription, LLMJudge judge)
      Creates a G-Eval evaluator for a specific evaluation criteria.
      Parameters:
      criteriaName - short identifier for this criteria (e.g. "coherence")
      criteriaDescription - human-readable description of what to evaluate
      judge - LLM function used for chain-of-thought judging

  • Method Details

    • name

      public String name()
      Description copied from interface: Evaluator
      Unique name for this evaluator.
      Specified by:
      name in interface Evaluator

    • evaluate

      public EvaluationResult evaluate(Evaluator.EvaluationInput context)
      Description copied from interface: Evaluator
      Evaluates agent behavior for a single interaction.
      Specified by:
      evaluate in interface Evaluator
      Parameters:
      context - evaluation context containing input, output, tool calls, etc.
      Returns:
      evaluation result