Package com.tnsai.annotations

Annotation Interface EvalMetric

@Retention(RUNTIME) @Target({METHOD,TYPE}) public @interface EvalMetric
Marks a method or class for evaluation metric collection.

When applied to agent methods, this annotation enables automatic metric collection for evaluation purposes. Metrics are captured during execution and can be used for:

  • Goal completion tracking
  • Tool selection analysis
  • Latency measurements
  • Quality assessments
  • Regression detection

Usage on Methods


 @EvalMetric(name = "goal_completion", category = MetricCategory.GOAL)
 public void onGoalCompleted(String goalId, boolean success) {
     // Method will be tracked for goal completion metrics
 }

Usage on Classes


 @EvalMetric(name = "research_agent", category = MetricCategory.AGENT)
 public class ResearchAgent extends Agent {
     // All agent interactions will be tracked
 }

Metric Categories

CategoryDescriptionExample Metrics
GOALGoal completionsuccess_rate, completion_time
TOOLTool usageselection_accuracy, call_count
LATENCYPerformanceresponse_time, processing_time
QUALITYOutput qualityrelevance, accuracy, coherence
COSTResource usagetoken_count, api_calls
AGENTOverall agentsession_success, efficiency
Author:
TnsAI Team
See Also:

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static enum 
    EvalMetric.MetricCategory
    Categories for evaluation metrics.

  • Required Element Summary

    Required Elements
    Modifier and Type
    Required Element
    Description
    String
    name
    The name of the metric.

  • Optional Element Summary

    Optional Elements
    Modifier and Type
    Optional Element
    Description
    EvalMetric.MetricCategory
    category
    The category of this metric.
    boolean
    critical
    Whether this metric is critical for pass/fail determination.
    String
    description
    Human-readable description of what this metric measures.
    boolean
    realtime
    Whether to track this metric in real-time.
    String[]
    tags
    Tags for grouping and filtering metrics.
    double
    threshold
    Minimum acceptable value for this metric (0.0 to 1.0).

  • Element Details

    • name

      String name
      The name of the metric. Used as identifier in evaluation reports.
      Returns:
      metric name

    • category

      EvalMetric.MetricCategory category
      The category of this metric.
      Returns:
      metric category
      Default:
      CUSTOM

    • description

      String description
      Human-readable description of what this metric measures.
      Returns:
      metric description
      Default:
      ""

    • critical

      boolean critical
      Whether this metric is critical for pass/fail determination. Critical metrics cause evaluation failure if below threshold.
      Returns:
      true if critical
      Default:
      false

    • threshold

      double threshold
      Minimum acceptable value for this metric (0.0 to 1.0). Values below this threshold are considered failures.
      Returns:
      threshold value
      Default:
      0.7

    • realtime

      boolean realtime
      Whether to track this metric in real-time. Real-time metrics are streamed as they are collected.
      Returns:
      true if real-time tracking is enabled
      Default:
      false

    • tags

      String[] tags
      Tags for grouping and filtering metrics.
      Returns:
      array of tags
      Default:
      {}