Package com.tnsai.llm.observability

Class LLMMetrics

java.lang.Object
com.tnsai.llm.observability.LLMMetrics
All Implemented Interfaces:
LLMObserver
public class LLMMetrics extends Object implements LLMObserver
Collects and reports metrics for LLM operations.

LLMMetrics tracks:

  • Request/response counts per provider
  • Token usage (estimated) per provider
  • Latency statistics (avg, p50, p95, p99)
  • Error rates per provider
  • Cost estimates based on token usage

Usage Example


 // Create metrics collector
 LLMMetrics metrics = new LLMMetrics();

 // Wrap your client
 LLMClient observedClient = new ObservableLLMClient(client, metrics);

 // Use the client normally
 observedClient.chat("Hello!");
 observedClient.chat("Another query");

 // Get metrics report
 LLMMetrics.Report report = metrics.getReport();
 System.out.println("Total requests: " + report.totalRequests());
 System.out.println("Total estimated cost: $" + report.totalEstimatedCost());
 System.out.println("Avg latency: " + report.avgLatencyMs() + "ms");

Per-Provider Metrics


 Map<String, LLMMetrics.ProviderMetrics> byProvider = metrics.getMetricsByProvider();
 for (var entry : byProvider.entrySet()) {
     System.out.println(entry.getKey() + ":");
     System.out.println("  Requests: " + entry.getValue().requests());
     System.out.println("  Errors: " + entry.getValue().errors());
     System.out.println("  Avg latency: " + entry.getValue().avgLatencyMs() + "ms");
 }

Integration with Micrometer


 // Export to Micrometer (for Prometheus, etc.)
 MeterRegistry registry = new SimpleMeterRegistry();
 metrics.exportToMicrometer(registry);
See Also:

  • Constructor Details

    • LLMMetrics

      public LLMMetrics()

  • Method Details

    • onRequest

      public void onRequest(com.tnsai.llm.LLMClient client, String message, Optional<String> systemPrompt, Optional<List<Map<String,Object>>> history, Optional<List<Map<String,Object>>> tools)
      Description copied from interface: LLMObserver
      Called before sending a request to the LLM.
      Specified by:
      onRequest in interface LLMObserver
      Parameters:
      client - The LLM client being used
      message - The user message
      systemPrompt - Optional system prompt
      history - Optional conversation history
      tools - Optional tools

    • onResponse

      public void onResponse(com.tnsai.llm.LLMClient client, com.tnsai.llm.ChatResponse response, long latencyMs)
      Description copied from interface: LLMObserver
      Called after receiving a successful response.
      Specified by:
      onResponse in interface LLMObserver
      Parameters:
      client - The LLM client that was used
      response - The response received
      latencyMs - Time taken in milliseconds

    • onError

      public void onError(com.tnsai.llm.LLMClient client, Exception error, long latencyMs)
      Description copied from interface: LLMObserver
      Called when an error occurs.
      Specified by:
      onError in interface LLMObserver
      Parameters:
      client - The LLM client that was used
      error - The exception that occurred
      latencyMs - Time taken before error in milliseconds

    • onStreamChunk

      public void onStreamChunk(com.tnsai.llm.LLMClient client, String chunk, int chunkIndex)
      Description copied from interface: LLMObserver
      Called for each streaming chunk received.
      Specified by:
      onStreamChunk in interface LLMObserver
      Parameters:
      client - The LLM client being used
      chunk - The content chunk
      chunkIndex - The index of this chunk (0-based)

    • onStreamComplete

      public void onStreamComplete(com.tnsai.llm.LLMClient client, int totalChunks, long latencyMs)
      Description copied from interface: LLMObserver
      Called when streaming completes.
      Specified by:
      onStreamComplete in interface LLMObserver
      Parameters:
      client - The LLM client that was used
      totalChunks - Total number of chunks received
      latencyMs - Total time taken in milliseconds

    • onStreamError

      public void onStreamError(com.tnsai.llm.LLMClient client, Exception error, int chunksReceived, long latencyMs)
      Description copied from interface: LLMObserver
      Called when streaming encounters an error.
      Specified by:
      onStreamError in interface LLMObserver
      Parameters:
      client - The LLM client that was used
      error - The exception that occurred
      chunksReceived - Number of chunks received before error
      latencyMs - Time taken before error in milliseconds

    • getReport

      public LLMMetrics.Report getReport()
      Gets a comprehensive metrics report.

    • getMetricsByProvider

      public Map<String,LLMMetrics.ProviderMetrics> getMetricsByProvider()
      Gets metrics per provider.

    • reset

      public void reset()
      Resets all metrics.