Package com.tnsai.quality.observability

Class LlmInstrumentation

java.lang.Object
com.tnsai.quality.observability.LlmInstrumentation
public class LlmInstrumentation extends Object
OpenTelemetry instrumentation for LLM calls following semantic conventions.

This class provides detailed tracing for LLM API calls with attributes that follow the OpenTelemetry GenAI semantic conventions for better interoperability with observability tools like Jaeger, Zipkin, and Grafana Tempo.

Semantic Conventions

This instrumentation follows the OpenTelemetry semantic conventions for Generative AI (GenAI) operations. Key attributes include:

  • gen_ai.system - The LLM provider (openai, anthropic, etc.)
  • gen_ai.request.model - The model name
  • gen_ai.request.max_tokens - Max tokens requested
  • gen_ai.request.temperature - Temperature setting
  • gen_ai.usage.prompt_tokens - Prompt tokens used
  • gen_ai.usage.completion_tokens - Completion tokens generated
  • gen_ai.response.finish_reasons - Finish reason (stop, length, etc.)

Usage Example


 TnsAITelemetry telemetry = TnsAITelemetry.builder()
     .serviceName("my-agent")
     .build();

 LlmInstrumentation llmInstr = new LlmInstrumentation(telemetry.getTracer());

 // Trace an LLM call
 ChatResponse response = llmInstr.traceChat(
     "openai",
     "gpt-4",
     () -> openAIClient.chat(message)
 );

 // Record token usage
 llmInstr.recordTokenUsage(currentSpan, 150, 50);

  • Field Details

    • GEN_AI_SYSTEM

      public static final io.opentelemetry.api.common.AttributeKey<String> GEN_AI_SYSTEM

    • GEN_AI_REQUEST_MODEL

      public static final io.opentelemetry.api.common.AttributeKey<String> GEN_AI_REQUEST_MODEL

    • GEN_AI_REQUEST_MAX_TOKENS

      public static final io.opentelemetry.api.common.AttributeKey<Long> GEN_AI_REQUEST_MAX_TOKENS

    • GEN_AI_REQUEST_TEMPERATURE

      public static final io.opentelemetry.api.common.AttributeKey<Double> GEN_AI_REQUEST_TEMPERATURE

    • GEN_AI_REQUEST_TOP_P

      public static final io.opentelemetry.api.common.AttributeKey<Double> GEN_AI_REQUEST_TOP_P

    • GEN_AI_USAGE_PROMPT_TOKENS

      public static final io.opentelemetry.api.common.AttributeKey<Long> GEN_AI_USAGE_PROMPT_TOKENS

    • GEN_AI_USAGE_COMPLETION_TOKENS

      public static final io.opentelemetry.api.common.AttributeKey<Long> GEN_AI_USAGE_COMPLETION_TOKENS

    • GEN_AI_USAGE_TOTAL_TOKENS

      public static final io.opentelemetry.api.common.AttributeKey<Long> GEN_AI_USAGE_TOTAL_TOKENS

    • GEN_AI_RESPONSE_FINISH_REASONS

      public static final io.opentelemetry.api.common.AttributeKey<String> GEN_AI_RESPONSE_FINISH_REASONS

    • GEN_AI_OPERATION_NAME

      public static final io.opentelemetry.api.common.AttributeKey<String> GEN_AI_OPERATION_NAME

    • TNSAI_AGENT_ID

      public static final io.opentelemetry.api.common.AttributeKey<String> TNSAI_AGENT_ID

    • TNSAI_AGENT_NAME

      public static final io.opentelemetry.api.common.AttributeKey<String> TNSAI_AGENT_NAME

    • TNSAI_MESSAGE_LENGTH

      public static final io.opentelemetry.api.common.AttributeKey<Long> TNSAI_MESSAGE_LENGTH

    • TNSAI_TOOL_USE_ENABLED

      public static final io.opentelemetry.api.common.AttributeKey<Boolean> TNSAI_TOOL_USE_ENABLED

    • TNSAI_TOOL_CALLS_COUNT

      public static final io.opentelemetry.api.common.AttributeKey<Long> TNSAI_TOOL_CALLS_COUNT

  • Constructor Details

    • LlmInstrumentation

      public LlmInstrumentation(io.opentelemetry.api.trace.Tracer tracer)
      Creates a new LlmInstrumentation with the given Tracer.
      Parameters:
      tracer - the OpenTelemetry Tracer instance

  • Method Details

    • traceChat

      public <T> T traceChat(String provider, String model, Supplier<T> operation)
      Traces an LLM chat completion operation.
      Type Parameters:
      T - the return type
      Parameters:
      provider - the LLM provider (e.g., "openai", "anthropic")
      model - the model name
      operation - the LLM operation to execute
      Returns:
      the result of the operation

    • traceChat

      public <T> T traceChat(String provider, String model, Integer maxTokens, Double temperature, Supplier<T> operation)
      Traces an LLM chat completion operation with request parameters.
      Type Parameters:
      T - the return type
      Parameters:
      provider - the LLM provider
      model - the model name
      maxTokens - maximum tokens (null if not specified)
      temperature - temperature setting (null if not specified)
      operation - the LLM operation to execute
      Returns:
      the result of the operation

    • traceStreamChat

      public <T> T traceStreamChat(String provider, String model, Supplier<T> operation)
      Traces an LLM streaming chat operation.
      Type Parameters:
      T - the return type (typically a Stream)
      Parameters:
      provider - the LLM provider
      model - the model name
      operation - the streaming operation to execute
      Returns:
      the result of the operation

    • traceEmbeddings

      public <T> T traceEmbeddings(String provider, String model, int inputCount, Supplier<T> operation)
      Traces an LLM embeddings operation.
      Type Parameters:
      T - the return type
      Parameters:
      provider - the LLM provider
      model - the model name
      inputCount - number of inputs to embed
      operation - the embedding operation to execute
      Returns:
      the result of the operation

    • recordTokenUsage

      public void recordTokenUsage(io.opentelemetry.api.trace.Span span, long promptTokens, long completionTokens)
      Records token usage on a span.
      Parameters:
      span - the span to add attributes to
      promptTokens - number of prompt tokens
      completionTokens - number of completion tokens

    • recordResponse

      public void recordResponse(io.opentelemetry.api.trace.Span span, long promptTokens, long completionTokens, String finishReason)
      Records token usage on a span from a ChatResponse-like object.
      Parameters:
      span - the span to add attributes to
      promptTokens - number of prompt tokens
      completionTokens - number of completion tokens
      finishReason - the finish reason (e.g., "stop", "length", "tool_calls")

    • recordToolUse

      public void recordToolUse(io.opentelemetry.api.trace.Span span, int toolCallsCount)
      Records tool use information on a span.
      Parameters:
      span - the span to add attributes to
      toolCallsCount - number of tool calls made

    • recordToolCallEvent

      public void recordToolCallEvent(io.opentelemetry.api.trace.Span span, String toolName, boolean success)
      Adds an event for a tool call within an LLM span.
      Parameters:
      span - the parent span
      toolName - the tool name
      success - whether the tool call succeeded

    • spanBuilder

      public LlmInstrumentation.LlmSpanBuilder spanBuilder(String provider, String model)
      Creates a builder for configuring and starting an LLM span.
      Parameters:
      provider - the LLM provider
      model - the model name
      Returns:
      a new LlmSpanBuilder