Package com.tnsai.context.assembler

Class TokenEstimator

java.lang.Object
com.tnsai.context.assembler.TokenEstimator
public final class TokenEstimator extends Object
Content-aware token estimator that improves on simple length/4 heuristics.

Accounts for whitespace density, punctuation density, structured content (JSON/XML), and model-specific tokenizer differences to produce more accurate token count estimates without requiring an actual tokenizer library.

Estimation Strategy

  1. Classify content type (natural language, JSON, code, etc.)
  2. Apply content-type-specific chars-per-token ratio
  3. Adjust for whitespace and punctuation density
  4. Apply model-family multiplier

Usage Example


 // Simple estimation
 int tokens = TokenEstimator.estimate("Hello, world!");

 // Model-aware estimation
 int tokens = TokenEstimator.estimate("Hello, world!", "claude-3-opus");

 // Estimate message list
 int tokens = TokenEstimator.estimateMessages(history);
Since:
0.3.0

  • Method Details

    • estimate

      public static int estimate(String text)
      Estimates token count for text using content-aware heuristics.
      Parameters:
      text - text to estimate
      Returns:
      estimated token count, 0 for null/empty

    • estimate

      public static int estimate(String text, String modelName)
      Estimates token count with model-aware adjustment.
      Parameters:
      text - text to estimate
      modelName - model identifier (e.g., "gpt-4", "claude-3-opus"), nullable
      Returns:
      estimated token count

    • estimateMessages

      public static int estimateMessages(List<Map<String,Object>> messages)
      Estimates tokens for a list of chat messages.

      Each message incurs a small overhead for role and delimiter tokens, in addition to the content token count.

      Parameters:
      messages - list of message maps (must contain "content" key)
      Returns:
      total estimated token count

    • estimateMessages

      public static int estimateMessages(List<Map<String,Object>> messages, String modelName)
      Estimates tokens for a list of chat messages with model awareness.
      Parameters:
      messages - list of message maps
      modelName - model identifier, nullable
      Returns:
      total estimated token count

    • estimateMessage

      public static int estimateMessage(Map<String,Object> message, String modelName)
      Estimates tokens for a single chat message.
      Parameters:
      message - message map with "role" and "content" keys
      modelName - model identifier, nullable
      Returns:
      estimated token count including overhead

    • detectContentType

      public static TokenEstimator.ContentType detectContentType(String text)
      Detects the content type of the text.
      Parameters:
      text - text to classify
      Returns:
      detected content type