Package com.tnsai.utils

Class TokenUtils

java.lang.Object

com.tnsai.utils.TokenUtils

public class TokenUtils
extends Object

Utility for estimating token counts.

Heuristic-based (1 token ≈ 4 chars). Not a substitute for a real tokeniser — use only for budgeting, context-window fit checks, and rough UI hints.

Field Summary

Fields

Modifier and Type	Field	Description
static final int	AUDIO_TOKEN_ESTIMATE	Conservative per-audio-part token cost.
static final int	IMAGE_TOKEN_ESTIMATE	Conservative per-image token cost.

Constructor Summary

Constructors

Constructor	Description
TokenUtils()

Method Summary

Modifier and Type	Method	Description
static int	estimateHistoryTokens(List<Map<String,Object>> history)	Estimates total tokens in a conversation history (rough estimate).
static int	estimateMessageTokens(Map<String,Object> msg)	Estimates tokens for a single message map.
static int	estimateTokens(String text)	Estimates the number of tokens in a text string.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

IMAGE_TOKEN_ESTIMATE

public static final int IMAGE_TOKEN_ESTIMATE

Conservative per-image token cost. Providers vary — OpenAI gpt-4o charges 85-1105 tokens depending on resolution; Anthropic bills ~1500 per image regardless of size. Use the Anthropic-style ceiling so budgeting never under-counts; callers can override by passing a custom estimator if they know their provider's precise formula.

See Also:

• Constant Field Values

AUDIO_TOKEN_ESTIMATE

public static final int AUDIO_TOKEN_ESTIMATE

Conservative per-audio-part token cost. Used when a multi-modal part declares type=audio or type=input_audio. Real tokenisation is provider-specific; this is a placeholder ceiling.

See Also:

• Constant Field Values

Constructor Details

TokenUtils

public TokenUtils()

Method Details

estimateTokens

public static int estimateTokens(String text)

Estimates the number of tokens in a text string. Uses the heuristic: length / 4.

estimateHistoryTokens

public static int estimateHistoryTokens(List<Map<String,Object>> history)

Estimates total tokens in a conversation history (rough estimate).

Recognises structured message content:

• String content → plain text estimate.
• List<Map> content parts → sum of per-part estimates (supports text, image, input_image, audio, input_audio, tool_use, tool_result).

Unknown part types contribute a single-token baseline so they don't silently vanish from the budget.

estimateMessageTokens

public static int estimateMessageTokens(Map<String,Object> msg)

Estimates tokens for a single message map. Exposed so callers who maintain their own history shape can reuse the per-message logic without building a wrapper list.

Parameters:

msg - message map with at minimum a content key

Returns:

estimated token count