TokenCounterUtil

Utility class for estimating token count in messages.

This class provides methods to estimate the number of input tokens that would be consumed when sending messages to an LLM. The estimation uses a character-based approximation that works reasonably well for both English and Chinese text.

Token estimation strategy:

• Text content: ~1 token per 2-4 characters (varies by language)

• Thinking content: Uses the same character-based estimate as text content

• Tool calls: Includes tool name, parameters, and structure overhead

• Tool results: Includes output content and structure overhead

• Message structure: Role, name, and formatting overhead