TokenCounterUtil
public class TokenCounterUtil源码Source
io/agentscope/core/memory/autocontext/TokenCounterUtil.java:41–256令牌 Counter Util，在多次调用之间保存和恢复会话与记忆数据。位于 io.agentscope.core.memory.autocontext 包。核心成员包括 calculateToken。
Utility class for estimating token count in messages.
This class provides methods to estimate the number of input tokens that would be consumed when sending messages to an LLM. The estimation uses a character-based approximation that works reasonably well for both English and Chinese text.
Token estimation strategy:
-
Text content: ~1 token per 2-4 characters (varies by language)
-
Tool calls: Includes tool name, parameters, and structure overhead
-
Tool results: Includes output content and structure overhead
-
Message structure: Role, name, and formatting overhead
方法摘要Method Summary
| 方法Method | 说明Description |
|---|---|
public static int calculateToken(List<Msg> messages)static | 设置该参数。 Calculates the estimated total input tokens for a list of messages. |
方法详情Method Detail
calculateToken
public static int calculateToken(List<Msg> messages)static
设置该参数。
Calculates the estimated total input tokens for a list of messages.
参数Parameters
| 名称Name | 类型Type | 说明Description |
|---|---|---|
messages | List<Msg> | 消息列表（类型 `List`）。the list of messages to estimate tokens for |
int返回数值结果。estimated number of input tokens
源码Source
io/agentscope/core/memory/autocontext/TokenCounterUtil.java:71–83