类class agentscope.rag

ApproxTokenChunker

class ApproxTokenChunker(ChunkerBase)

源码Source agentscope/rag/_chunker/_approx_token_chunker.py:20–240

Approx 令牌切分器，把私有文档转化为可检索的知识。位于 agentscope.rag 包，继承自 ChunkerBase。核心成员包括 chunk、chunk_size、overlap。

Text sections are sliced into pieces of at most chunk_size approximate tokens, with overlap approximate tokens shared between two consecutive pieces. The token count of a string is approximated as len(text.encode("utf-8")) // 4, so no tokenizer dependency is required.

Sections carrying a :class:~agentscope.message.DataBlock (images, video, etc.) are passed through unchanged as a single chunk. Empty or whitespace-only text produces no chunk.

嵌套类摘要Nested Class Summary

类Class	类型Kind	说明Description
`Parameters`	类class	参数，把私有文档转化为可检索的知识。 The tunable parameters of the approximate-token chunker.

构造器摘要Constructor Summary

构造器Constructor	说明Description
`def __init__(parameters: 'ApproxTokenChunker.Parameters \| None' = None, **kwargs: Any) -> None`	初始化 `ApproxTokenChunker` 实例。 Initialize the approx token chunker.

属性摘要Property Summary

属性Property	说明Description
`chunk_size`	获取分块大小。 The maximum number of approximate tokens per chunk.
`overlap`	读取 `overlap` 属性。 The number of approximate tokens shared between chunks.

属性摘要Attribute Summary

属性Attribute	类型Type	说明Description	值Value
`chunker_type`	—	切分器类型字段。	`'approx_token'`

方法摘要Method Summary

方法Method	说明Description
`async def chunk(sections: list[Section]) -> list[Chunk]`async	设置分块。 Chunk the input sections into smaller chunks based on an approx token counting strategy.

构造器详情Constructor Detail

ApproxTokenChunker

def __init__(parameters: 'ApproxTokenChunker.Parameters | None' = None, **kwargs: Any) -> None

初始化 ApproxTokenChunker 实例。

Initialize the approx token chunker.

参数Parameters

名称Name	类型Type	说明Description
`parameters`默认default `None`	'ApproxTokenChunker.Parameters \| None'	参数（类型 `'ApproxTokenChunker.Parameters \| None'`）。The chunker parameters (`chunk_size` and `overlap`). Defaults to `Parameters()` when not provided.
`kwargs`	Any	关键字参数（类型 `Any`）。The kwargs.

异常Raises

可能抛出 `Exception`。`TypeError`:
当If`parameters`是不`Parameters`实例时抛出 `Exception`。If `parameters` is not a `Parameters` instance.
可能抛出 `Exception`。`ValueError`:
可能抛出 `Exception`。If `chunk_size` is not positive, or `overlap` is
可能抛出 `Exception`。negative or not smaller than `chunk_size`.

源码Source agentscope/rag/_chunker/_approx_token_chunker.py:69–117

属性详情Attribute Detail

chunker_type

 = 'approx_token'

切分器类型字段。

The chunker type field.

源码Source agentscope/rag/_chunker/_approx_token_chunker.py:37

属性详情Property Detail

chunk_size

def chunk_size() -> int

获取分块大小。

The maximum number of approximate tokens per chunk.

返回Returns int

返回数值结果。Returns a numeric result.

源码Source agentscope/rag/_chunker/_approx_token_chunker.py:120–122

overlap

def overlap() -> int

读取 overlap 属性。

The number of approximate tokens shared between chunks.

返回Returns int

返回数值结果。Returns a numeric result.

源码Source agentscope/rag/_chunker/_approx_token_chunker.py:125–127

方法详情Method Detail

chunk

async def chunk(sections: list[Section]) -> list[Chunk]

async

设置分块。

Chunk the input sections into smaller chunks based on an approx token counting strategy.

参数Parameters

名称Name	类型Type	说明Description
`sections`	list[Section]	sections（类型 `list[Section]`）。A list of sections to chunk.

返回Returns list[Chunk]

返回分块列表。`list[Chunk]`:

源码Source agentscope/rag/_chunker/_approx_token_chunker.py:129–172

嵌套类详情Nested Class Detail

Parameters

class Parameters

参数，把私有文档转化为可检索的知识。位于 agentscope.rag 包，继承自 ChunkerBase.Parameters。

The tunable parameters of the approximate-token chunker. Declared in agentscope.rag. Extends ChunkerBase.Parameters.