20 个真实高频事故，练出排障直觉20 real high-frequency incidents to build debugging intuition

现象Symptom

• 同一会话持续调用同一个工具，参数几乎不变；回复永远不结束或达到迭代上限。One session keeps calling the same tool with nearly identical arguments and never finishes until the iteration cap.
• 日志里反复出现同一个 tool call id 或相同工具名，token 用量线性上涨。Logs repeat the same tool name or call id while token usage grows linearly.

根因Root cause

工具失败时返回了模糊或空结果，模型无法判断是否完成；同时没有迭代上限和重复调用检测，形成“调用—失败—再调用”的循环。The tool returned an ambiguous or empty result, so the model could not tell whether the task was done; with no iteration cap or loop detection, it retried forever.

定位Diagnosis

1. 按会话统计工具调用次数与参数哈希，确认是否重复。Count tool calls per session and hash arguments to confirm repetition.
2. 检查工具错误是否被包装成自然语言但丢失了失败状态。Check whether tool failures are wrapped as prose and lose the failure state.
3. 查看是否触发 ExceedMaxItersEvent，以及最大迭代次数配置。Inspect ExceedMaxItersEvent and the configured max iterations.

修复Fix

1. 工具返回结构化错误（错误码、是否可重试、已完成步骤），而不是空字符串。Return structured errors (code, retryable flag, completed steps) instead of empty strings.
2. 设置 ReActConfig 的迭代上限与 token 预算，超限时返回可解释的终止消息。Set iteration and token budgets in ReActConfig and return an explainable termination message.
3. 对相同工具 + 相同参数做短窗口去重，第二次调用直接返回缓存或失败提示。Deduplicate identical tool + arguments within a short window and return cache or a failure hint.

预防Prevention

把“工具必须幂等、可超时、返回结构化错误”写进工具开发规范，并在网关层设置全局预算告警。Make idempotency, timeouts and structured errors part of the tool contract, and add global budget alerts at the gateway.

复现 / 修复示意Reproduction / fix sketch

async def search_orders(query: str) -> dict:
    rows = await db.fetch(query)
    if not rows:
        return {"ok": False, "code": "NO_MATCH", "retryable": False,
                "hint": "No orders matched; do not retry with the same query."}
    return {"ok": True, "rows": rows[:20]}

现象Symptom

• 模型厂商返回 invalid tool result / missing tool_call_id 一类 400 错误。The vendor returns 400 errors such as invalid tool result or missing tool_call_id.
• 换一家模型或切换 Formatter 后开始出现，单测却全部通过。It starts after switching providers or formatters, while unit tests still pass.

根因Root cause

工具结果没有携带模型原始调用的 id，或不同 Formatter 对 id 的命名、嵌套位置不一致，导致结果无法与调用关联。Tool results do not carry the original call id, or formatters disagree on its name or nesting, so results cannot be correlated.

定位Diagnosis

1. 打印 Formatter 产出的最终请求体，逐条核对 tool call 与 tool result 的 id。Print the formatter-produced request body and match tool calls with results by id.
2. 确认同一会话只使用一个厂商对应的 Formatter。Ensure one session uses the formatter for its provider.
3. 检查自定义工具包装是否重建了 ToolResultBlock 而丢掉 id。Check whether custom tool wrappers rebuild ToolResultBlock and drop the id.

修复Fix

1. 从 ToolCallBlock 原样传递 id 与工具名到 ToolResultBlock。Copy the id and tool name from ToolCallBlock into ToolResultBlock unchanged.
2. 为每个厂商固定 Formatter，不在运行中混用。Pin one formatter per provider and never mix them at runtime.
3. 在工具执行层加断言：结果必须带 id，否则直接失败而不是发送给模型。Assert at the execution layer that results carry an id; fail fast instead of sending invalid payloads.

预防Prevention

在集成测试里对每家模型做一次完整的工具调用回环，验证 id 能原样回填。Add a full tool-call round trip per provider in integration tests and verify ids survive unchanged.

复现 / 修复示意Reproduction / fix sketch

result = ToolResultBlock(
    id=call.id,          # never regenerate
    name=call.name,
    output=output,
)

现象Symptom

• 前端先逐字显示，结束后整段文字又出现一次。The UI shows text token by token, then the whole message appears again at the end.
• 工具调用和思考块顺序错乱，或结束事件后仍有增量。Tool calls and thinking blocks arrive out of order, or deltas continue after the end event.

根因Root cause

前端同时消费增量事件和最终消息，却没有按块 id 合并；或者后端把增量与最终消息都当作新消息推送。The UI consumes both delta events and the final message without merging by block id, or the backend pushes both as new messages.

定位Diagnosis

1. 记录事件序列，确认 Start/Delta/End 是否成对。Record the event sequence and verify Start/Delta/End triplets.
2. 检查前端是否在 ReplyEndEvent 后再次 append 最终消息。Check whether the UI appends the final message again after ReplyEndEvent.
3. 确认同一内容块只对应一个渲染槽位。Ensure one content block maps to exactly one render slot.

修复Fix

1. 按块 id 建立渲染状态机：Start 建槽、Delta 追加、End 冻结。Build a per-block state machine: Start creates a slot, Delta appends, End freezes.
2. 最终消息只用于校准或补齐缺失块，不重复追加。Use the final message only to reconcile or fill missing blocks, never to append duplicates.
3. 在服务端确保增量事件与最终消息来自同一次 reply。Guarantee server-side that deltas and the final message belong to the same reply.

预防Prevention

把事件序列写入回放测试，覆盖文本、思考、工具调用三种块。Add replay tests over the event sequence covering text, thinking and tool-call blocks.

现象Symptom

• 会话进行到几十轮后开始报 context length exceeded。After dozens of turns the session starts failing with context length exceeded.
• 重试无效，缩短输入后才恢复；成本和延迟同步飙升。Retries do not help until the input is shortened, while cost and latency spike.

根因Root cause

历史消息无上限增长，工具结果和检索内容被原样塞回上下文；没有压缩、摘要或截断策略。History grows without bound and tool results or retrieved content are replayed verbatim, with no compression, summary or truncation.

定位Diagnosis

1. 按会话统计消息数与 token 占比，找出增长最快的来源。Measure message counts and token share per session to find the fastest-growing source.
2. 检查工具结果是否远超必要长度。Check whether tool outputs are far larger than necessary.
3. 确认是否在每轮都重新注入完整知识库内容。Verify whether the full knowledge base is re-injected every turn.

修复Fix

1. 调用 compress_context 或摘要中间件压缩历史，保留任务目标与未完成事项。Use compress_context or summary middleware to keep goals and open items while shrinking history.
2. 工具结果做截断、分页或引用化，只把必要片段放回上下文。Truncate, paginate or reference tool results and only feed necessary excerpts back.
3. 为每次请求设置 token 预算，超出时优先丢弃低价值历史。Set a per-request token budget and drop low-value history first when exceeded.

预防Prevention

上线前用最长会话压测，并对上下文占用建立监控与告警。Load-test with the longest realistic sessions and monitor context occupancy.

复现 / 修复示意Reproduction / fix sketch

if estimate_tokens(msgs) > budget * 0.8:
    msgs = await agent.compress_context(msgs, keep_goals=True)

现象Symptom

• 模型开始限流后，所有实例同时重试，错误率迅速扩散到其他接口。Once the model rate-limits, every instance retries at once and errors spread to other endpoints.
• 队列积压、线程池耗尽，最终连健康检查都超时。Queues back up, thread pools saturate and even health checks time out.

根因Root cause

重试没有指数退避和抖动，也没有并发隔离；单个模型故障被放大成全站故障。Retries lack exponential backoff, jitter and concurrency isolation, amplifying one model failure into a site-wide outage.

定位Diagnosis

1. 统计重试次数与 429 比例，确认是否存在同步重试尖峰。Measure retries and 429 ratios to confirm synchronized retry spikes.
2. 检查模型调用是否与业务请求共享线程池。Check whether model calls share a thread pool with business requests.
3. 确认超时时间是否大于上游限流窗口。Verify timeouts exceed the upstream rate-limit window.

修复Fix

1. 在 ExecutionConfig 中设置有限重试、指数退避与抖动。Configure bounded retries with exponential backoff and jitter in ExecutionConfig.
2. 为模型调用使用独立线程池/信号量，并加熔断与队列上限。Isolate model calls with dedicated pools or semaphores plus circuit breakers and queue limits.
3. 过载时返回降级结果或排队提示，而不是继续放大重试。Degrade or queue with a user-facing message instead of amplifying retries.

预防Prevention

对每家模型设置配额与并发上限，并用混沌测试验证限流场景。Set quotas and concurrency caps per model and chaos-test rate-limit scenarios.

复现 / 修复示意Reproduction / fix sketch

ExecutionConfig.builder()
    .timeout(Duration.ofSeconds(30))
    .maxAttempts(3)
    .initialBackoff(Duration.ofMillis(300))
    .backoffMultiplier(2.0)
    .build();

现象Symptom

• 知识库文档明明存在，查询却始终返回空或极低相似度。Documents exist but queries return nothing or extremely low similarity.
• 日志里出现向量维度不一致或 collection schema 不匹配。Logs show vector dimension mismatches or collection schema errors.

根因Root cause

新 embedding 模型维度不同，旧索引仍保存旧模型的向量；或者 collection 未按新维度重建。The new embedding model has a different dimension while the index still holds old vectors, or the collection was not recreated for the new dimension.

定位Diagnosis

1. 对比 embedding 模型输出维度与 collection 定义。Compare the embedding dimension with the collection definition.
2. 抽样查询向量与库内向量做距离检查。Sample query vectors and compute distances against stored vectors.
3. 确认索引写入与查询使用的是同一个模型版本。Verify indexing and querying use the same model version.

修复Fix

1. 新建 collection 并按新模型重建全部文档向量。Create a new collection and re-embed all documents with the new model.
2. 在元数据里记录 embedding 模型与版本，查询前校验。Record embedding model and version in metadata and validate before querying.
3. 切换期间双写或灰度，验证召回后再切流量。Dual-write or canary during migration, then shift traffic after recall is verified.

预防Prevention

把 embedding 模型版本作为索引的一部分，任何变更都走重建流程。Treat the embedding model version as part of the index and always rebuild on change.

现象Symptom

• 知识库页面已经更新，但回答和引用仍是旧内容。The knowledge base page is updated, but answers and citations still show old content.
• 同一问题在不同时间得到互相矛盾的答案。The same question yields contradictory answers at different times.

根因Root cause

更新文档时只插入新向量，没有删除旧向量；或者删除按文档 id 未覆盖所有块。Updates insert new vectors without deleting old ones, or deletion by document id misses some chunks.

定位Diagnosis

1. 按文档 id 列出所有块，比较创建时间与版本。List all chunks by document id and compare creation time and version.
2. 检查搜索结果中的来源版本是否一致。Check whether source versions in search results are consistent.
3. 确认删除逻辑覆盖全部 chunk 而不是仅主记录。Verify deletion covers every chunk, not just the main record.

修复Fix

1. 以文档 id + 版本写入，更新时先删旧版本再写新版本。Write by document id plus version and delete the old version before writing the new one.
2. 搜索时按版本过滤，只返回当前有效版本。Filter by version at search time and return only the active version.
3. 回答强制附带来源与版本，便于发现过期内容。Require source and version citations so stale content is visible.

预防Prevention

把删除、版本和引用纳入知识库写入流水线的验收条件。Make deletion, versioning and citations acceptance criteria for the ingestion pipeline.

现象Symptom

• 同一用户刷新页面后上下文错乱，历史消息时而出现时而消失。After a refresh the same user's context is scrambled and history appears and disappears.
• 扩容到多实例后问题明显，单实例无法复现。The problem appears only with multiple instances and cannot be reproduced on one.

根因Root cause

状态保存在 Agent 实例字段或本地文件，负载均衡把请求发到不同实例；并发写入又是最后写入覆盖。State lives in agent fields or local files while the load balancer hits different instances, and concurrent writes are last-write-wins.

定位Diagnosis

1. 确认状态存储位置与实例本地性。Check where state is stored and whether it is instance-local.
2. 检查会话是否固定路由到同一实例。Check whether sessions are pinned to one instance.
3. 查看是否存在版本冲突异常或静默覆盖。Look for version-conflict exceptions or silent overwrites.

修复Fix

1. 把 AgentState 落到共享存储，并按 user_id + session_id 读取。Persist AgentState in shared storage keyed by user_id + session_id.
2. 使用版本号或 saveIfVersion 做乐观锁，冲突时重试或合并。Use versions or saveIfVersion for optimistic locking and retry or merge on conflict.
3. 保证同一会话的写操作串行化。Serialize writes for the same session.

预防Prevention

上线前做多实例并发回放测试，监控冲突率与状态加载延迟。Run multi-instance concurrent replay tests and monitor conflict rates and load latency.

现象Symptom

• 工具调用前延迟固定增加数百毫秒，MCP 服务端连接数暴涨。Every tool call adds hundreds of milliseconds and MCP server connections explode.
• 高峰期出现端口耗尽、连接被拒或子进程过多。At peak, ports exhaust, connections are refused or too many subprocesses appear.

根因Root cause

在每次工具调用或每个请求中创建 MCPClient，连接没有复用，stdio 还反复拉起子进程。A new MCPClient is created per tool call or request, connections are not reused, and stdio repeatedly spawns subprocesses.

定位Diagnosis

1. 统计 MCP 连接建立次数与工具调用次数是否接近。Compare MCP connection counts with tool-call counts.
2. 观察服务端连接数与子进程数。Watch server connection and subprocess counts.
3. 检查 MCPClient 的作用域是否在请求内部。Check whether MCPClient is scoped inside a request.

修复Fix

1. 在应用启动时连接一次，作为长生命周期单例复用。Connect once at startup and reuse the client as a long-lived singleton.
2. 为连接设置健康检查、重连退避和最大并发。Add health checks, reconnect backoff and concurrency caps.
3. 应用关闭时统一 close，释放子进程与网络连接。Close all clients on shutdown to release subprocesses and sockets.

预防Prevention

把 MCP 连接纳入资源池管理，并对连接数建立告警。Manage MCP connections as a resource pool and alert on connection counts.

现象Symptom

• 模型调用了预期之外的工具实现，参数含义完全不同。The model invokes an unexpected implementation whose arguments mean something else.
• 同一工具名在两个 MCP 服务或本地都存在，行为随机变化。The same tool name exists locally and in two MCP servers, and behavior changes unpredictably.

根因Root cause

多个来源注册了同名工具，Toolkit 按注册顺序覆盖；没有命名空间或白名单。Multiple sources register tools with the same name and the toolkit resolves by registration order, without namespacing or allowlists.

定位Diagnosis

1. 列出 Toolkit 中所有工具名与来源。List every tool name and source in the toolkit.
2. 比对模型实际调用与期望实现。Compare the actual invocation with the intended implementation.
3. 检查是否存在重复注册或静默覆盖。Check for duplicate registrations or silent shadowing.

修复Fix

1. 为 MCP 工具添加服务前缀，形成唯一名称。Prefix MCP tools with their server name to make names unique.
2. 注册时检测重名并显式报错，而不是覆盖。Detect duplicate names at registration and fail loudly instead of overwriting.
3. 按业务域使用 ToolGroup 和白名单限制可调用集合。Use tool groups and allowlists per business domain.

预防Prevention

把工具命名规范与冲突检测加入启动自检。Add naming conventions and conflict detection to startup self-checks.

现象Symptom

• Agent 执行了 rm -rf 或覆盖关键文件，造成数据丢失。The agent runs rm -rf or overwrites critical files, causing data loss.
• 日志显示工具调用没有经过任何确认或拒绝。Logs show tool calls passed without confirmation or denial.

根因Root cause

生产环境使用了绕过权限的模式，或规则过于宽泛；危险工具没有人工确认和路径白名单。Production bypasses permission checks or uses overly broad rules, and dangerous tools lack human confirmation and path allowlists.

定位Diagnosis

1. 检查 PermissionMode 与允许/拒绝规则。Inspect PermissionMode and allow/deny rules.
2. 确认危险工具是否声明了只读与危险路径。Check whether dangerous tools declare read-only and dangerous-path metadata.
3. 回放请求，查看权限决策链路。Replay the request and inspect the permission decision chain.

修复Fix

1. 默认拒绝，仅对必要工具与路径开白名单。Default deny and allowlist only necessary tools and paths.
2. 对写文件、执行命令、删除等操作强制人工确认。Require human confirmation for writes, command execution and deletion.
3. 在沙箱中执行，限制可访问的宿主路径。Run in a sandbox with restricted host paths.

预防Prevention

把权限配置纳入发布审计，危险工具变更必须走评审。Audit permission config on release and require review for dangerous tool changes.

现象Symptom

• 沙箱内代码可以读写宿主项目文件或云凭证。Code inside the sandbox can read and write host project files or cloud credentials.
• 一次提示词注入就能修改宿主配置或窃取密钥。A single prompt injection can modify host config or steal secrets.

根因Root cause

启动沙箱时把宿主目录、Docker socket 或云凭证直接挂载进去，隔离边界被主动打开。The sandbox mounts host directories, the Docker socket or cloud credentials, deliberately opening the isolation boundary.

定位Diagnosis

1. 列出沙箱挂载点、环境变量与网络权限。List sandbox mounts, environment variables and network permissions.
2. 检查是否可访问 Docker socket 或元数据服务。Check access to the Docker socket or metadata service.
3. 确认工作区后端与资源限制配置。Review the workspace backend and resource limits.

修复Fix

1. 只挂载任务所需的临时目录，默认拒绝宿主路径。Mount only task-scoped temporary directories and deny host paths by default.
2. 禁止 Docker socket、云元数据地址和宿主网络。Block the Docker socket, cloud metadata endpoints and host networking.
3. 为 CPU、内存、磁盘与执行时长设置上限并强制回收。Cap CPU, memory, disk and duration and enforce reclamation.

预防Prevention

把沙箱配置做成不可变模板，并在 CI 中检查挂载与网络策略。Make sandbox config an immutable template and check mounts and network policy in CI.

现象Symptom

• 同一句话机器人回复两次或多次。The bot replies twice or more to one user message.
• 渠道平台侧显示回调超时，随后重复推送同一事件。The platform reports callback timeouts and re-delivers the same event.

根因Root cause

回调处理同步等待模型生成，超过平台超时；同时没有按事件 id 去重，平台的每次重推都触发新回复。The callback waits synchronously for model generation and exceeds platform timeouts, while missing event-id deduplication lets every retry trigger a new reply.

定位Diagnosis

1. 核对平台回调超时时间与接口耗时。Compare platform callback timeouts with endpoint latency.
2. 统计同一 event id 的处理次数。Count processing attempts per event id.
3. 检查是否缺少验签与幂等键。Check for missing signature verification and idempotency keys.

修复Fix

1. 回调只做验签、去重和入队，立即 ACK。Verify, deduplicate and enqueue in the callback, then ACK immediately.
2. 异步消费消息并生成回复，用 event id 作为幂等键。Consume asynchronously and use the event id as the idempotency key.
3. 把发送回复与消费进度绑定，失败可安全重试。Bind reply delivery to consumption progress so retries are safe.

预防Prevention

对所有渠道统一实现验签、去重、快速 ACK 和异步处理四件套。Implement signature verification, deduplication, fast ACK and async processing uniformly for every channel.

现象Symptom

• 任务在多个 Agent 之间来回转交，token 消耗快速上涨。The task bounces between agents while token consumption rises fast.
• 每个 Agent 都认为应该由另一个 Agent 处理。Every agent believes another agent should handle it.

根因Root cause

角色边界模糊，且委派没有深度、预算和白名单限制；子智能体之间可以任意互相调用。Role boundaries are vague and delegation has no depth, budget or allowlist limits, so subagents can call each other freely.

定位Diagnosis

1. 画出实际委派图，统计每个 Agent 的调用次数与耗时。Draw the actual delegation graph and measure calls and latency per agent.
2. 检查是否存在 A→B→A 的循环。Look for A→B→A cycles.
3. 确认委派是否携带验收标准与终止条件。Check whether delegation carries acceptance criteria and termination conditions.

修复Fix

1. 明确每个角色的负责范围与交付物，设置唯一负责人。Define each role's scope and deliverable with a single owner.
2. 限制委派深度、总预算和可调用 Agent 白名单。Limit delegation depth, total budget and the callable agent allowlist.
3. 子任务返回结构化结果，未达成验收标准时由父级决定重试或终止。Return structured results and let the parent decide retry or termination against acceptance criteria.

预防Prevention

对多智能体任务做预算与深度硬限制，并监控委派图深度。Enforce hard budget and depth limits and monitor delegation-graph depth.

现象Symptom

• 用户插话后，机器人继续说完上一段，甚至同时播放两段音频。After the user interrupts, the bot finishes the previous reply or plays two audio streams at once.
• 文本已经停止，但扬声器还有声音。Text stops while the speaker keeps playing.

根因Root cause

检测到语音活动后没有取消模型响应，也没有清空传输层待播缓冲区。Voice activity is detected but the model response is not cancelled and the transport's pending audio buffer is not cleared.

定位Diagnosis

1. 记录 VAD 事件与取消调用是否成对出现。Check whether VAD events are paired with cancellation calls.
2. 检查传输层缓冲区在打断后是否清空。Verify the transport buffer is cleared after barge-in.
3. 确认新轮次是否等待旧响应结束。Check whether the new turn waits for the old response to finish.

修复Fix

1. VAD 判定打断时立即 cancel_response。Call cancel_response immediately when VAD detects barge-in.
2. 清空待播音频并丢弃旧响应后续增量。Clear pending audio and discard later deltas from the old response.
3. 新轮次使用独立的响应 id，避免旧事件污染。Use a new response id per turn so old events cannot leak in.

预防Prevention

把打断作为实时链路的必测用例，覆盖弱网和连续插话。Make barge-in a mandatory realtime test case covering weak networks and repeated interruptions.

现象Symptom

• 大部分请求正常，少量返回多段 JSON、缺少字段或夹带解释文字。Most requests work, but some return multiple JSON fragments, missing fields or extra prose.
• 重试后有时成功，有时仍失败。Retries sometimes succeed and sometimes fail again.

根因Root cause

只靠提示词要求 JSON，没有使用模型原生结构化输出；解析失败后也没有 schema 校验和纠正策略。The prompt alone requests JSON without native structured output, and there is no schema validation or repair strategy on failure.

定位Diagnosis

1. 统计失败样本的原始输出，确认是格式还是字段问题。Inspect raw outputs from failed samples to distinguish format from field issues.
2. 检查模型是否支持原生结构化输出。Check whether the model supports native structured output.
3. 确认解析异常是否被吞掉并转成空结果。Verify parse errors are not swallowed into empty results.

修复Fix

1. 优先使用原生结构化输出与严格 schema。Prefer native structured output with a strict schema.
2. 解析失败时携带错误信息进行一次纠正重试，仍失败则降级。On parse failure, retry once with the error message, then degrade.
3. 对必填字段做运行时校验，不信任模型输出。Validate required fields at runtime and never trust model output blindly.

预防Prevention

为结构化输出建立黄金样本回归集，覆盖边界字段。Maintain a golden-sample regression set for structured output covering edge fields.

现象Symptom

• 用户投诉回答错误，但日志只有最终文本。Users report wrong answers but logs only contain the final text.
• 无法判断是检索、模型、工具还是权限环节出了问题。You cannot tell whether retrieval, the model, a tool or permissions failed.

根因Root cause

没有统一 trace id，也没有记录模型调用、工具执行、重试和状态变更。There is no shared trace id and no record of model calls, tool executions, retries or state changes.

定位Diagnosis

1. 检查日志中是否存在请求级关联 id。Check logs for a request-level correlation id.
2. 确认模型与工具是否有开始/结束事件。Verify start/end events for model calls and tool executions.
3. 核对是否能还原一次完整回复的步骤序列。Check whether the full step sequence of one reply can be reconstructed.

修复Fix

1. 在入口生成 trace id，贯穿模型、工具、子智能体和渠道。Generate a trace id at the entry and propagate it through models, tools, subagents and channels.
2. 接入 TracingMiddleware 或 Java Hook，记录每个阶段输入、输出、耗时与错误。Use TracingMiddleware or Java hooks to record stage inputs, outputs, latency and errors.
3. 把关键事件落到可检索的日志或追踪系统，并保留脱敏后的请求快照。Ship key events to searchable logs or tracing systems with redacted request snapshots.

预防Prevention

把可观测性作为上线门槛：没有 trace 的调用路径不允许发布。Make observability a release gate: no untraced call path ships.

复现 / 修复示意Reproduction / fix sketch

class TraceMiddleware(MiddlewareBase):
    async def on_model_call(self, ctx, next):
        with tracer.start_as_current_span("model.call") as span:
            span.set_attribute("model", ctx.model_name)
            return await next(ctx)

现象Symptom

• 少量并发就把服务拖慢，线程 CPU 不高但吞吐极低。A little concurrency slows the service drastically; CPU is low but throughput collapses.
• 日志显示事件循环线程长时间停在某个工具调用。Logs show event-loop threads stuck inside a tool call for a long time.

根因Root cause

在 Reactor 事件循环中执行阻塞 IO、sleep 或同步 HTTP 调用，占住了非阻塞线程。Blocking IO, sleep or synchronous HTTP runs on the Reactor event loop and occupies non-blocking threads.

定位Diagnosis

1. 抓取线程栈，确认阻塞发生在 Reactor 线程。Capture thread dumps to confirm blocking on Reactor threads.
2. 检查工具是否声明为异步或运行在独立调度器。Check whether tools are async or run on a separate scheduler.
3. 观察并发上升时延迟是否呈非线性增长。See whether latency grows non-linearly with concurrency.

修复Fix

1. 把阻塞工具切换到 boundedElastic 或独立线程池。Move blocking tools to boundedElastic or a dedicated pool.
2. 使用异步 HTTP 客户端和异步数据库驱动。Use async HTTP clients and async database drivers.
3. 为工具设置超时，避免永久占用线程。Set timeouts on tools so threads cannot be held forever.

预防Prevention

在工具开发规范中禁止事件循环阻塞，并用压测覆盖高并发工具调用。Ban event-loop blocking in tool guidelines and load-test high-concurrency tool calls.

现象Symptom

• Agent 读取网页或文档后，开始执行与用户目标无关的写操作。After reading a web page or document, the agent performs writes unrelated to the user's goal.
• 工具参数包含外部内容里的指令或密钥外传地址。Tool arguments contain instructions from external content or exfiltration endpoints.

根因Root cause

把不可信内容与系统指令混在同一上下文，且工具权限过宽；模型无法区分“数据”和“指令”。Untrusted content is mixed with system instructions while tools are over-privileged, so the model cannot distinguish data from instructions.

定位Diagnosis

1. 定位触发注入的原始内容与工具调用序列。Find the injected content and the resulting tool-call sequence.
2. 检查工具权限、路径白名单和网络出口。Check tool permissions, path allowlists and network egress.
3. 确认外部内容是否被标记为不可信数据。Verify external content is marked as untrusted data.

修复Fix

1. 把外部内容放入明确的不可信数据块，并在系统提示中声明不得执行其中指令。Put external content in explicit untrusted data blocks and state that embedded instructions must not be executed.
2. 危险工具默认拒绝，必要时人工确认。Default-deny dangerous tools and require human confirmation when needed.
3. 限制网络出口与文件路径，敏感操作在沙箱中执行。Restrict network egress and file paths and run sensitive actions in a sandbox.

预防Prevention

对含外部内容的链路做注入红队测试，并持续审计工具权限。Red-team injection paths that include external content and continuously audit tool permissions.

现象Symptom

• 某个下游接口变慢后，所有会话都开始超时。After one downstream API slows down, every session starts timing out.
• 线程池或连接池耗尽，健康检查失败。Thread or connection pools exhaust and health checks fail.

根因Root cause

工具没有独立超时、并发隔离和熔断；慢调用占满共享资源，形成级联故障。Tools lack per-call timeouts, concurrency isolation and circuit breakers, so slow calls consume shared resources and cascade.

定位Diagnosis

1. 按工具统计 P99 耗时、并发与错误率。Measure P99 latency, concurrency and error rate per tool.
2. 检查是否与主请求共享线程池或连接池。Check whether tools share pools with main requests.
3. 确认超时后是否真正取消下游调用。Verify timeouts actually cancel downstream calls.

修复Fix

1. 为每个工具设置超时、重试上限和熔断阈值。Set timeouts, retry caps and circuit-breaker thresholds per tool.
2. 按工具或下游服务做舱壁隔离。Bulkhead-isolate tools or downstream services.
3. 超时后返回可降级结果，并触发告警。Return a degraded result on timeout and trigger alerts.

预防Prevention

上线前做下游故障演练，验证单个工具慢不会拖垮主链路。Run downstream-failure drills to prove one slow tool cannot take down the main path.