运维指挥Ops Commander

项目简报Brief

产物契约Artifact contract

先冻结产物形状，再写 Agent。后续里程碑与评测都读取这份契约。Freeze the shape before writing the agent. Every later milestone and evaluation reads this contract.

{
  "incident_id": "INC-2401",
  "stage": "mitigation_pending_approval",
  "evidence": ["runbook:db-pool#step-3", "metric:pool_wait_p99"],
  "proposed_action": "raise pool size to 40",
  "approval": {"status": "pending", "approver": null},
  "idempotency_key": "INC-2401:pool:40",
  "resume_token": "sess-ops-2401"
}

里程碑Milestones

每条验收成立后再勾选里程碑。小时数是预算，不是秒表目标。Check each milestone only when its acceptance holds. Hours are a budget, not a stopwatch target.

1. MCP 接入MCP integration 4h

   通过 stdio 启动 runbook 服务；完成发现、调用、超时与关闭；工具输出按不可信数据处理。

   Start the runbook service over stdio; discover, call, time out and close; treat tool output as untrusted data.

   验收Acceptance

   工具列表与 runbook 服务一致；超时可控；finally 关闭无残留子进程。

   The tool list matches the server; timeouts are bounded; finally closes with no orphan processes.

2. 事故流程Incident flow 5h

   分诊 → 证据 → 缓解方案 → 人工审批 → 执行；破坏性动作必须有审批记录；verifier 决定是否继续。

   Triage → evidence → mitigation → human approval → execute; destructive actions need an approval record; a verifier decides whether to continue.

   验收Acceptance

   演练中所有破坏性动作都有 pending/approved 记录；未批准时不会执行。

   In drills, every destructive action has a pending/approved record; nothing executes before approval.

3. 状态与恢复State and resume 4h

   每个阶段后持久化 AgentState；重启后用同一 session id 恢复；已执行动作靠幂等键去重。

   Persist AgentState after each stage; resume with the same session id after a restart; dedupe executed actions with idempotency keys.

   验收Acceptance

   中途杀掉进程再恢复，流程从原阶段继续；同一缓解动作不会执行两次。

   Kill the process mid-flow and resume from the same stage; the same mitigation does not run twice.

4. 可观测与评测Observability and evaluation 4h

   结构化 trace + 指标；跑 3 个演练（数据库连接池、缓存击穿、发布回滚）；统计安全闸门与恢复成功率。

   Structured traces and metrics; run three drills (DB pool, cache stampede, release rollback); measure safety gates and resume success.

   验收Acceptance

   3 个演练全部通过审批闸门；至少 1 次重启恢复；指标可复算。

   All three drills pass the approval gate; at least one restart-resume; metrics are recomputable.

5. 复盘与交付Postmortem and ship 3h

   写 postmortem：时间线、根因、行动、引用、回滚与未解问题；用费曼方式解释“为什么要保存阶段与幂等键”。

   Write the postmortem: timeline, root cause, actions, citations, rollback and open questions; explain in Feynman style why stage and idempotency keys must persist.

   验收Acceptance

   每条行动可定位到 runbook/指标；README 能让同事复现一次演练。

   Every action maps to a runbook section or metric; a colleague can reproduce one drill from the README.

交付物Deliverables

每项交付物必须真实存在且可复现。完成门槛会检查全部勾选，再要求复盘。Every deliverable must exist on disk and be reproducible. The finish gate checks all boxes, then asks for a reflection.

1. 事故轨迹Incident transcript lab/work/projects/p4/incident.jsonl

   阶段、工具调用、证据、审批状态与恢复点；每行可独立解析。

   Stage, tool calls, evidence, approval state and resume point; each line parses independently.

2. 复盘报告Postmortem lab/work/projects/p4/postmortem.md

   时间线、根因、缓解、引用、未解问题与回滚方案；每条行动带证据。

   Timeline, root cause, mitigation, citations, open questions and rollback; every action is backed by evidence.

3. 恢复状态Resumable state lab/work/projects/p4/state.json

   会话 id、上下文、阶段、已执行动作与幂等键；重启后可继续。

   Session id, context, stage, executed actions and idempotency keys; survives a restart.

4. 运行指标Run metrics lab/work/projects/p4/metrics.json

   每阶段耗时、token、工具调用次数、审批等待与错误；可按事故聚合。

   Per-stage latency, tokens, tool calls, approval wait and errors; aggregate by incident.

5. 演练评测Drill evaluation lab/work/projects/p4/eval.json

   3 个事故演练：全部完成审批闸门；重启恢复至少 1 次；无重复副作用。

   Three drills: all pass the approval gate, at least one restart-resume, and no duplicated side effects.

6. 交付说明Delivery README lab/work/projects/p4/README.md

   MCP 启动方式、环境变量、恢复与审批流程、安全边界。

   MCP launch, environment variables, resume and approval flow, and safety boundaries.

评分表Rubric

宣告完成前诚实自评。低于 80 分意味着该迭代，而不是该交付。Grade yourself honestly before calling it done. Anything below 80 is a signal to iterate, not to ship.

双轨实现Build on your track

交付物不绑定语言。按团队需要选择轨道；完成记录会保存你交付所用的轨道。The deliverable is language-agnostic. Follow the track that fits the team; the finish record stores the track you shipped on.

client = MCPClient(StdioMCPConfig(
    command=sys.executable,
    args=["-m", "runbook_server"],
    cwd=str(runbook_root),
))
try:
    tools = await client.list_tools()
    assert "lookup_runbook" in [tool.name for tool in tools]
    # approval first, execute second, persist after every stage
    state_path.write_text(agent.state.model_dump_json(indent=2), encoding="utf-8")
finally:
    await client.close()

AgentStateStore store = new JsonFileAgentStateStore(
        work.resolve("state-store"));

try (McpClient client = McpClientBuilder.builder()
        .stdio(serverCommand)
        .build()) {
    ReActAgent agent = ReActAgent.builder()
            .name("ops").model(model).toolkit(toolkit)
            .stateStore(store).defaultSessionId("INC-2401")
            .build();
    agent.streamEvents(List.of(Msg.user(alert)), ctx)
         .doOnNext(event -> trace(event.getType()))
         .blockLast();
}

进阶与资源Stretch goals and resources

• 加入“审批拒绝”演练：拒绝后不执行，Agent 给出替代方案并复述风险。

  Add an approval-rejection drill: nothing executes, and the agent offers an alternative and restates the risk.

• 把指标接入真实 tracing（OpenTelemetry），按事故生成火焰图式时间线。

  Wire metrics into real tracing (OpenTelemetry) and render a flame-style incident timeline.