沙箱升级与混沌回归Sandbox upgrade and chaos regression

先说问题The problem

先动手Do it first

先在页面上接线——页面只懂规则。然后跑下面那条判题，那才是真会失败的东西。Wire it on the page — the bench only knows the rules. Then run the check below; that is the part that can actually fail.

create_app(…)

为什么这样设计Why it is built this way

用到的 APIAPIs in play

代码The code

# 1. chaos table: the fault has to be DECLARED retryable to be absorbed
class ChaosModel(ScriptedModel):
    def _get_retryable_exceptions(self):        # base default is () — nothing retries
        return (openai.APIConnectionError, openai.InternalServerError)
# a 5xx of a type nobody declared escapes on attempt 1 -> [RED]

# 2. determinism, with the tape rebuilt every run (mutable blocks, see the trap)
def golden():                                   # not a module-level list
    return [tool_response("lookup", call_id="lookup-1", order_id="A-1001"),
            text_response("order A-1001 is refundable")]
runs = [await trajectory(agent, golden()) for _ in range(3)]
assert len({tuple(r) for r in runs}) == 1
assert "tool_result:success" in runs[0]          # or the equality passes vacuously

# 3. the sandbox swap is one block, and eviction is lazy
app = create_app(**kwargs_for({**DEFAULT_ASSEMBLY, "workspace_manager": "docker_workspace"}))
# ttl expiry is checked inside get_workspace: the old lease is closed, a new one built
# local workdir files survive that; a container or remote lease does not

判题The check

python lab.py run c7

它断言了什么What it asserts

• workspace_manager local -> docker: 7 create_app kwargs, same key set, only workspace_manager differs — the other 6 blocks are untouched
• block docker_workspace on this host -> ValueError: block 'docker_workspace' needs `pip install aiodocker`
• block e2b_workspace on this host -> ValueError: block 'e2b_workspace' needs `pip install e2b`
• isolation grain: PER_AGENT two sessions -> 4bf1218b (shared), PER_SESSION -> 846d9c09/d8228b97 (split), PER_USER two agents -> 48ea3da4 (one tenant-wide sandbox)
• ttl=0 lease: the next get_workspace evicted and rebuilt the sandbox (w-lease -> new object, cache 1); a local workdir keeps its files across eviction, a container/remote lease does not — that is the turn you cannot resume
• chaos table, 4 absorbed / 2 fatal, and every row landed where the framework's own retry rules say it should
• [GREEN] provider 5xx, exception type declared retryable -> 3 attempt(s) for 2 tape step(s), tool still ran, answer intact
• [RED] provider 5xx, exception type NOT declared -> Mystery500: HTTP 500 nobody declared
• [RED] stream ends without is_last=True -> AssertionError: the cut-off text became the answer: 'order A-1001 is ref'
• [GREEN] asyncio.CancelledError mid-model-call -> finished_reason=interrupted, reply 'I notice the interruption. How can', no crash
• [GREEN] tool arguments cut off mid-JSON -> repaired into {'order_id': 'A-1001'} and executed
• [GREEN] tool arguments that are not JSON -> tool ran 0x, tool_result:error handed back to the model
• same script rebuilt per run, 3 replays -> one trajectory ['text', 'hint', 'tool_call:lookup', 'tool_result:success', 'text'], 1 distinct
• replay trap, the reason each replay above rebuilds its tape: hand the same ChatResponse objects to a second run and the agent sees its own finished marker on the tool call — run 1 executed the tool ([{'order_id': 'A-1001'}]), run 2 executed nothing ([]), yet both answered 'order A-1001 is refundable'. A text-only evaluation scores that as two passes while the side effect quietly disappeared
• pass@1 measured 5 ways: 5 runs of ONE task under 40% injected 5xx with max_retries=1 -> 4/5 passed (80%); one run of that task alone says True. A single green run is a coin flip.
• kill -9 during a turn of 24 refunds: 7 landed before death, a second process over the same storage reads status=idle (not running, and nothing detected the death), 10 ledger rows after 3s of waiting, unanswered tool calls in the tail: []

会踩的坑Where it hurts

• 用「跑一次通过」当发布门槛。一次通过只证明这次运气好；把 k 设成 5，你才第一次看到真实确定率。

  Shipping on a single passing run. One pass only proves you got lucky; set k to 5 and you see the real determinism for the first time.

费曼闸门Say it back