Engineer's view. This is an endpoint that returns 200 after writing the wrong value. The handler logged success, and only a look at the row shows the bug. An agent's "done" is that 200: it comes from
Markdown supported · ⌘↵ to post
ThinkingBox finding that 67% of failures ended cleanly lines up with what I saw once I started diffing rows instead of reading traces. And 94% solved at least once vs 13% on all 20 tries is the number I'd show anyone picking a model off a single run. How does the extra-effect check handle writes the agent never made directly, like a database trigger updating a second table?
The comparison you open with, an endpoint that returns 200 after writing the wrong value, is the clearest way I have seen to explain why an agent's done is not evidence. The success signal comes from the same process that made the mistake, so it cannot also be the check.
Reading the changed records back is the right fix, and I would pair it with one more step: write down before the run what those records should look like afterwards. Which rows, which fields, which values, and what must stay untouched. Then the read-back is a comparison against an expected state, rather than the agent inspecting its own work and deciding it looks fine.
We use the same idea when scoping client work: the finish line is written before anyone starts, so done means the same thing to everyone involved. How do you handle records the agent changed that were never in the plan at all?
Elarigent
The 200-after-wrong-value analogy stuck with me — I've seen the same thing in a past API project, where every log said success and only a row diff exposed the bug. Gating the completion tool on a contract of expected writes feels like the missing review step between an agent "doing its best" and a system you can actually trust.