The finding that a reasoning model burned the whole output cap on thinking before writing the handoff is a great example of why fake-model evals aren't enough, and I'm taking that into our own eval harness. Keeping the accepted/rejected constraints verbatim and un-summarizable is a smart call. Did you consider also having the check for a constraint surviving a reset run as a deterministic assertion on the handoff text itself, rather than only through the model's last-turn behavior? That would catch drops without needing a real-model run each time.
I'd make undo check each file before restoring it. File tools snapshot before a write and shell_exec changes aren't rolled back, so a turn that edits calc.go with file_edit and then runs gofmt -w . leaves undo holding a snapshot from before both. Restoring it quietly drops the formatter's change to calc.go, while every other file gofmt rewrote stays changed, and the tree ends up half undone. Recording a hash after each file-tool write, and refusing to restore or showing the diff when the current hash differs, turns that into a visible conflict instead of a silent loss.
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Prompt per call down 16.9% looks like a win until calls go up 8.3% and the total only moves 6.8%. I've shipped token cuts on that first number alone, so a hard cap on extra calls is going into my own checks. When compare labeled the lazy-loading run, did the hurt tasks cluster on one kind of work or spread across the set?