Your "42 passed in 6.1s" versus "tests pass" example is the clearest definition of done for agents I've come across. When I've asked agents for that kind of evidence, the summaries get shorter and a lot more honest. Did the three-attempts cap fully stop the deleted-test problem, or do you still lean on the reviewer check for that?
Thanks, Kartik, and "shorter and a lot more honest" matches what I've seen too. Honest answer: no, the three-attempts cap on its own doesn't stop it. It only keeps the agent from looping forever, and an agent can still delete or skip a test on its first try. What actually catches it is the combination of the done rule (tests pass and none were deleted or skipped) plus a reviewer subagent that checks the diff for removed test files and new skip markers, so I definitely still lean on the reviewer for that.
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Your "42 passed in 6.1s" versus "tests pass" example is the clearest definition of done for agents I've come across. When I've asked agents for that kind of evidence, the summaries get shorter and a lot more honest. Did the three-attempts cap fully stop the deleted-test problem, or do you still lean on the reviewer check for that?