Imagine giving an AI coding agent a fairly ordinary ticket: update authentication across a service, repair the tests that break, and open a pull request. The interesting question is no longer whether
Markdown supported · ⌘↵ to post
That’s a fair criticism. The repository, test suite, and retry opportunities can all influence the outcome, so a benchmark needs to account for those factors before drawing conclusions about the models. I agree that the number of attempts and whether agents could revisit failing tests are especially important details. What would you consider the fairest setup for comparing these tools on real-world coding tasks?
Adam Lewis
Product Engineer/Architect navigating the AI revolution
Benchmarks like these need unpicking long before pricing comes into it. A ticket like that one runs on a test suite the agent can read and a repo it can navigate, and whichever model wins there tells me about that repo more than about the work. What did you hold constant across the two runs, and did either of them get a second pass at the failing tests after a human would have stopped?