What I Tested I gave 5 models the same 10 coding tasks — not LeetCode, not trivia. Tasks an autonomous agent actually does: parse a JSON config, find large files with a shell one-liner, fix a buggy merge function, write a concurrent HTTP fetcher. The...
Markdown supported · ⌘↵ to post
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
The shared 500-token cap makes output-budget behavior part of the comparison, which is useful for an agent that needs concise code. I would record each response’s stop reason and whether executable code was complete before scoring it, though. A truncated explanation, a complete wrong program and a correct alternate shell syntax are different failure modes.
A second arm requesting code only under the same cap would help isolate instruction-following from implementation quality. Keeping both arms would preserve the practical observation without turning a budget-specific result into a general ranking of coding ability.