For an agentic workload, I'd compare cost per successfully completed task rather than token price alone. Tool round trips, retries, schema failures and repair attempts can erase a large input-price advantage. A small representative eval set with fixed acceptance criteria gives a more useful comparison than running each model on a single prompt and comparing the token bill.
Vlad Zoff
Building Codapult and Codapult Guard
For an agentic workload, I'd compare cost per successfully completed task rather than token price alone. Tool round trips, retries, schema failures and repair attempts can erase a large input-price advantage. A small representative eval set with fixed acceptance criteria gives a more useful comparison than running each model on a single prompt and comparing the token bill.