Cost per successful task is the right unit, but it assumes "successful" means the same thing across models, and none of the benchmarks you're citing really guarantee that. AutomationBench, FrontierCode and DeepSWE are all graded against some verifier, and two models can land the same aggregate score while failing on completely different slices of the task distribution — one nails the common case and blows the tail, the other does the opposite. That matters for the cost math because in production you don't just eat a failure, you escalate it to something more expensive, and a model that fails on a different slice than the one it replaced can trigger that escalation path just as often even with a better average score. So cost per successful task for a single model isn't really the number that matters, it's cost per successful task blended across whatever tier a failure falls back to, and that's a harder thing to measure than anything in either announcement.
That instrumentation is clean. Tagging the resolving tier and charging every attempt in the chain makes the blend accountable, and tracking escalation rate beside it is the right daily control. One distinction I would keep is between escalation forced by capability limits and escalation forced by verifier disagreement. The first is a routing cost you can tune with the cheap tier. The second is a measurement cost that a single success label will hide even after you split by tier. Curious how you separate those two in your Forge workflow.
Honest answer is I haven't built that particular split yet -- the tagging scheme I described tracks which tier resolved a task, not why the cheap tier's attempt didn't stick. But the distinction maps cleanly onto when you can detect it: a capability-limit escalation is something the cheap tier can flag about itself, low confidence, a refusal, a schema it can't fill in, so you catch it at generation time. A verifier-disagreement escalation only exists after the fact, when the cheap tier was confident and something downstream disagreed anyway. Those need separate hooks, one at the model call and one at the verifier, and folding them into a single escalated bucket is exactly what would hide the second kind of cost you're describing.
That split between generation time signals and verifier time disagreement is useful. I would treat the first as a routing signal and the second as an observability signal, then keep both in the task trace so the blended cost can be attributed without turning confidence into a proxy for correctness. This also makes it possible to compare models on escalation cause, not only on which one eventually resolved the task.
Cost per successful task is the right unit, but it assumes "successful" means the same thing across models, and none of the benchmarks you're citing really guarantee that. AutomationBench, FrontierCode and DeepSWE are all graded against some verifier, and two models can land the same aggregate score while failing on completely different slices of the task distribution — one nails the common case and blows the tail, the other does the opposite. That matters for the cost math because in production you don't just eat a failure, you escalate it to something more expensive, and a model that fails on a different slice than the one it replaced can trigger that escalation path just as often even with a better average score. So cost per successful task for a single model isn't really the number that matters, it's cost per successful task blended across whatever tier a failure falls back to, and that's a harder thing to measure than anything in either announcement.