Dr Srinivas Padmanabhuni

The near-miss routing suite is the right idea, and the customer_get_profile example shows why: the new tool's own tests pass while the billing path quietly loses requests. I would add three things. First, track the routing decision as a rate over repeated runs, say 20 per request, since selection at temperature above zero can flip, and a single pass proves little. Second, rerun the suite with the tool list shuffled, because you cite order bias and a test that only passes in one ordering is fragile. Third, keep a held-out set of requests that should route to no tool at all, so a broad new description cannot grab them. Gate the merge on any previously correct request dropping below its baseline rate.

The Codify step is the one I'd stress most: turning every confirmed failure into a permanent eval is what stops regressions after a model or prompt change. I like your split between LLM judges for fuzzy things and code checks for hard rules like a refund limit. For the code checks I'd assert on the tool-call arguments and side effects in the trace, not just the final reply, so refund(amount) is checked against the limit even when the agent's message sounds fine. For the judge-based evals, I'd periodically score a small human-labelled set to see whether the judge has drifted, since a judge that quietly gets lenient makes a green nightly run misleading. When you block a merge on a policy failure, do you rerun the eval a few times first to separate real regressions from run-to-run variance?