The Codify step is the one I'd stress most: turning every confirmed failure into a permanent eval is what stops regressions after a model or prompt change. I like your split between LLM judges for fuzzy things and code checks for hard rules like a refund limit. For the code checks I'd assert on the tool-call arguments and side effects in the trace, not just the final reply, so refund(amount) is checked against the limit even when the agent's message sounds fine. For the judge-based evals, I'd periodically score a small human-labelled set to see whether the judge has drifted, since a judge that quietly gets lenient makes a green nightly run misleading. When you block a merge on a policy failure, do you rerun the eval a few times first to separate real regressions from run-to-run variance?
The Codify step is the one I'd stress most: turning every confirmed failure into a permanent eval is what stops regressions after a model or prompt change. I like your split between LLM judges for fuzzy things and code checks for hard rules like a refund limit. For the code checks I'd assert on the tool-call arguments and side effects in the trace, not just the final reply, so refund(amount) is checked against the limit even when the agent's message sounds fine. For the judge-based evals, I'd periodically score a small human-labelled set to see whether the judge has drifted, since a judge that quietly gets lenient makes a green nightly run misleading. When you block a merge on a policy failure, do you rerun the eval a few times first to separate real regressions from run-to-run variance?