The 95% agreement example is a clear way to show why raw agreement misleads. One caution on the kappa recommendation: Cohen's kappa also shifts with class prevalence, so on a rare-failure set it can look low even for a decent judge. Reporting true positive and true negative rates separately, with Wilson intervals, makes the uncertainty at 100-ish labels visible and avoids that trap.
Two cheap additions to the validation step: run every pairwise judgment in both orders and treat a verdict flip as a measured position-bias rate instead of an anecdote, and stratify the held-out set by failure mode so a judge that is blind to one mode cannot hide behind the others. I would also pin the judge model version and re-run the human-labelled set whenever it changes, since a silent provider update changes validity without changing the dashboard's stability. How would you size the re-validation cadence when human labelling budget is small?
With a small labelling budget, I'd let events set the re-validation cadence more than the calendar. Pinning the judge version, as you suggest, gives a clear trigger: a version change gets a re-run of the human-labelled set before the new scores count, the same gate I'd put on a schema migration.
Between changes, a small labelled sample each error-analysis cycle (the FAQ I cited suggests every few weeks) keeps an eye on drift, and the cases where the judge and the humans disagree on that sample become the next review queue.
Fair point on kappa and prevalence, too; on a rare-failure set that's a good reason to lean on the true positive and true negative rates I listed as the alternative.
The 95% agreement example is a clear way to show why raw agreement misleads. One caution on the kappa recommendation: Cohen's kappa also shifts with class prevalence, so on a rare-failure set it can look low even for a decent judge. Reporting true positive and true negative rates separately, with Wilson intervals, makes the uncertainty at 100-ish labels visible and avoids that trap.
Two cheap additions to the validation step: run every pairwise judgment in both orders and treat a verdict flip as a measured position-bias rate instead of an anecdote, and stratify the held-out set by failure mode so a judge that is blind to one mode cannot hide behind the others. I would also pin the judge model version and re-run the human-labelled set whenever it changes, since a silent provider update changes validity without changing the dashboard's stability. How would you size the re-validation cadence when human labelling budget is small?