The 95% agreement example is a clear way to show why raw agreement misleads. One caution on the kappa recommendation: Cohen's kappa also shifts with class prevalence, so on a rare-failure set it can look low even for a decent judge. Reporting true positive and true negative rates separately, with Wilson intervals, makes the uncertainty at 100-ish labels visible and avoids that trap.

Two cheap additions to the validation step: run every pairwise judgment in both orders and treat a verdict flip as a measured position-bias rate instead of an anecdote, and stratify the held-out set by failure mode so a judge that is blind to one mode cannot hide behind the others. I would also pin the judge model version and re-run the human-labelled set whenever it changes, since a silent provider update changes validity without changing the dashboard's stability. How would you size the re-validation cadence when human labelling budget is small?