TL;DR: The pass rate a grading model reports is part your product and part the grader's own mistakes. Measure those mistakes on a small set a person has checked, and you can subtract them back out. En
Markdown supported · ⌘↵ to post
The Rogan-Gladen step divides by TPR+TNR-1, so the uncertainty in TNR matters most, and a random 100-label sample at a ~90% pass rate leaves only about 10 truly bad outputs to estimate it from. It may be worth stratifying the labeling (oversample judge-Pass items, or known-bad ones) and bootstrapping a confidence interval on the corrected rate. Also, TPR/TNR are only valid for that judge model and prompt version, so any judge change would seem to need a fresh label check. Have you looked at how stable they are across prompt revisions?
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Your stale-correction example is the one I needed. The labels say the judge fails 88% of bad outputs, today's subtler ones only 60%, and the stale correction reports 74% when the true rate is 60%. I'm putting the label date next to the corrected pass rate on my dashboard, so a number built on old labels at least looks old.