export_golden only pulls status="applied", so a draft a reviewer rejected never becomes a test case. I'd keep those rejections in the golden set as negatives with the decision_note attached, because they're the drafts I'd most want a new prompt version tested against.
Good call. Keeping rejected drafts as negatives with the reviewer's decision_note makes the golden set cover the failures that actually happened, not just the approved outputs. The main thing to watch is label quality: a rejection can mean a wrong answer or just a style preference, so I'd tag the rejection reason and only treat the clear-cut ones as hard-fail cases in the eval gate.
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
export_golden only pulls status="applied", so a draft a reviewer rejected never becomes a test case. I'd keep those rejections in the golden set as negatives with the decision_note attached, because they're the drafts I'd most want a new prompt version tested against.