The interesting part is that the agent can improve against the test suite while the test suite itself is still missing important cases. For meeting summaries, I’d keep a holdout set with things like corrected figures, negated decisions, and action items assigned to the wrong person, then check whether changes actually improve those cases too. Otherwise, you can end up optimizing for the harness rather than the product. The test harness needs testing as well.
Sina Rezaei
**Project Manager | Digital Strategist | Analytics | Creator | Full-Stack Developer** 🗼 🏰 ❾¾ 🦁⃝⃒⃤ 🪄 🦉 | 별 | 🐉 ⛩️ 夢 ☯ ރ
The interesting part is that the agent can improve against the test suite while the test suite itself is still missing important cases. For meeting summaries, I’d keep a holdout set with things like corrected figures, negated decisions, and action items assigned to the wrong person, then check whether changes actually improve those cases too. Otherwise, you can end up optimizing for the harness rather than the product. The test harness needs testing as well.