Sundarapandy

That’s a useful distinction. Separating known limitations from regression checks gives each set a clear purpose. Moving a case into the regression suite once it’s fixed also helps ensure the improvement is preserved through future prompt changes.

Good point - I'll admit I was thinking of the eval set mostly as a regression check, but you're right that an all green suite eventually just confirms the harness works, not that the prompt is good. Deliberately keeping a few should fail cases flips it into an actual signal. Worth building into how I set these up going forward.

Absolutely. Treating prompts as production artifacts is an important part of making prompt engineering reliable. Versioning prompts and rerunning a fixed evaluation set after every change gives teams a much better way to catch regressions before users do. I also agree on the output contract. When LLM output feeds another system or code path, schema validation creates a clear boundary where malformed responses can fail loudly instead of becoming downstream issues. Great point!