Haithem Slimi

Yes, that’s correct. With summary accuracy at 0%, six-field exact match is necessarily 0%, so it doesn’t reveal how often the other five fields were all correct. A separate five-field structured exact-match score would be more informative, while the summary could be evaluated independently for semantic adequacy rather than exact wording. The unknown_worker_saturation example would indeed count as a full structured match under that metric. I’d keep both metrics rather than replace the original: six-field exact match measures the complete output contract, and five-field exact match isolates structured classification performance.

That’s a plausible hypothesis, although the aggregate results alone don’t establish it. I’d need to inspect the raw predictions and compare the misleading cues across examples. Human-written paraphrases would be a good follow-up: keep the same labels and held-out scenarios, remove some of the obvious lexical cues, and evaluate the same adapter without retraining. That would help distinguish scenario understanding from keyword matching. And yes, fenced JSON was intentionally a hard failure because the output was defined as a strict application interface rather than text to be repaired silently.