Discussion on "Teaching a Small Language Model to Triage Incidents: A Beginner’s Unsloth and QLoRA Tutorial"

Haithem Slimi · 2026-10-30T00:35:01.434Z

Most fine-tuning tutorials stop when the training loss decreases. I wanted to answer a more useful operational question: can a small language model learn to turn an unstructured incident alert into a

Summary accuracy at 0.00% means exact match could never rise above 0.00%, so that row mostly measures the summary. I'd compute exact match over the five structured fields and score the summary on its own. The unknown_worker_saturation case would then count as a full match.

Yes, that’s correct. With summary accuracy at 0%, six-field exact match is necessarily 0%, so it doesn’t reveal how often the other five fields were all correct. A separate five-field structured exact-match score would be more informative, while the summary could be evaluated independently for semantic adequacy rather than exact wording. The unknown_worker_saturation example would indeed count as a full structured match under that metric. I’d keep both metrics rather than replace the original: six-field exact match measures the complete output contract, and five-field exact match isolates structured classification performance.

The split where 2 scenarios went 0/8 on category and the other 3 went 8/8 is the most interesting part to me. Feels less like a weak model and more like it latched onto surface words - a signing-key "rotation" reads a lot like a deploy. Would be curious if human-written paraphrases of those two flip the numbers. Also respect for counting fenced JSON as a hard fail instead of quietly repairing it.

That’s a plausible hypothesis, although the aggregate results alone don’t establish it. I’d need to inspect the raw predictions and compare the misleading cues across examples.

Human-written paraphrases would be a good follow-up: keep the same labels and held-out scenarios, remove some of the obvious lexical cues, and evaluate the same adapter without retraining. That would help distinguish scenario understanding from keyword matching.

And yes, fenced JSON was intentionally a hard failure because the output was defined as a strict application interface rather than text to be repaired silently.

Search Hashnode

Teaching a Small Language Model to Triage Incidents: A Beginner’s Unsloth and QLoRA Tutorial

Responses4