Kartik N V J K

Summary accuracy at 0.00% means exact match could never rise above 0.00%, so that row mostly measures the summary. I'd compute exact match over the five structured fields and score the summary on its own. The unknown_worker_saturation case would then count as a full match.

On tests for code that already works, the wrong test I run into doesn't fail loudly. It asserts whatever the code does today, bug included, and goes green. So I check where each expected value came from before I trust a generated test.

Since a prompt change means a refit and the calibration rerun takes about 3 minutes, I'd store each fitted T with the prompt version it came from and refuse to load one fitted on a different version. The intent step landing at 1.138 on BANKING77 and 0.619 on CLINC150 shows how far apart two fits can sit.

For the full eval set going through the batch API, I'd keep an eye on markForRetry. If errored rows sit waiting for a retry while the succeeded ones get scored, the eval number comes from whatever finished first. I hold the report until every custom_id has a result or a recorded error.

After consolidating, I'd also assert on block.input in that same call, say that 4412 lands in the order id argument, since you point out errors move into arguments. It still stops before anything executes, so the suite stays quick.

With a 3072-bit key written as 0xC00 getting flagged, I'd want to read the reviewer's stated reason next to the flag. If it says the key is too small, it misread the value. If it complains about a magic number, a style note got scored as a security finding. Did the false positives come with reasons you could sort that way?

Ten days of calling parser breaks prompt problems sounds familiar to me. I'd still watch for a reply that passes the schema with a wrong value, the right vendor and a confidently wrong amount, which only your field-level accuracy on the fifty examples would catch. When the failure rate dropped by half, did that count both kinds or just schema failures?

Your nine-attempt math should be plain in the logs. "Number of attempts, where available" gets fuzzy when the SDK retries inside what the job runner sees as one try, so I log the count at each layer separately. Then a slow request tells me which layer spent the time.

161 subagents on September 26 alone, two days after you switched to GPT-6 Sol, and by then there was no way to tell whether the model or the P50 and P51 rounds drove it. I'd put the subagent count into those three plain daily sentences too. A burst like that should get flagged the same day.

I'd run your known-answer canary on the model path as well. The ambiguous remainder still goes to a model behind that thin interface, and one fixed input sent there each run would show a provider-side change before the real queue does. Do you run one there, or only on the compiled workflow?