The relay reserving maxOutputTokens against the cap before every call means a runaway loop stops early. Since the grader runs on whatever snapshot is left after agent.service stops, I assume a capped run still gets graded on partial work. Does run-summary.json mark those runs? If baseline runs hit the cap more often, a token-saving harness picks up some pass rate from that alone.
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
The relay reserving maxOutputTokens against the cap before every call means a runaway loop stops early. Since the grader runs on whatever snapshot is left after agent.service stops, I assume a capped run still gets graded on partial work. Does run-summary.json mark those runs? If baseline runs hit the cap more often, a token-saving harness picks up some pass rate from that alone.