The in-flight count needs a precise release event when a caller disconnects. Releasing its slot immediately can undercount work if cancellation has not yet stopped the provider request; retaining it forever can leak capacity after an acknowledged cancellation.
I would distinguish caller completion from upstream attempt termination and exercise both races in the saturation test. A replacement request should acquire capacity only according to the resource actually being protected. That makes max_inflight meaningful during disconnect storms, when frontend request counts and active provider work can diverge sharply.
Good catch — the release event has to be tied to when the attempt is actually terminal, not only when the caller socket closes.
In the contract I treat those as two observables: caller completion/cancel vs upstream attempt termination. Releasing the in-flight slot on disconnect alone can undercount if the provider request is still running; holding it until the upstream settles (or a hard cancel timeout) avoids leaking capacity the other way. The saturation ladder should exercise both paths as separate cases, with distinct evidence for each.
I’ll tighten that wording in the next edit of the guide. Thanks for pushing on the precision.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
The in-flight count needs a precise release event when a caller disconnects. Releasing its slot immediately can undercount work if cancellation has not yet stopped the provider request; retaining it forever can leak capacity after an acknowledged cancellation.
I would distinguish caller completion from upstream attempt termination and exercise both races in the saturation test. A replacement request should acquire capacity only according to the resource actually being protected. That makes max_inflight meaningful during disconnect storms, when frontend request counts and active provider work can diverge sharply.