The 60-second coalescing window is the part I'd want to stress-test: if the leader request is a long generation that gets cut off or fails post-call guardrails, do the waiters all fall through to provider calls at once and recreate the herd you just removed? We've hit a similar pattern with retried agent calls and ended up adding jittered fall-through plus a per-key cap on concurrent followers. Also curious whether tool definitions in the key mean a re-ordered tools array busts the cache, since that's a common source of silent misses.
AI Developer
build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results.
The 60-second coalescing window is the part I'd want to stress-test: if the leader request is a long generation that gets cut off or fails post-call guardrails, do the waiters all fall through to provider calls at once and recreate the herd you just removed? We've hit a similar pattern with retried agent calls and ended up adding jittered fall-through plus a per-key cap on concurrent followers. Also curious whether tool definitions in the key mean a re-ordered tools array busts the cache, since that's a common source of silent misses.