Discussion on "Rate Limiting Algorithms: How to Protect Your API Without Annoying Your Users"

ZyVOP · 2026-10-26T08:48:56.695Z

Every API eventually meets a client that sends too much traffic. Sometimes it’s a buggy retry loop, sometimes a scraper, and sometimes a perfectly well-meaning customer whose cron job just got ten tim

One more thing for the IP section: on IPv6, key on the /64, not the full address. One machine usually gets a whole /64 and can rotate through addresses for free, so per-address limits barely slow it down.

Mobile carriers are the opposite problem. A lot of people sit behind carrier-grade NAT, so in some regions one IPv4 can be a whole neighborhood. Per-IP limits only really make sense as a loose outer layer.

Great points! IPv6 address rotation makes per-address limits much easier to bypass, so aggregating requests by a /64 prefix is a useful additional safeguard. On the other hand, carrier-grade NAT shows why strict IP-based limits can unfairly affect legitimate users sharing the same IPv4 address.

I’d treat IP-based throttling as one layer in a broader strategy, combining it with account-level limits, API keys where applicable, and endpoint-specific quotas. The challenge is balancing abuse prevention with avoiding false positives for legitimate users. Thanks for highlighting both sides!

For variable request costs, I would distinguish an empty bucket from a request that costs more than the bucket can ever hold. Waiting and retrying can fix the first case but not the second, so the client needs a different explanation

Absolutely. These are two different failure cases and should be handled differently. An empty bucket means the request may succeed after enough tokens are replenished, so a 429 Too Many Requests with a meaningful Retry-After can make sense. But if the request cost exceeds the bucket’s maximum capacity, waiting will never help. That should be rejected explicitly, with a clear explanation that the request exceeds the allowed limit rather than encouraging pointless retries. Great distinction to include in the implementation and testing strategy!

One extra layer I’d add for long-running LLM requests is an in-flight concurrency limit. A token bucket caps arrivals, but if calls become much slower, the number running at once can still grow while the arrival rate stays within budget. For usage that isn’t known up front, reserving an upper bound and reconciling with actual usage also needs its own accounting. I’d test a slowdown and a client disconnect separately: the client going away doesn’t necessarily mean the provider stopped doing work.

Exactly. This is why an explicit unknown state is important in the reservation lifecycle. A client disconnect should not automatically release reserved budget when the provider’s outcome is still unknown. The reservation should remain unresolved until final usage is reconciled or the case is flagged for review, with retries accounting for that outstanding reservation. Otherwise, a transient disconnect could turn into a double-spend.

I’d also make this a separate failure scenario in the test suite: the client disconnects, the provider continues processing, and a retry arrives before the original request’s outcome is known. Great catch—this makes the accounting model much more robust!

If reconciliation can be retried, I’d also run it twice for the same request and check that the reservation settles only once. That would cover duplicate settlement as well as the retry race.

Agreed. Reconciliation should be idempotent, not just retryable. I’d run it twice for the same request and verify that the reservation is settled only once, with no duplicate budget release or usage charge. An atomic state transition and a unique settlement key can help enforce that guarantee. This catches duplicate settlement in addition to the retry race. Great addition to the test checklist!

Search Hashnode

Rate Limiting Algorithms: How to Protect Your API Without Annoying Your Users

Responses10