A rate limit says how many requests one client can make in a period of time. A request over the limit gets a 429 error.
- Every figure below uses the same limit: 5 requests per 4 seconds.
- A solid dot is a request that passed.
- An orange ring got a
429.
Fixed window
You cut time into 4-second windows and count the requests in each window.
- You store one number per client, so it is cheap.
- At each window edge the counter starts from 0 again.
- Press Edge burst to see the problem. 5 requests before the edge and 5 after it all pass, so the server gets 10 in one second.
Sliding window log
You store the time of every request and count the ones from the last 4 seconds.
- The window moves with the clock, so the same edge burst stops at 5.
- You pay with memory, because you store one timestamp per request. With a large limit that is a lot.
Sliding window counter
You keep two counters, one for this window and one for the window before. The previous window counts only for the part that the last 4 seconds still cover.
- It is two numbers per client, so memory is not a problem.
- It assumes that the old requests arrived evenly. Sometimes they did not, and then the result is a little wrong.
- Cloudflare tested it on 400 million requests, and 0.003% of them got the wrong answer.
Token bucket
Tokens drop into a bucket at a fixed rate, and each request takes one. If the bucket is empty, the request gets a 429.
- The refill rate is the long-term average.
- The capacity is the largest burst you allow.
- A client that waited has a full bucket, so it can send a burst at once.
- Stripe and the Claude API both limit requests this way.
Leaky bucket
Requests go into a queue, and the queue drains at a fixed rate. If the queue is full, the request gets a 429.
- The server gets evenly spaced requests, whatever arrives.
- A request in the queue has to wait, so you pay with delay.
- NGINX limits requests this way.
Side by side
Here the same requests go through four limiters at the same time.
- With steady traffic, the fixed window passes 5 in a row and then rejects everything until the window ends.
- The token bucket spends its 5 tokens first and then rejects one request at a time.
- With edge bursts, only the fixed window passes all the requests.
- The leaky bucket is not in the figure, because it accepts about the same requests as the token bucket. It only makes them wait.
More than one server
One server can keep the counters in its own memory. With two servers you need one shared counter, usually in Redis, and you get a new problem.
GET count gives 44GET count gives 4SET count 55SET count 56 requests passed. The limit is 5.
- Both servers read 4, so both pass the request and both write 5.
- To fix it, the read, the check and the write must be one step.
- In Redis that step is
INCRfor a counter. A token bucket needs a Lua script.
What the client sees
The limiter answers every request with headers. Send a few requests and read them.
429andRetry-Afterare standard HTTP.- The other header names are different in each API. GitHub sends
x-ratelimit-remainingand the Claude API sendsanthropic-ratelimit-requests-remaining. - When a client gets a
429, it waits forRetry-Afterand then sends the request again.
Which one to use
Pick what you need. If you are not sure, start with a token bucket.
Sources
Newsletter
Email from me
Occasional notes on agentic development — new posts, videos, and tools I'm actually using.