Guides

System design

Rate limiting: a visual guide

Five ways to limit requests. All of them run on this page, so you can send requests and see which ones pass.

Piotr Jura,

A rate limit says how many requests one client can make in a period of time. A request over the limit gets a 429 error.

A rate limiterlimit: 5 per second
clientserver
429limiter
3 / s
0.0 / s passed, 0.0 / s rejected
Move the slider past 5. The limiter passes 5 requests per second and sends the others back with a 429.
  • Every figure below uses the same limit: 5 requests per 4 seconds.
  • A solid dot is a request that passed.
  • An orange ring got a 429.

Fixed window

You cut time into 4-second windows and count the requests in each window.

Fixed window5 per 4 s
Counter: 0 / 5Resets in 4.0 sNext request: passes
200429
0 / 50 / 50 / 5
0 passed, 0 rejected
The dashed lines are the window edges. The number on each window is its counter.
  • You store one number per client, so it is cheap.
  • At each window edge the counter starts from 0 again.
  • Press Edge burst to see the problem. 5 requests before the edge and 5 after it all pass, so the server gets 10 in one second.

Sliding window log

You store the time of every request and count the ones from the last 4 seconds.

Sliding window log5 per 4 s
In the last 4 s: 0 / 5Stored timestamps: 0Next request: passes
200429
now
last 4 s
0 passed, 0 rejected
The box is the last 4 seconds. A request passes when fewer than 5 dots are in it.
  • The window moves with the clock, so the same edge burst stops at 5.
  • You pay with memory, because you store one timestamp per request. With a large limit that is a lot.

Sliding window counter

You keep two counters, one for this window and one for the window before. The previous window counts only for the part that the last 4 seconds still cover.

Sliding window counter5 per 4 s
previous 0 × 100% + current 0 = 0.0Next request: passes
200429
000
last 4 s
0 passed, 0 rejected
The box covers a part of the previous window and all of the current one. The number above the timeline is the estimate.
  • It is two numbers per client, so memory is not a problem.
  • It assumes that the old requests arrived evenly. Sometimes they did not, and then the result is a little wrong.
  • Cloudflare tested it on 400 million requests, and 0.003% of them got the wrong answer.

Token bucket

Tokens drop into a bucket at a fixed rate, and each request takes one. If the bucket is empty, the request gets a 429.

Token bucketcapacity 5, 1.25 tokens per second
Tokens: 5.0 / 5Next request: passes
200429
now
0 passed, 0 rejected
5
1.25 / s
Press Burst of 8 on a full bucket, then again at once. Then set the capacity to 1.
  • The refill rate is the long-term average.
  • The capacity is the largest burst you allow.
  • A client that waited has a full bucket, so it can send a burst at once.
  • Stripe and the Claude API both limit requests this way.

Leaky bucket

Requests go into a queue, and the queue drains at a fixed rate. If the queue is full, the request gets a 429.

Leaky bucketqueue of 5, 1 out every 0.8 s
In the queue: 0 / 5Next request: goes into the queue
in429out
now
0 passed, 0 rejected
The top lane is what arrives. The bottom lane is what the server gets.
  • The server gets evenly spaced requests, whatever arrives.
  • A request in the queue has to wait, so you pay with delay.
  • NGINX limits requests this way.

Side by side

Here the same requests go through four limiters at the same time.

Same requests, four limiters5 per 4 s
Fixed window0 passed, 0 rejected
now
Sliding window log0 passed, 0 rejected
now
Sliding window counter0 passed, 0 rejected
now
Token bucket0 passed, 0 rejected
now
Each row gets the same requests at the same time. Change the traffic to compare.
  • With steady traffic, the fixed window passes 5 in a row and then rejects everything until the window ends.
  • The token bucket spends its 5 tokens first and then rejects one request at a time.
  • With edge bursts, only the fixed window passes all the requests.
  • The leaky bucket is not in the figure, because it accepts about the same requests as the token bucket. It only makes them wait.

More than one server

One server can keep the counters in its own memory. With two servers you need one shared counter, usually in Redis, and you get a new problem.

Two servers, one counterthe counter starts at 4, the limit is 5
Server ARedisServer B
GET count gives 44
4GET count gives 4
4 is less than 5. Pass. SET count 55
54 is less than 5. Pass. SET count 5

6 requests passed. The limit is 5.

Two requests from one client reach two servers at the same moment. Only one more fits under the limit.
  • Both servers read 4, so both pass the request and both write 5.
  • To fix it, the read, the check and the write must be one step.
  • In Redis that step is INCR for a counter. A token bucket needs a Lua script.

What the client sees

The limiter answers every request with headers. Send a few requests and read them.

The responsetoken bucket, 5 per 4 s
HTTP/1.1 ...X-RateLimit-Limit: 5X-RateLimit-Remaining: 5Retry-After: -
Ready to send
200429
now
0 passed, 0 rejected
The card is the answer to the last request. After a 429, the bar shows how long the client must wait.
  • 429 and Retry-After are standard HTTP.
  • The other header names are different in each API. GitHub sends x-ratelimit-remaining and the Claude API sends anthropic-ratelimit-requests-remaining.
  • When a client gets a 429, it waits for Retry-After and then sends the request again.

Which one to use

Pick what you need. If you are not sure, start with a token bucket.

Which one fitspick what you need
StrictBurstsMemorySmooth
Token bucketPublic APIs
Sliding window counterStrict limits for many clients
Sliding window logSmall limits that must be exact
Fixed windowDaily or monthly quotas
Leaky bucketA slow service behind the limiter
A filled square means the limiter gives you that. Pick one or more needs. The rows that fit stay lit.

Sources

Share

Newsletter

Email from me

Occasional notes on agentic development — new posts, videos, and tools I'm actually using.

By submitting, you agree to receive emails from Piotr Jura at The Agentic Developer. Unsubscribe anytime. Articles and videos when I publish.