Immutable pizza

I swear it never happens until it happens: API is perfectly fine at 50 req/s, then a freaking marketing email lands in 400k inboxes and 2,000 requests show up in the same second. The average load over the minute is still fine. The peak looks like scrambled eggs 🍳.

You have two options: let the burst through and let it melt whatever it’s coming next (very often an unhappy, super expensive third party SaaS provider), or smooth it out in a way downstream consumers don’t even notice. The leaky bucket is the oldest and simplest trick in the book to solve that, and it’s one of those algorithms that’s easier to feel than to read about.

The metaphor

Picture a bucket with a small hole in the bottom.

You can pour water in as fast or as unevenly as you like.
Water leaks out of the hole at a constant rate, no matter how full the bucket is.
If you pour faster than it leaks for long enough, the bucket fills up and the excess spills over the rim.

Swap water for requests and you have a traffic shaper. It has exactly three knobs:

Leak rate (r): the throughput you promise downstream.
Capacity (b): how big a burst you’re willing to absorb.
Overflow policy: what happens to a request that doesn’t fit. Usually you drop it, or answer 429 Too Many Requests.

A few things worth trying:

Set incoming below the leak rate. The bucket stays almost empty and requests pass through with near-zero delay. The shaper is invisible when you don’t need it.
Push incoming above the leak rate. The bucket fills, then spills. Look at the bottom though: the output pace doesn’t change. Ever.
Hit Burst +20 with a small capacity, then again with a big one. Same burst, very different outcome: one drops most of it, the other just makes it wait.

That last point is the whole trade-off, and we’ll come back to it.

The algorithm

The queue version is almost embarrassingly small:

class LeakyBucket<T> {
  private queue: T[] = [];

  constructor(
    private capacity: number,
    ratePerSec: number,
    private send: (item: T) => void
  ) {
    setInterval(() => this.leak(), 1000 / ratePerSec);
  }

  // Called on every incoming request
  offer(item: T): boolean {
    if (this.queue.length >= this.capacity) return false; // spill → 429
    this.queue.push(item);
    return true;
  }

  // Called by the clock, never by the client
  private leak() {
    const item = this.queue.shift();
    if (item !== undefined) this.send(item);
  }
}

The interesting part isn’t any single line, it’s who drives what: offer is driven by the clients, leak is driven by the clock. The input and the output are decoupled, and the only thing connecting them is a bounded queue. That’s what makes the output smooth: it physically can’t go faster than the timer.

Real implementations rarely fire one timer per request. At high rates setInterval gets coarse, so you leak in batches: on each tick, send floor(elapsed × rate) items and carry the remainder over.

Capacity is a latency budget

Here’s the thing that took me embarrassingly long to internalise: the capacity of the bucket isn’t really a “how many requests” setting. It’s a “how long can a request wait” setting.

A request that lands in a bucket holding n items waits roughly n / r seconds. So the worst case is:

max wait ≈ capacity / leak rate

20 slots leaking at 5 req/s means a request can sit there for up to 4 seconds. If your client times out after 2, half your bucket is just queueing up requests that are already dead.

The widget below runs a full minute of traffic through a bucket in one go. Pick a traffic pattern, then play with the two knobs and watch the drops (✕) and the wait times move against each other.

Sixty seconds of trafficSame traffic, your knobs. Hover the chart to inspect any second.

Leak rate 5 req/sCapacity 20

Arrived

Dropped

Avg wait

Max wait

ArrivedSent downstream✕Dropped (overflow)

Bucket level (peak each second)

Show data table

Second	Arrived	Sent	Dropped	Level

Things to notice:

Spike at 5 req/s, capacity 5: 93 requests dropped, but nobody waits at all. Capacity 60: drops fall to 35, but the unlucky last request waits 11 seconds. Neither is “right”; it depends on whether your caller prefers a fast no or a slow yes.
Random averages 5.6 req/s. With the leak rate at 6 the bucket copes: 15 drops, under a second of average wait. Drop it to 5, just below the average, and the bucket never gets to drain: drops triple and the average wait doubles. Queues near 100% utilisation behave terribly.
Bursty at 3 req/s: the average input is 4.8, so the bucket stays full and drops 42% of everything. This is the important one. A leaky bucket smooths bursts, it cannot fix sustained overload. If the average input is above r, you need more capacity downstream, not a bigger bucket.

Which gives a handy rule of thumb for sizing one:

Knob	Must be at least	Must be at most
Leak rate `r`	your average input rate	what downstream can take
Capacity `b`	your typical burst size	`r × acceptable wait`

If those two ranges don’t overlap, no amount of tuning will save you. That’s a capacity planning problem wearing a rate limiting costume.

Two buckets, one name

So far the bucket has been a queue: requests go in, wait, and come out evenly. That’s a shaper. It delays traffic.

But the name has been used for a second, subtly different thing since it first showed up in ATM networks in the ’80s: the leaky bucket as a meter. Here the bucket doesn’t hold requests at all. It holds a number. Every request adds 1 to the level, the level drains at rate r, and a request that would make it overflow is rejected on the spot. Nothing waits. That’s a policer.

The beauty of the meter is that it needs no queue and no timer. You don’t have to leak continuously, you can compute how much would have leaked since the last request, lazily, whenever someone knocks:

class LeakyMeter {
  private level = 0;
  private last = performance.now();

  constructor(private capacity: number, private ratePerSec: number) {}

  tryAcquire(now = performance.now()) {
    // leak whatever drained since we last looked
    const leaked = ((now - this.last) / 1000) * this.ratePerSec;
    this.level = Math.max(0, this.level - leaked);
    this.last = now;

    if (this.level + 1 > this.capacity) {
      const retryAfter = (this.level + 1 - this.capacity) / this.ratePerSec;
      return { allowed: false, retryAfter };
    }
    this.level += 1;
    return { allowed: true, retryAfter: 0 };
  }
}

Two numbers of state per client. That’s why basically every API rate limiter is some variation of this. Here it is, wired to a button. Try clicking at a calm pace first, then mash it.

Notice the Retry-After: because the leak rate is known, the server can tell the client exactly when its next request would be accepted. Good clients use it, bad clients ignore it and keep hitting the 429 wall.

Where you’ve already met it

nginx’s limit_req module is documented as using the leaky bucket method, and it lets you pick which of the two flavours you want with a single word:

limit_req_zone $binary_remote_addr zone=api:10m rate=5r/s;

server {
  location /api/ {
    # shaper: queue up to 20 excess requests, release them at 5r/s
    limit_req zone=api burst=20;

    # meter: forward immediately, still reject anything beyond the burst
    # limit_req zone=api burst=20 nodelay;

    limit_req_status 429; # the default is 503, which lies to your clients
  }
}

Shopify’s REST Admin API is probably the most famous public example: a 40-request bucket that leaks 2 requests per second, with the current level handed back on every response in an X-Shopify-Shop-Api-Call-Limit: 32/40 header, so well-behaved clients can pace themselves before ever seeing a 429.

Leaky bucket vs token bucket

You can’t talk about the leaky bucket without someone bringing up the token bucket, so let’s get it over with.

A token bucket fills with tokens at rate r, up to a maximum of B. Each request spends a token; no token, no service. The key difference is that idle time banks credit: after a quiet minute the bucket is full, and the next B requests go through all at once.

Same input, same rate, same queue in front of both. Only the release rule differs:

Drag the token bucket size down to the rate and the two lines become identical. Drag it up and the token bucket happily forwards each burst almost as it came in, while the leaky bucket keeps its flat line and makes the burst wait instead. On the default settings the token bucket also drops a third as much (15 vs 51), because it spends credit it banked while idle. Whether that’s a feature depends entirely on whether downstream can take a 20 req/s spike.

Here’s the bit that confuses people: the leaky bucket as a meter and the token bucket are the same algorithm seen in a mirror. A meter at level L is exactly a token bucket holding B − L tokens. They admit the same requests at the same times. So the real choice isn’t “leaky vs token”, it’s shaper vs policer:

	Leaky bucket (queue)	Leaky meter / token bucket
Output	perfectly smooth, never > `r`	bursts up to `b`, averages `r`
Excess traffic	waits, then spills	rejected immediately
State	a queue + a clock	two numbers
Latency	adds up to `b / r`	adds none
Great for	protecting a fragile downstream, calling third-party APIs with strict quotas	public API rate limits, per-user fairness

If you’re calling someone else’s rate-limited API, you want a shaper on your side, so you never trip their policer. If you’re exposing an API, you want a policer, so slow clients don’t eat your memory with queued requests.

Things that bite in production

Use a monotonic clock. performance.now() or process.hrtime, never Date.now(). NTP can move wall-clock time backwards, and your bucket will happily un-leak.
One bucket per instance is N buckets. An in-memory limiter behind a load balancer with 6 replicas lets through 6× the rate. For a shared limit, keep the state in Redis and make the read-modify-write atomic (a Lua script, or a single SET of the value below).
Store one number, not two. GCRA, the formalisation that came out of ATM, collapses the meter into a single theoretical arrival time (tat): the moment the bucket would be empty again. With T = 1 / r:
```
const tat = Math.max(storedTat, now) + T;
if (tat - now > capacity * T) reject();
else storedTat = tat;
```
tat − now is just the bucket level measured in seconds. One timestamp per client, no float drift, trivially atomic.
A queue in memory is a queue you lose. A shaper holding 10k requests during a deploy is 10k requests that never get an answer. Keep shaping buckets small, or put a real queue behind them.
Don’t shape interactive traffic. Making a user stare at a spinner for 8 seconds is worse than a fast, honest 429 with a Retry-After. Shape background work, police user-facing work.

Wrapping up

The leaky bucket is two ideas in a trench coat. As a queue, it decouples the rate things arrive from the rate things leave, trading latency for smoothness. As a meter, it’s a two-number rate limiter that powers half the internet’s 429s. Both come with the same lesson: the leak rate is a promise to whoever sits downstream, and the capacity is how much waiting you’re willing to put on whoever sits upstream. Pick both on purpose.

The leaky bucket: shaping traffic one drop at a time