HTTP server: Rate limiting

A rate limiter protects a server from clients that send too many requests. The token bucket algorithm allows short bursts while enforcing a steady average rate. Each client gets a bucket of tokens that refills over time, and every request spends one token. When the bucket is empty the server answers 429 with a retry-after header.

Edit on GitHub

Each client may average REFILL_RATE requests per second, with bursts of up to BURST requests when the bucket is full.

const BURST = 5;
const REFILL_RATE = 1;

interface Bucket {
  last: number;
  tokens: number;
}

Buckets are keyed by client IP and live in memory.

const buckets = new Map<string, Bucket>();

function take(ip: string): boolean {
  const now = Date.now();
  const bucket = buckets.get(ip) ?? { last: now, tokens: BURST };

Refill lazily based on how much time passed since the last request, instead of running a timer per client.

  const elapsed = (now - bucket.last) / 1000;
  bucket.tokens = Math.min(BURST, bucket.tokens + elapsed * REFILL_RATE);
  bucket.last = now;
  buckets.set(ip, bucket);

  if (bucket.tokens < 1)
    return false;

  bucket.tokens -= 1;
  return true;
}

Buckets for idle clients are useless after they refill completely, so a periodic sweep keeps the Map from growing forever. Ende.unrefTimer marks the interval as non-blocking, so the process can still exit normally.

const IDLE_MS = (BURST / REFILL_RATE) * 1000;

const sweeper = setInterval(() => {
  const cutoff = Date.now() - IDLE_MS;

  for (const [ip, bucket] of buckets) {
    if (bucket.last < cutoff)
      buckets.delete(ip);
  }
}, 30_000);

Ende.unrefTimer(sweeper);

Ende.serve((_req, info) => {

The second handler argument carries connection details, including the network address of the client.

  const ip = info.remoteAddr.hostname;

  if (!take(ip)) {

retry-after tells well-behaved clients how long to back off. One token takes 1 / REFILL_RATE seconds to appear.

    return new Response("too many requests\n", {
      headers: { "retry-after": String(Math.ceil(1 / REFILL_RATE)) },
      status: 429
    });
  }

  return new Response("ok\n");
});

Behind a load balancer the socket address belongs to the proxy, not the client. In that case derive the key from a trusted header such as x-forwarded-for instead of info.remoteAddr.

Sending requests faster than the refill rate drains the bucket. The first five requests spend the burst capacity, then the server starts refusing: for i in $(seq 1 7); do curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8000/; done 200 200 200 200 200 429 429

Run this example locally using the Ende CLI:

ende run -N https://docs.ende.sh/examples/scripts/http_server_rate_limit.ts

HTTP server: Rate limiting

Additional resources