Module 04 // PerformanceLesson 16 / 3417 min read
Rate limiting
A rate limiter caps how many requests each client can make in a period of time, protecting your service from overload, abuse and runaway scripts.
Think of it like this
Analogy
A theme-park ride lets on a fixed number of riders every few minutes. During quiet hours you walk straight on. When a crowd shows up, the line grows, and if it gets too long the staff stop people from joining: “come back later”. Nobody gets hurt, the ride keeps running at full speed, and everyone gets a fair turn.
The problem it solves
Without limits, one client can take down a service for everybody. That could be a buggy script retrying in a tight loop, a scraper, an attacker guessing passwords, or simply a big customer running a job at full speed. Every request uses CPU, database connections and memory. Past a point the service slows down for everyone, then fails for everyone.
A real example
Example
A login form under attack
A bot starts trying passwords on one account: 10,000 attempts a minute, from a handful of IP addresses.
The site adds two limits. Each account allows 5 failed logins per 15 minutes, and each IP allows 100 login attempts per minute. Requests over a limit get 429 Too Many Requests with a Retry-After header. The attack slows to a crawl, while a real user who mistypes twice doesn't notice anything.
The same site sells API access. Free keys get 100 requests a minute and paid keys get 1,000, enforced with a token bucket so a short burst is fine but the average is capped.
How it works
Rate limiting counts requests per client (an API key, user ID or IP address) and rejects those over the limit, usually with 429 Too Many Requests.
The algorithms
| Algorithm | Idea | Good | Watch out |
|---|---|---|---|
| Token bucket | Tokens refill at rate R up to size B; each request spends one | Allows bursts up to B, average R; tiny state (2 numbers) | Bursts can still hit backends hard |
| Leaky bucket | Requests queue and leave at a fixed rate | Perfectly smooth output | Bursts wait or are dropped; adds delay |
| Fixed window | Count per calendar minute | Simplest | Double bursts at window edges |
| Sliding window log | Keep timestamps of recent requests | Exact | Memory per request |
| Sliding window counter | Blend current and previous window counts | Nearly exact, tiny state | Approximate |
Token bucket is the most common choice for APIs, and it's what many cloud API gateways use.
Where to enforce it
- At the edge (CDN or web application firewall): stops floods and abusive IPs before they reach you.
- At the API gateway: per API key or user, with plans like “free: 100/min, paid: 1,000/min”.
- Inside services: protect a specific expensive operation or a fragile dependency.
Distributed rate limiting
With many servers, local counters don't add up (see the last quiz question). Common approaches:
- A shared counter in Redis with an atomic increment-and-expire, or a small script that runs the token bucket atomically. Adds about 1 ms per request.
- Local buckets with periodic syncing: each server gets a share of the budget. Faster, slightly less exact.
- Consistent hashing: route each client to one limiter node that owns its counter.
Lesson 31 designs a full distributed rate limiter.
Being a good citizen
Tell clients where they stand:
HTTP/1.1 429 Too Many Requests
Retry-After: 12
RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 12Clients should back off and retry later with exponential backoff and jitter (lesson 23).
Rate limiting vs load shedding
Rate limiting is per client and about fairness. Load shedding is about the whole service: when it's overloaded, drop or degrade low-priority work no matter who sent it, so important requests still succeed.
Trade-offs
What you gain
- Protects the service from overload and abuse.
- Keeps things fair between clients and enables paid tiers.
- Limits the cost of runaway clients.
What it costs you
- Legitimate users can be blocked by limits set too low.
- Distributed limits need a shared store or accept some inaccuracy.
- Limiting by IP can punish many users behind one office or carrier network.
When not to use it
- Don't rate-limit internal health checks or critical control traffic in ways that cause outages.
- Don't use rate limiting as your only protection against overload; you also need autoscaling, timeouts and load shedding.
In the cloud
Rate limitingEvery major cloud sells this as a managed service, so you configure it rather than build it.
AWS2
- AWS WAF rate-based rules (opens vendor docs)
Blocks or counts requests from an IP or key once it passes a request rate you set.
- Amazon API Gateway throttling (opens vendor docs)
Token-bucket limits per API, stage or client usage plan.
Azure2
- Azure API Management rate-limit policies (opens vendor docs)
Limit calls per key or subscription over a time window, or set long-term quotas.
- Azure Web Application Firewall rate limiting (opens vendor docs)
Rate-limit rules on Front Door or Application Gateway's firewall.
Google Cloud2
- Google Cloud Armor rate limiting (opens vendor docs)
Throttle or temporarily ban clients that exceed a request rate at the load balancer.
- Apigee quota and spike arrest (opens vendor docs)
Policies that cap calls per developer or app and smooth sudden bursts.
Limits are enforced at the edge before requests reach your servers.
- CloudFront
- AWS WAF rate-based rule
- API Gateway usage plan
- Backend
Interview questions
Try answering out loud first, then open the model answer.
Explain the token bucket algorithm and why it's popular.
Model answer
Each client has a bucket of up to B tokens that refills at R tokens per second. A request takes a token if one is available; otherwise it's rejected or delayed. B controls how large a burst is allowed and R sets the sustained rate. State is just two numbers per client (token count and last refill time), and refill can be computed lazily on each request, so it's cheap to store in memory or Redis. It matches real usage, which is bursty, better than strict per-second limits.
How would you rate-limit an API served by 50 servers?
Model answer
Keep counters in a shared low-latency store like Redis, keyed by client and window, updated atomically with a Lua script implementing a token bucket or sliding window counter. Put the check in the API gateway or middleware. To cut latency and avoid making Redis a single point of failure, shard Redis, use a short local cache of “this client is blocked”, and decide whether to fail open (allow) or fail closed (deny) if Redis is unavailable. For most APIs I'd fail open with local fallback limits.
What's the problem with fixed-window rate limiting?
Model answer
The counter resets at fixed boundaries, so a client can send a full window's worth just before the boundary and another just after, which is twice the limit in a moment. Sliding windows count over the last N seconds from now, which removes the edge effect. A sliding window counter approximates this with two counters, weighting the previous window by how much of it still overlaps.
Quiz
Pick an answer to see whether it’s right and why.
0 of 7 answered
Key terms
- Rate limiting
- Capping how many requests a client can make in a period of time.
- Token bucket
- A rate-limiting algorithm where requests spend tokens that refill at a steady rate, allowing short bursts.
- Leaky bucket
- A rate-limiting algorithm that queues requests and lets them out at a fixed rate.
- Sliding window
- A rate-limiting algorithm that counts requests in the last N seconds measured from now, avoiding burst gaps at window edges.
- 429 Too Many Requests
- The HTTP status code a server returns when a client is over its rate limit.
- API gateway
- The single front door for clients that routes requests to services and handles auth, limits and logging.
Progress is saved in this browser only.