Module 02 // ScalingLesson 08 / 3414 min read
Auto-scaling
Auto-scaling watches a metric like CPU or queue length and adds servers when load rises and removes them when it falls, so you have enough capacity without paying for idle machines.
Think of it like this
Analogy
A supermarket opens more checkout lanes when the lines get long and closes them when it's quiet. The manager doesn't keep every lane staffed all day, and doesn't wait until people are leaving their carts to react. They watch the lines, open a lane when they pass a few people, and give each new cashier a minute to settle in before deciding again.
The problem it solves
Traffic is rarely flat. It rises in the morning, peaks in the evening, spikes on launch days and drops at night. If you buy enough servers for the peak, you pay for idle machines most of the day. If you buy for the average, you fall over at the peak. Changing capacity by hand is slow and someone has to be awake to do it.
A real example
Example
Dinner time at a food delivery app
Orders on a delivery app triple between 18:00 and 21:00 every day. The team sets up three scaling rules.
A scheduled rule adds servers at 17:30, before the rush, because new servers take a few minutes to boot and warm up. A target-tracking rule keeps average CPU near 60%, which handles surprises like a rainy evening. And a minimum of 3 servers, spread across zones, stays on overnight, so one failure doesn't take the app down.
Scale-in is slower on purpose: servers are removed in small steps after 22:00, with a cooldown between steps. This stops the group from shrinking and growing again every few minutes.
How it works
Auto-scaling needs stateless servers behind a load balancer (lessons 6 and 7). A group of identical servers built from one template has a minimum, a maximum and a desired count. A scaling policy changes the desired count based on metrics.
Kinds of policies
- Target tracking: “keep average CPU at 60%”. The system works out how many servers that takes. The simplest and usually the best default.
- Step scaling: “CPU over 70%: add 2; over 90%: add 5”. More control for sharp spikes.
- Scheduled: “at 8:55 every weekday, set minimum to 20”. For predictable peaks.
- Predictive: the cloud forecasts daily patterns from history and scales ahead of them.
Choosing the metric
| Workload | Good signal |
|---|---|
| Web/API servers | CPU, or requests per server, or p95 latency |
| Queue workers | Messages waiting per worker, or age of the oldest message |
| Memory-heavy services | Memory usage |
| Serverless functions and containers | Concurrent requests (scaled for you) |
Doing it well
- Leave headroom. Target 50–70% CPU, not 90%, so there's room while new servers start.
- Start fast. Bake dependencies into images or containers so a new server is ready in seconds, not minutes.
- Scale out fast, scale in slowly. Removing capacity too eagerly causes flapping. Use a cooldown.
- Set a maximum. A bug or attack shouldn't be able to launch a thousand servers and a huge bill.
- Check what's behind you. Ten times more app servers send ten times more queries to the database. Auto-scaling the stateless tier just moves the bottleneck if the database can't follow.
- Drain before removing. Let in-flight requests finish (connection draining) before a server shuts down.
Serverless: scaling you don't configure
Functions and serverless containers scale per request, often down to zero. You trade control for simplicity, and you may see cold starts: a slower first request while a new instance starts.
Trade-offs
What you gain
- Capacity follows demand, so you pay much less than provisioning for the peak.
- Survives spikes without someone watching dashboards.
- Replaces unhealthy servers automatically.
What it costs you
- Reacts after load rises; very sudden spikes can still overwhelm you while servers boot.
- Bad settings cause flapping or runaway costs.
- Moves the bottleneck to whatever doesn't scale, usually the database.
When not to use it
- Stateful components like databases can't just add nodes on a timer; scale them deliberately.
- Steady, predictable load gains little; a fixed-size group with reserved pricing may be cheaper.
In the cloud
Auto-scalingEvery major cloud sells this as a managed service, so you configure it rather than build it.
AWS2
- Amazon EC2 Auto Scaling (opens vendor docs)
Adds or removes EC2 instances based on metrics like CPU or request count, or on a schedule.
- AWS Fargate (opens vendor docs)
Runs containers without servers; you scale the number of tasks and AWS finds the capacity.
Azure2
- Azure Monitor autoscale (opens vendor docs)
Scales Virtual Machine Scale Sets and App Service plans on metrics or schedules.
- Azure Container Apps (opens vendor docs)
Serverless containers that scale on HTTP traffic or events, down to zero.
Google Cloud2
- Managed instance group autoscaler (opens vendor docs)
Grows and shrinks a managed instance group on CPU, load-balancer usage or custom metrics.
- Cloud Run (opens vendor docs)
Runs containers that scale automatically with requests, including to zero.
Metrics feed a scaling policy, which changes the size of the server group behind the load balancer.
- CloudWatch metric
- Target tracking policy
- EC2 Auto Scaling group
Interview questions
Try answering out loud first, then open the model answer.
How would you configure auto-scaling for a web tier?
Model answer
Put stateless servers in a scaling group behind a load balancer with health checks. Set minimum capacity to survive losing a zone (for example 2 per zone), a sensible maximum to cap cost, and a target-tracking policy on CPU around 60% or on requests per target. Add scheduled scaling for known peaks. Use pre-baked images for fast startup, a warm-up period so new instances don't count until ready, connection draining when scaling in, and alarms if the group sits at its maximum.
Your service gets a 10× spike within a minute. Auto-scaling can't keep up. What do you do?
Model answer
Short term: keep more headroom (a lower target and a higher minimum), scale ahead of known events, and shed or queue load gracefully, with rate limiting and serving cached or degraded responses instead of falling over. Longer term: make startup much faster (containers, smaller images, warm pools), move slow work to queues so the front end only accepts requests, and put a CDN and caches in front so the spike doesn't all reach the servers.
How do you scale a pool of queue workers?
Model answer
On backlog per worker rather than CPU: for example, if each worker handles 10 messages a second and you want messages processed within 30 seconds, target about 300 waiting messages per worker. Also watch the age of the oldest message. Scale in slowly, and make sure workers finish or hand back their current message when stopped, using visibility timeouts and idempotent processing.
Quiz
Pick an answer to see whether it’s right and why.
0 of 6 answered
Key terms
- Auto-scaling
- Adding or removing servers automatically based on load, so you pay for what you need.
- Scaling policy
- The rule auto-scaling follows, such as "keep average CPU near 60%".
- Cooldown
- A waiting period after a scaling action so new servers can warm up before the system scales again.
- Horizontal scaling
- Adding more machines and splitting the work between them. Also called scaling out.
- Stateless service
- A service that keeps no user data between requests, so any copy of it can answer any request.
Progress is saved in this browser only.