Autoscaling lag
Autoscaling adds replicas after the load has risen, and a new replica needs time to boot. If a replica takes 90 s to start, the replicas that are ready at the start must carry the rise for 90 s. Keep headroom or a higher floor for that time.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | One replica, scales after the rise | Three replicas ready before the rise |
|---|---|---|---|
| p99 latency | under 150 ms | 10.1 s Fail | 24 ms Pass |
| Error rate | under 0.1% | 21.7% Fail | 0% Pass |
| Monthly cost | 22 USD/mo | 24 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Time for a new replica to boot | 90 s |
| Wait between scaling steps | 120 s |
| Default CPU target for scaling | 0.7 ratio |
280 req/s · CPU 70% · mean 22 ms
How it works
An autoscaler watches a metric, such as CPU use, and adds replicas when it passes a target. This takes time. The metric must show the rise, the scaler must decide, and the new replica must start. A replica that is still booting serves no requests.
In the mini board traffic rises at 20 s, and a replica takes 90 s to boot. With one replica ready, the load is on that one replica for the whole boot time. Its queue grows and p99 passes the 150 ms limit, no matter how many replicas the scaler asks for.
You cannot make a booting process serve requests. You can change what is ready before the rise. A higher minimum number of replicas gives ready capacity. A lower CPU target makes the scaler start earlier. Both cost money, because you pay for replicas that wait.
A short boot time helps more than a clever rule. Smaller images, warm pools and pre-loaded caches all cut the time that a new replica needs. Scale-in also needs care: a long cooldown keeps replicas for a while after the peak.
In the levels you set the minimum, the target and the cooldown so the first minutes of a rise stay inside the limits.