Utilization and wait
Waiting time grows fast as a server nears full load. At 2,100 accepted requests/s the API uses about 63 percent of one core, and the spare capacity absorbs bursts. Extra requests are rejected early, so accepted ones stay under 100 ms.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | All 6,000 requests/s enter | Limit of 2,100 requests/s |
|---|---|---|---|
| CPU use | under 70% | 180% Fail | 63% Pass |
| Throughput | at least 1,800 req/s | 2,763 req/s Pass | 2,116 req/s Pass |
| p99 latency | under 100 ms | 10.1 s Fail | 68 ms Pass |
| Monthly cost | 8 USD/mo | 8 USD/mo | |
| Error rate | 54% | 64.7% |
Key numbers
| Number | Value |
|---|---|
| CPU time for one API call | 0.3 ms |
| Default CPU target for scaling | 0.7 ratio |
| Default rate limit of a limiter | 10,000 req/s |
4 workers · 3 jobs waiting · wait 33 ms
How it works
A server takes one request at a time for each core. When a request arrives and the core is busy, the request waits. At low load the wait is rare. As load nears 100 percent, waits build up, because the server has no spare time to clear them.
A simple model with random arrivals shows the shape. At 50 percent load a request waits about as long as the work takes. At 90 percent it waits about 9 times as long. At 99 percent it waits about 99 times as long. The curve is not a line, so the last 10 percent of capacity costs most of the latency.
In the mini board 6,000 requests/s reach an API that needs 0.3 ms of CPU for each request. One core can do at most about 3,300 requests/s. If all requests enter, queues grow and p99 passes the limit. A limit of 2,100 requests/s keeps the API near 63 percent, and the rest are rejected at once.
Rejecting is a choice. It keeps latency low for the work you accept, and the caller learns early that it must retry later. Pick the limit from the CPU target, not from the peak.
In the levels you set a safe limit and read the p99 and the CPU.