Concurrency
Concurrency is the number of requests that a service works on at the same time. An API server has 64 workers by default, and each worker holds memory while it waits. Too few workers make requests wait in line, and too many use up the memory.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Pool of 2 workers | Pool of 12 workers |
|---|---|---|---|
| p99 latency | under 1 s | 10.1 s Fail | 388 ms Pass |
| Error rate | under 0.1% | 75.8% Fail | 0% Pass |
| Monthly cost | 160 USD/mo | 160 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Workers in one API server, default | 64 workers |
| Memory that one worker holds | 8 MB |
| Database connections from one API server | 20 connections |
| Connections that one Postgres database accepts | 200 connections |
280 req/s · CPU 70% · mean 22 ms
How it works
A request that waits for a database, a disk or another service still holds a worker. The number of requests in progress equals the arrival rate times the time that each request takes. At 60 requests per second and 0.5 s for each request, 30 requests are in progress at any moment. With fewer workers than that, new requests wait in line, even when the CPU is idle.
Each worker costs memory, and the connections to a database have their own limit. An API worker holds 8 MB by default, and a Postgres database accepts 200 connections. If 10 servers each open 20 connections, the database is full. More workers help only until memory, CPU or a limit further down the path runs out. After that point, they add contention and make p99 worse.
Size a pool from three numbers: the request rate, the time for each request and the memory for each worker. Then keep room for slow periods, when each request takes longer and more requests are in progress at once.
In the levels you set worker pools and connection pools, so that requests do not wait in line and memory does not run out.