Push instead of polling
Polling cost follows the poll rate, not the change rate. 100,000 clients that poll every 2 s make 50,000 calls/s for only 5,000 changes/s. A push stream sends only the changes, but it holds one connection for each client.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Clients poll every 2 s | Server pushes each change |
|---|---|---|---|
| p99 latency | under 200 ms | 2.5 s Fail | 69 ms Pass |
| CPU use | under 60% | 375% Fail | 37.5% Pass |
| Monthly cost | 16 USD/mo | 16 USD/mo | |
| Error rate | 72.8% | 0% |
Key numbers
| Number | Value |
|---|---|
| Header bytes for each HTTP/2 request | 60 bytes |
| Header bytes for each stream event (SSE) | 10 bytes |
| Requests per second, one API server | 10,000 ops/s |
HTTP/1.1 · connections: 6 · 30 files in 7 round trips, 630 ms
How it works
A client that polls asks the same question again and again. Most answers say nothing new. Each poll still costs a request, a response header and some CPU on the server. When the poll interval is short and the clients are many, the polls become the load.
In the mini board 100,000 clients poll every 2 s, and a client gets a new value only every 20 s. The server handles 50,000 calls each second to deliver 5,000 values each second. Nine calls in ten carry no news, and the API needs about 3.8 times the CPU it has.
A push stream reverses this. The client opens one connection, and the server sends a message when a value changes. Server-sent events do this over plain HTTP in one direction. The work follows the change rate, and each event has a small header.
Push has a cost of its own. The server keeps one open connection for each client, and each connection uses memory. A restart makes all clients reconnect at the same time. Plan for reconnects with backoff and for the memory of 100,000 connections.
In the levels you replace polling with a stream and check p99 and CPU.