Request coalescing
Request coalescing merges concurrent misses for one key into a single read from the source. If a hot key with 5,000 readers per second expires, the database gets 1 read and not 5,000. The other readers wait for that read and share its answer.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Each miss reads the database | One read for each key, others wait |
|---|---|---|---|
| p99 latency | under 150 ms | 24 ms Pass | 24 ms Pass |
| Error rate | under 0.01% | 0% Pass | 0% Pass |
| CPU use | under 70% | 1,000% Fail | 41% Pass |
| Monthly cost | 8 USD/mo | 8 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Reads per second, one cache | 100,000 ops/s |
| Queries per second, one database | 15,000 ops/s |
| Database read, median | 0.5 ms |
Coalescing · database reads after one expiry: 1
How it works
A hot key can have thousands of readers each second. While the key is in the cache, they all hit. When the entry expires, they all miss at the same moment, and each one starts its own database read. This is a cache stampede.
The database cannot do thousands of equal reads at once. In the mini board it has 0.5 vCPU and serves about 500 reads per second, and the burst asks for 10 times that. If reads time out, they do not fill the cache, so more readers miss and the stampede goes on.
Coalescing gives the fetch job to the first miss. Every other reader of the same key waits for that one read in flight. When it returns, the cache stores the value and all waiting readers get it. The database sees 1 read for the key.
Coalescing has limits. Waiting readers still pay the fetch time, so they are slower than a hit. If the fetch fails, all waiters fail together, so put a timeout on the fetch. One cache node merges only its own readers. Many nodes still send one fetch each.
In the levels you turn on coalescing and check that the miss burst no longer reaches the database.