Hot-key copies
Hashing spreads many keys, but 1 key always lands on 1 owner. When 1 key gets 90 percent of 200,000 reads/s, that owner overloads while the rest sit idle. Local copies of the hot key spread those reads.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | One owner serves the hot key | Local copies of the hot key |
|---|---|---|---|
| p99 latency | under 150 ms | 364 ms Fail | 23 ms Pass |
| Error rate | under 0.1% | 34.7% Fail | 0% Pass |
| Monthly cost | 80 USD/mo | 80 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Reads per second, one cache | 100,000 ops/s |
| Reads per second, one local cache | 100,000 ops/s |
| Local cache read, median | 0.1 ms |
Local copies · owner of the hot key 5,000 reads/s, one cache serves 100,000
How it works
A partitioned cache assigns each key to one owner by a hash. With many keys of similar load, the owners share the work evenly. The rule fails when one key is hot. Its owner serves every read of that key, and no other server can help.
In the mini board four cache replicas have plenty of CPU in total. At 0:10 one key starts to get 90 percent of 200,000 reads/s. That is 180,000 reads/s on one owner, and one cache serves about 100,000 ops/s. The owner overloads, and p99 and errors rise, while three servers do little.
The fix is to put a copy of the hot key near the readers. Each application server keeps a small local copy, and a read hits that copy first. The 180,000 reads/s then spread over all the servers, and the owner sees only the refreshes.
The local copy is an extra cache with its own risk. It can be old, so give hot keys a short life or send invalidations. And you must find the hot keys: count the reads for each key, and copy only the keys that pass a rate.
A second answer is to split the key, for example key#1 to key#8, and read any one of them.
In the levels you copy the hot key to the readers and check the load of the owner.