Memory hierarchy
The memory hierarchy is the ladder of places that keep data, from CPU caches and RAM to disks and remote storage. A read from an in-process cache takes 0.1 ms here, and a read from an object store 30 ms. Keep hot data on the fast steps of the ladder.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Cache too small for the hot keys | Cache sized to hold the hot keys |
|---|---|---|---|
| Hit ratio | at least 90% | 79.3% Fail | 92.2% Pass |
| p99 latency | under 150 ms | 25 ms Pass | 24 ms Pass |
| Monthly cost | 8 USD/mo | 8 USD/mo | |
| Error rate | 0% | 0% |
Key numbers
| Number | Value |
|---|---|
| Read from an in-process cache, median | 0.1 ms |
| Network round trip in one zone | 0.5 ms |
| Database read (Postgres), median | 0.5 ms |
| Database read (Postgres), p99 | 2 ms |
| Object store read, median | 30 ms |
NVMe drive · 75 µs per read
How it works
Data lives on many levels. CPU caches answer in nanoseconds but keep only megabytes. Main memory keeps gigabytes and answers in about 100 nanoseconds. A local SSD keeps terabytes, but each read takes tens of microseconds. A database on another machine adds a network round trip. Object storage keeps almost anything, but each request takes tens of milliseconds.
Each step down is slower, larger and cheaper for each GB. A system is fast when most reads hit the upper steps. The working set, the data that requests touch often, must fit in the fastest level that can keep it. When it does not fit, reads fall to the next level, and latency can become ten or a hundred times longer.
Caches use this ladder on purpose. An in-process cache keeps hot keys in the memory of the service itself, with no network hop. A shared cache keeps more keys for all servers, one round trip away. The database and its disk keep everything.
Each copy on a higher step uses memory and can be out of date. Keep only hot data high, and set how long each copy lives.
In the levels you size memory so that the hot data fits, and let cold data stay on cheaper storage.