Compaction debt
Compaction debt is merge work not yet done. 10,000 writes/s of 1 KB make about 100 MB/s of disk work after write amplification. If compaction falls behind, the debt reaches 1,024 MB and writes stall.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Compaction 20 MB/s, disk 50 MB/s | Compaction 160 MB/s, disk 200 MB/s |
|---|---|---|---|
| p99 latency | under 100 ms | 10.1 s Fail | 25 ms Pass |
| Error rate | under 0.1% | 68.5% Fail | 0% Pass |
| Monthly cost | 16 USD/mo | 16 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Default compaction speed | 100 MB/s |
| Compaction debt at which writes stall | 1,024 MB |
| Default disk throughput | 500 MB/s |
5 MB/s · logs kept on the host · disk full in 33 minutes
How it works
An LSM store makes writes cheap. It adds each write to memory and to a log, and it flushes sorted files to disk. Later, background compaction merges the files so that reads stay fast. The merge work does not go away. It moves from the write to the background.
Write amplification measures that work. In the mini board every source byte causes about 10 bytes of disk work. At 10,000 writes/s of 1 KB, the source rate is 10 MB/s and the disk work is about 100 MB/s. Compaction must run at that speed, or it falls behind.
When compaction falls behind, debt builds up: the more unmerged files, the slower the reads, and the more disk space. At some point the store protects itself and stalls writes. In the mini board the store starts with 512 MB of debt, and it stalls at 1,024 MB. With a disk of 50 MB/s and a compaction speed of 20 MB/s, it reaches the stall point, and p99 and errors rise.
To fix it, give compaction more speed and the disk more throughput, here 160 MB/s and 200 MB/s. The debt then falls and does not reach the limit.
Watch the debt as a metric. It is the early warning, before the stall.
In the levels you set compaction and disk speed from the write rate.