Log retention sizing
Retention sets how long a log keeps history. A 3-day replay window is the minimum. Ten 1,024-byte events/s for 3 days need about 2.65 GB, or 1.33 GB with compression.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Retention 168 hours | Retention 72 hours |
|---|---|---|---|
| Disk | under 2 GB | 2.21 GB Fail | 1.33 GB Pass |
| Lost writes | none | 0 Pass | 0 Pass |
| Monthly cost | 12 USD/mo | 12 USD/mo | |
| p99 latency | 100 ms | 100 ms | |
| Error rate | 0% | 0% |
Key numbers
| Number | Value |
|---|---|
| Default retention of a log | 168 h |
| Disk throughput of a log | 500 MB/s |
| Object storage price | 0.015 USD/GB-mo |
5 MB/s · logs kept on the host · disk full in 33 minutes
How it works
A log keeps events for a set time so that readers can replay them. A consumer that was down for a day can catch up. A new service can read the history. The retention period is the promise: events younger than this are still there.
Storage follows from three numbers: the event rate, the size of each event and the retention. In the mini board the log accepts ten events each second of 1,024 bytes, and the readers need a replay window of at least 3 days. That is 10 x 1,024 x 86,400 x 3 bytes, about 2.65 GB. Compression halves it to about 1.33 GB. The disk limit is 2 GB.
The default retention of the log part is 168 hours, which is 7 days. The log gets events for five days, so at that retention all five days stay. With compression that is about 2.2 GB, which is over the limit of 2 GB. A retention of 72 hours covers the replay window and keeps about 1.33 GB.
Do not set the retention shorter than the replay window. Extra disk cannot bring back events that the policy already deleted. Also add the number of copies, and add headroom for a burst of events.
In the levels you set the retention and the compression from the replay need and the disk size.