Time-series downsampling
Keep recent data raw and old data as aggregates. 200,000 streams of 16-byte points need about 3,871 GB for 7 raw days. 1-minute aggregates keep the rest of the year in about 3,299 GB.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Raw points for a year | 7 raw days, then 1-minute aggregates |
|---|---|---|---|
| Disk | under 9,000 GB | 10,506.24 GB Fail | 7,170.05 GB Pass |
| Error rate | under 0.1% | 94.8% Fail | 0% Pass |
| Monthly cost | 128 USD/mo | 128 USD/mo | |
| p99 latency | 24 ms | 25 ms |
Key numbers
| Number | Value |
|---|---|
| Points per second, one database vCPU | 10,000 points/s |
| Default downsample interval | 60 s |
| Object storage price | 0.015 USD/GB-mo |
5 MB/s · logs kept on the host · disk full in 33 minutes
How it works
A time-series store receives a point for every stream at a steady rate. The data grows with time. A year of raw points would not fit on the disk that the budget gives you. But raw detail is useful only for a short time. Nobody needs a point for every second of a day last March.
Downsampling replaces old raw points with aggregates, for example the average, the minimum and the maximum for each minute. An aggregate takes the place of 60 points, so it uses much less space, and it still answers the questions about trends.
In the mini board 200,000 streams each write one 16-byte point each second, plus 16 bytes of metadata. The requirement is to keep one year, with 7 days of raw points and 1-minute aggregates after that. The raw days need about 3,871 GB. The remaining 358 days need about 3,299 GB as aggregates. The total, about 7,170 GB, fits below the limit of 9,000 GB. Without downsampling the data needs about 10,500 GB, which is over the limit.
Set the rule from the questions that you must answer. If you need the exact value of each second for 30 days, raw data must stay that long. An aggregate cannot give back a lost detail.
In the levels you set when raw points become aggregates and check the disk use.