Log sampling
Sampling keeps the log lines that matter and drops the rest by rule. Errors are rare, so keep every one. Keeping 800 of 20,000 info lines/s costs 1,650 USD/mo and not about 41,000 USD/mo.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Sampling limit too high | Errors kept, few info lines kept |
|---|---|---|---|
| Error rate (Errors) | under 1% | 0% Pass | 0% Pass |
| Throughput (Info) | at least 400 req/s | 4,082 req/s Pass | 819 req/s Pass |
| Monthly cost | 9,880 USD/mo | 3,264 USD/mo | |
| p99 latency | 15 ms | 15 ms | |
| Level rules | none broken | Budget Fail | Pass |
Key numbers
| Number | Value |
|---|---|
| Log intake price | 0.12 USD/GB |
| Lines per second, one log intake | 10,000 lines/s |
| Default sampling rate (keep all) | 1 ratio |
1,500 req/s · Node API · 3 hosts · 18 USD/mo
How it works
A managed log service bills each line. Info lines are common and each one says little. Error lines are rare and say a lot. A fleet that sends 20,000 info lines/s and 200 error lines/s pays mostly for lines that nobody reads.
Sampling sets a rule for each class of line. Keep all errors. Keep a small share of info lines, here 4 percent. In the mini board each info line per second adds about 2 USD/mo, so all 20,000 info lines/s would cost about 41,000 USD/mo. Keeping 800 lines/s costs 1,650 USD/mo, and every error stays.
Two limits bound the rule. In an incident errors can reach 3,000 lines/s. An error limit of 6,000 lines/s or more keeps them all. And the info sample must stay large enough to show what happened around an error. Below about 780 lines/s here the sample is too small.
A sampler that treats all lines alike drops errors in an incident, when you need them most. Split the limit by class, and watch how many lines the sampler drops.
In the levels you set the limit for each class and keep the bill under the budget.