Metric cardinality
A metric series is one unique set of label values, and each new label value multiplies the series. A user ID label made 600,000 series and 40,000 points/s, 112 times the other three families together. It wrote 84 GB/day.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | User ID label on a metric | Labels with few values |
|---|---|---|---|
| Error rate (Route series) | under 0.1% | 65.8% Fail | 0% Pass |
| Error rate (Latency series) | under 0.1% | 65.8% Fail | 0% Pass |
| Error rate (Tenant series) | under 0.1% | 65.8% Fail | 0% Pass |
| Disk | under 320 GB | 400.98 GB Fail | 10.33 GB Pass |
| Monthly cost | 32 USD/mo | 32 USD/mo | |
| p99 latency | 15 ms | 15 ms |
Key numbers
| Number | Value |
|---|---|
| Default series limit of a metrics store | 100,000 series |
| Points per second, one database vCPU | 10,000 points/s |
| Default retention | 30 days |
1,500 req/s · Node API · 3 hosts · 18 USD/mo
How it works
A metric such as request count has labels: route, status, region. The store keeps one series for each unique set of label values. With 10 routes, 5 statuses and 3 regions that is 150 series. Add a label with a value for each user, and the count becomes the number of users times 150.
In the mini board a 400 GB metrics store receives points from four exporters. Three of them send about 356 points/s in total. The fourth has a label with the ID of each user. It makes 600,000 series, and each series sends one point every 15 s, which is 40,000 points/s. At 24 bytes per point the store gets 84 GB/day and is full on day 5. Dashboards and alerts then go blind.
The fix is not more disk, because the store is locked. The fix is fewer labels. Keep labels with a few values, such as route, status and region. Put IDs in logs or traces, where one line belongs to one event.
Set a limit on the series count for each metric, and alert when a metric nears it. A new label is a change to the bill, so review it like one.
In the levels you remove the label that creates the series and keep the other metrics.