Metric cardinality

  • Fix
  • Watch 03

Difficulty 2 of 5

A 400 GB metrics store gets points from four exporters. Disk use grows 84 GB/day and the store fills on day 5, so dashboards and alerts go blind. Keep route, latency and tenant metrics for 14 days with no gaps. Keep disk under 320 GB. The store is locked. At most 3 edits.

Day 5, 19:00 UTC. Metrics store disk is full. Dashboards and alerts are empty. Graph: disk use up 84 GB/day for 5 days.

Play level

HTTP/2HTTP/2HTTP/2HTTP/2Partner apps200 req/sPartner apps96 req/sPartner apps60 req/sPartner apps40,000 req/sTime series…6 vCPU · 8 GB
HTTP/2HTTP/2HTTP/2HTTP/2Partner…Partner…96 req/sPartner…60 req/sPartner…Time se…
The board at the start of the level
A graph for each resourceTap to add or remove the probes

One CPU graph · CPU 40% · disk no graph · the fault is hidden

Goals and limits

Error rate (Route series)
under 0.1%
Error rate (Latency series)
under 0.1%
Error rate (Tenant series)
under 0.1%
Disk
under 320 GB
Parts you can add
Log store
Part limit
3
Edit limit
3
Traffic
200 req/s

Concepts

How players did

Player results load from the server.

All levels