Alert fatigue

  • Choose
  • Watch 06

Difficulty 2 of 5

The on-call team got 188 pages in one day. Many came from 1-minute blips of a flaky backend, and a slow error leak ran for 6 hours. The SLO is 99.9 percent. Choose the alert rules: at most 6 pages a day, no false pages, and each incident found within 6 h.

Play level

Options

  • Page when errors pass 1 percent over 1 min
  • Page when errors stay over 1 percent for 5 min
  • Page when the error budget burns 14.4 times too fast over 1 h, or 6 times over 6 h
A graph for each resourceTap to add or remove the probes

Goals and limits

Alert pages
under 6
False alert pages
none
Detection time
under 21,600 s
Monthly cost
under 100 USD/mo
Traffic
500 req/s

Concepts

How players did

Player results load from the server.

All levels