Alert fatigue

Watch 06

Choose

The on-call team got 188 pages in one day. Many came from 1-minute blips of a flaky backend, and a slow error leak ran for 6 hours. The SLO is 99.9 percent. Choose the alert rules: at most 6 pages a day, no false pages, and each incident found within 6 h.

Goals

Not tested yet

Time from a fault until an alert fires.What the servers and services cost each month.
Choose a design
Page when errors pass 1 percent over 1 min
Page when errors stay over 1 percent for 5 min
Page when the error budget burns 14.4 times too fast over 1 h, or 6 times over 6 h