Chaos day: four failure drills
Difficulty 5 of 5
Four 50 s waves each inject one failure, in order: an API instance crashes, the database runs 12 times slower, zone A fails, an API instance runs 6 times slower. Keep browse p99 under 300 ms and errors under 2 percent for 95 percent of the run. Each wave adds budget.
3 of 3 up · availability 99.9999%
Goals and limits
- Time within SLO
- at least 95%
- p99 latency (Browse)
- under 300 ms
- Error rate (Browse)
- under 2%
- Error rate (Orders)
- under 5%
- Monthly cost
- under 24 USD/mo
- Waves
- 4
- Time within SLO
- at least 95%
- Parts you can add
- Node API, Load balancer (L7), Redis cache
- Part limit
- 10
- Traffic
- 1,080 req/s
Concepts
How players did
Player results load from the server.