Canary with automatic rollback
Difficulty 4 of 5
At 0:20 a new build reached every API server at once. The API ran 10 times slower and 0.5 percent of requests failed until a person rolled it back at 1:50. A bad build may reach at most 5 percent of traffic and must be undone in 30 s. Errors under 0.1 percent, p99 under 40 ms.
0:20. p99 almost doubles on every API server right after a deploy, and errors rise a little. Graph: p99 by build.
Canary · 5% of requests fail for 15 seconds · 750 failed requests
Goals and limits
- Error rate
- under 0.1%
- p99 latency
- under 40 ms
- Change exposure
- under 5%
- Rollback time
- under 30 s
- Monthly cost
- under 60 USD/mo
- Edit limit
- 3
- Traffic
- 1,000 req/s
Concepts
How players did
Player results load from the server.