Global config push

  • Fix
  • Watch 10

Difficulty 3 of 5

Three regions each serve 1,200 requests/s. At 0:20 a config push reaches every API server at once. All regions fail until a person reverts it at 1:20. A bad change may reach at most 40 percent of servers and must be undone within 30 s. Keep eu and us errors under 0.1 percent.

00:22. Errors climb in eu, us and ap at the same time, 2 s after a config push. Graph: errors by region.

Play level

RegionRegionRegionSQLSQLSQLUsers1,200 req/sUsers1,200 req/sUsers1,200 req/sAPI server4 vCPU · 4 GBPostgres4 vCPU · 8 GBAPI server4 vCPU · 4 GBPostgres4 vCPU · 8 GBAPI server4 vCPU · 4 GBPostgres4 vCPU · 8 GB
SQLSQLSQLUsersUsersUsersAPI serv…PostgresAPI serv…PostgresAPI serv…Postgres
The board at the start of the level
A canary before the whole fleetTap to change the deployment method

Canary · 5% of requests fail for 15 seconds · 750 failed requests

Goals and limits

Change exposure
under 40%
Rollback time
under 30 s
Error rate (Eu)
under 0.1%
Error rate (Us)
under 0.1%
Monthly cost
under 300 USD/mo
Parts you can add
Alerts
Part limit
12
Edit limit
3
Traffic
1,200 req/s

Concepts

How players did

Player results load from the server.

All levels