Deep health checks

  • Fix
  • Guard 12

Difficulty 3 of 5

At 0:20 one third of orders started to fail. All three API servers still pass their health checks, and the balancer still sends each one a third of the traffic. Keep errors under 1 percent and p99 under 100 ms. You have 3 edits.

0:20. Order errors jump to 33 percent. The balancer shows 3 of 3 API servers healthy. Graph: errors by API server.

Play level

HTTP/2SQLSQLHTTP/2SQLUsers1,800 req/sLoad balanc…1 vCPU · 1 GBNode API1 vCPU · 1 GBConnection…0.5 vCPU · 1 GBNode API1 vCPU · 1 GBConnection…0.5 vCPU · 1 GBNode API1 vCPU · 1 GBConnection…0.5 vCPU · 1 GBPostgres4 vCPU · 8 GB
HTTP/2SQLSQLHTTP/2SQLSQLHTTP/2SQLSQLUsers1,800 req/sLoad balan…Node APIConnection…Node APIConnection…Node APIConnection…Postgres
The board at the start of the level
Three failure domainsTap to fail one replica. The fourth tap heals all.

3 of 3 up · availability 99.9999%

Goals and limits

Error rate
under 1%
p99 latency
under 100 ms
Monthly cost
under 80 USD/mo
Edit limit
3
Traffic
1,800 req/s

Concepts

How players did

Player results load from the server.