Sticky trouble

  • Fix
  • Scale 05

Difficulty 3 of 5

Each deploy restarts the 3 API nodes one at a time, for 10 s each. Since then, p99 reaches 6 s and 9 percent of page views fail, while one node idles. 2,400 views/s, 90 percent in signed-in sessions. Bring p99 under 150 ms and errors under 2 percent. Max 4 edits, no new spend.

1:05. Rolling deploy in progress. p99 is 7 s. Graph: CPU by API node: api-c 100 percent, api-a 43 percent, api-b 6 percent.

Play level

HTTP/2HTTP/2HTTP/2HTTP/2Users2,400 req/sLoad balanc…1 vCPU · 1 GBNode API1.5 vCPU · 1.5 GBNode API1.5 vCPU · 1.5 GBNode API1.5 vCPU · 1.5 GB
HTTP/2HTTP/2HTTP/2HTTP/2UsersLoad bala…Node APINode APINode API
The board at the start of the level
One balancer, many serversDrag or use arrow keys to add or remove servers

5 servers · 72% busy · p99 66 ms

Goals and limits

p99 latency
under 150 ms
Error rate
under 2%
Monthly cost
under 22 USD/mo
Edit limit
4
Traffic
2,400 req/s

Concepts

How players did

Player results load from the server.

All levels