Scatter-gather

  • Fix
  • Scale 14

Difficulty 4 of 5

Support looks up a customer's orders 40 times/s. The orders are sharded by order ID, so each lookup asks all 16 shards. Orders arrive at 800/s. Bring lookup p99 under 150 ms and order p99 under 100 ms, with at most 4 edits.

14:20. Order lookup p99 is 10 s. Graph: all 16 order shards at 100 percent CPU, each reading 1,600 MB/s from a 500 MB/s disk.

Play level

HTTP/2HTTP/2SQLSQLSQLSQLSQLSQLSQLSQLSQLSQLUsers800 req/sUsers40 req/sNode API2 vCPU · 2 GBLoad bala…2 vCPU · 1 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GBPostgres1 vCPU · 2 GB
UsersUsersNode…Load…Post…Post…Post…Post…Post…Post…Post…Post…Post…Post…Post…Post…Post…Post…Post…Post…
The board at the start of the level
Keys on a hash ringArrow keys add or remove a node. Tap to switch the hash.

Hash ring · 3 to 4 nodes · 8 of 36 keys move

Goals and limits

p99 latency (Lookup)
under 150 ms
p99 latency (Order)
under 100 ms
Error rate (*)
under 0.1%
Monthly cost
under 200 USD/mo
Edit limit
4
Traffic
40 req/s

Concepts

How players did

Player results load from the server.

All levels