Throttle the backfill

  • Build
  • Flow 10

Difficulty 4 of 5

A logic fix means a year of daily totals must be recomputed: 30 million row upserts into the serving database. The live pipeline writes 3,000 rows/s to the same database. Finish the backfill within 2 hours and keep live p99 below 50 ms.

Play level

HTTP/2SQLAMQPUsers3,000 req/sWorker2 vCPU · 2 GBPostgres16 vCPU · 16 GBUsers0 req/sMessage bro…4 vCPU · 4 GB
HTTP/2SQLAMQPUsers3,000 req/sWorker2 vCPU · 2 GBPostgresUsers0 req/sMessage b…4 vCPU · 4 GB
The board at the start of the level
Each worker adds one network cardDrag or use arrow keys to change the workers

2 workers · 1 TB moves in 1.1 h · 1,000 Mbps per card

Goals and limits

p99 latency (Live)
under 50 ms
Completion time (Backfill)
under 7,200 s
Lost writes
none
Monthly cost
under 160 USD/mo
Parts you can add
Worker, Batch engine, Rate limiter
Part limit
4
Traffic
3,000 req/s

Concepts

How players did

Player results load from the server.

All levels