Throttle the backfill

Flow 10
HTTP/2SQLAMQPUsers3,000 req/sWorker2 vCPU · 2 GBPostgres16 vCPU · 16 GBUsers0 req/sMessage bro…4 vCPU · 4 GB
HTTP/2SQLAMQPUsers3,000 req/sWorker2 vCPU · 2 GBPostgresUsers0 req/sMessage b…4 vCPU · 4 GB

Turn every goal green.

Traffic
3,000 req/s · writes only
Limit
Worker answers about 6,700 req/s
Bill
  • Worker8 USD/mo
  • Postgres64 USD/mo
  • Message broker16 USD/mo
88 of 160 USD/mo

Goals

Not tested yet

99 of 100 requests finish faster than this.Time until all the work is done.Saved data that was lost later.What the servers and services cost each month.

A logic fix means a year of daily totals must be recomputed: 30 million row upserts into the serving database. The live pipeline writes 3,000 rows/s to the same database. Finish the backfill within 2 hours and keep live p99 below 50 ms.

Each worker adds one network cardDrag or use arrow keys to change the workers

2 workers · 1 TB moves in 1.1 h · 1,000 Mbps per card

Parts you can add: Worker, Batch engine, Rate limiter. At most 4 parts. Budget: 160 USD/mo.

Tap a part to add it, or drag it onto a trace.Click a part to add it, or drag it onto a trace.