Backfill throttle
A backfill shares its sinks with the live pipeline, so it needs a speed limit that comes from the spare capacity. Here the disk has room for about 9,500 extra rows/s. A backfill at 6,700 rows/s takes 75 minutes, and live p99 stays near 7 ms.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Backfill with 4 vCPU, no limit | Backfill with 2 vCPU |
|---|---|---|---|
| p99 latency (Live) | under 50 ms | 2.6 s Fail | 6.6 ms Pass |
| Completion time (Backfill) | under 7,200 s | 2,399.58 s Pass | 4,499.58 s Pass |
| Lost writes | none | 0 Pass | 0 Pass |
| Monthly cost | 104 USD/mo | 96 USD/mo | |
| Error rate | 5.8% | 0% |
Key numbers
| Number | Value |
|---|---|
| Writes per second, one database vCPU | 2,500 writes/s |
| Disk operations per second | 50,000 IOPS |
| Disk operations for one write | 2 IOPS |
2 workers · 1 TB moves in 1.1 h · 1,000 Mbps per card
How it works
A backfill recomputes old data after a logic fix. It runs beside the live pipeline and writes to the same database. If it runs as fast as it can, it takes the database from the live traffic, and live p99 rises.
In the mini board a year of daily totals must be recomputed, which is 30 million row upserts. The live pipeline writes 3,000 rows/s to the same database. The backfill must finish within 2 hours, and live p99 must stay below 50 ms.
Start from the spare capacity. The database is limited by its disk. It has room for about 9,500 extra rows/s on top of the live load. A backfill that writes faster than this slows the live writes. A backfill at 6,700 rows/s uses most of the spare room, finishes in about 75 minutes, and live p99 stays near 7 ms.
The speed limit can be a rate limiter, or it can come from the size of the worker pool, as in the mini board. The workers are smaller, so they cannot write too fast.
A backfill also needs to be safe to stop and start: save its position, and write each row so that a repeat does not change the result. Run it at night if the live load is lower.
In the levels you size the backfill from the spare capacity and check the live p99.