Parallel workers and a shared limit
Independent work splits over many workers, until a shared limit stops the gain. One worker that waits 270 ms for each block indexes 3.7 blocks/s, so 1,000,000 blocks take 3 days. Four workers do about 15 blocks/s, and more than 7 gain nothing against a limit of 25 calls/s.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | One worker | Four workers on a shared queue |
|---|---|---|---|
| Completion time | under 86,400 s | 90,000 s Fail | 67,991.28 s Pass |
| Lost writes | none | 0 Pass | 0 Pass |
| Monthly cost | 16 USD/mo | 16 USD/mo | |
| Error rate | 0% | 0% | |
| Level rules | none broken | Board is not complete Fail | Pass |
Key numbers
| Number | Value |
|---|---|
| Messages per second, one queue | 20,000 ops/s |
| Blocks per second, one indexer part | 10,000 ops/s |
| Indexer step, median | 0.3 ms |
2 GPUs · batch 64 · job 1.5 h · 2.1 USD
How it works
Some jobs are made of many independent pieces. An indexer reads one block at a time from a provider, decodes it and stores its events. The blocks do not depend on each other, so many workers can take different blocks at once.
In the mini board an indexer must backfill 1,000,000 blocks within 24 hours. Each block needs one call that takes 250 ms, and the decode and write add 20 ms. One worker waits 270 ms for each block, so it indexes 3.7 blocks/s. A million blocks need about 3 days by this arithmetic, and in the run one worker misses the 24-hour limit.
A queue holds every block number, and several workers take blocks from it. Each worker still waits 250 ms for each call. But four workers together index about 15 blocks/s, and the backfill finishes in about 18 hours. A block leaves the queue only after its events are stored, so the index has no gaps.
The gain stops at a shared limit. The provider allows 25 calls/s. About 7 workers already reach that. A tenth worker adds nothing, because the provider rejects the extra calls.
So the rule has two parts. Split the work into independent pieces. Then find the first shared resource that the workers need, and stop adding workers when you reach its limit.
In the levels you scale the workers up to the limit of the part that they share.