Backlog drain rate
The drain rate is the backlog divided by the deadline, plus the new arrivals. 2 million jobs in 10 minutes need at least 3,334 jobs/s above the arrival rate. The slowest part of the worker path sets the real limit.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | One consumer | 100 consumers and 2 worker replicas |
|---|---|---|---|
| Completion time | under 600 s | 650 s Fail | 416.27 s Pass |
| Lost writes | none | 0 Pass | 0 Pass |
| Monthly cost | 32 USD/mo | 48 USD/mo | |
| Error rate | 0% | 0% | |
| Level rules | none broken | Board is not complete Fail | Pass |
Key numbers
| Number | Value |
|---|---|
| Messages per second, one queue | 20,000 ops/s |
| CPU time for one worker job | 0.3 ms |
| Jobs per second, one worker part | 10,000 ops/s |
2 consumers · 6,154 jobs/s, 3,334 needed · drained in 325 s, deadline 600 s
How it works
A backlog is work that already waits. After an outage or a burst, the queue holds millions of jobs, and a deadline says when they must be done. The first step is a division: 2 million jobs in 600 s is 3,333 jobs/s. New jobs keep arriving during the drain, so add the arrival rate to that.
Then check each stage. In the mini board a queue holds 2 million jobs. One consumer cannot reach the needed rate, and the drain takes about 650 s, which is more than the deadline of 600 s. More consumers raise the rate. But more consumers help only while the other parts have room: worker CPU, disk, partitions and the service the workers call.
The workers use about 0.3 ms of CPU for each job. At 3,334 jobs/s that is one full vCPU, plus headroom. So the worker pool needs several replicas, not one. The part with the lowest rate on the path decides the speed of the whole path.
Cost enters here. Capacity for a drain is needed for a short time. Find the lowest setting that meets the deadline, and scale back when the queue is empty.
In the levels you set the consumers and the worker capacity for the rate that the deadline needs.