Log replay
A log keeps events after delivery, so analytics can move its position back and replay them. A work queue keeps each job only until a worker completes it. Here the log replayed 100,000 events in 5 s.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Work queue only | Log with a position for each reader |
|---|---|---|---|
| Replay time | under 10 s | 30 s Fail | 5 s Pass |
| Completion time (Jobs) | under 20 s | 5 s Pass | 5 s Pass |
| Lost writes | none | 0 Pass | 0 Pass |
| Monthly cost | 44 USD/mo | 40 USD/mo | |
| Error rate | 0% | 0% | |
| Level rules | none broken | Board is not complete Fail | Pass |
Key numbers
| Number | Value |
|---|---|
| Events per second, one log | 100,000 ops/s |
| Log append, median | 2 ms |
| Messages per second, one queue | 20,000 ops/s |
13 deliveries · 10 payments applied · 0 charged twice
How it works
Two designs look alike and behave differently. A work queue holds jobs that wait to be done. Each job goes to one worker, and when the worker confirms it, the queue deletes it. A log holds events in order. Readers keep a position, and the log keeps the events after they are read.
The difference matters when you need the past. In the mini board analytics must replay 100,000 saved events within 10 s after a code change. A work queue has already deleted them, so it keeps no history, and in the run the replay takes 30 s. A log still has them. Analytics sets its position back to the start and reads them again. In the run the log replayed 100,000 events in 5 s.
The opposite need is also in the mini board. 100 image jobs take 1 s each and must all finish within 20 s, with no lost job. A work queue fits this. It spreads the jobs over competing workers and tracks each one until it completes. With 20 consumers the 100 jobs finish in 5 s.
So use a log when many readers need the same history in order, and when a reader may need to start again. Use a work queue when each job needs one worker, and the job is done once it is done.
In the levels you choose the design from the way the data is read.