Durable execution
Durable execution saves each step when it finishes. A 10 s crash repeated 120 finished steps without a journal and none with it. A step that was running at the crash can still run again, so make each step safe to retry.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Progress only in memory | Journal of finished steps |
|---|---|---|---|
| Duplicates | none | 120 Fail | 0 Pass |
| Completion time | under 42 s | 46.74 s Fail | 33.84 s Pass |
| Lost writes | none | 0 Pass | 0 Pass |
| Data mismatches | none | 0 Pass | 0 Pass |
| Monthly cost | 36 USD/mo | 36 USD/mo | |
| Error rate | 0% | 0% |
Key numbers
| Number | Value |
|---|---|
| Steps per second, one scheduler | 10,000 ops/s |
| Scheduler step, median | 0.3 ms |
| Time for a scheduler to boot | 90 s |
Backup every 60 minutes · RPO 60 minutes · RTO 11 hours at 100 Mbps
How it works
A long task has many steps: plan, search, read, draft, file a ticket. Each step can take seconds and cost money, because it calls a model or an outside service. A crash in the middle is not rare. If the runner keeps its progress only in memory, a crash erases it.
In the mini board forty research tasks each run five steps. The agent host crashes at 0:15 for 10 s. After the restart, each open task starts again at step 1. The steps that were already finished run a second time. In this run, 120 finished steps ran twice. The work was not only slow: a repeated step can send a second ticket or a second email. The batch also ended late.
Durable execution keeps a journal. When a step finishes, the runner writes its result to durable storage. After a restart the runner reads the journal. Each task continues at its next step, and it reuses the saved results of the earlier steps. No finished step repeats, and every task finishes by 0:42.
A small gap remains. A step that was running at the moment of the crash did not write its result, so it runs again. Make each step safe to repeat, for example with an idempotency key.
The journal has a cost: each step does one write.
In the levels you make the runner durable and count repeated steps.