Recovery pile-up
A slow call holds a worker, and retries add work during the fault. A deadline of 100 ms with no retries restores the service in 2 edits. A circuit breaker also stops failed calls, but it can reject some calls after the recovery.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | 10 s timeout, 4 immediate retries | 100 ms deadline, no retries, breaker |
|---|---|---|---|
| p99 latency | under 350 ms | 3.5 s Fail | 102 ms Pass |
| Availability | at least 85% | 97% Pass | 89% Pass |
| Monthly cost | 8 USD/mo | 8 USD/mo | |
| Error rate | 3% | 11% |
Key numbers
| Number | Value |
|---|---|
| Default call timeout | 10,000 ms |
| Default worker pool | 64 workers |
| Default breaker cooldown | 30 s |
Backup every 60 minutes · RPO 60 minutes · RTO 11 hours at 100 Mbps
How it works
A fault can end and still leave the system broken. This happens when the response to the fault keeps the load high. Two causes are common: slow calls and retries.
A slow call holds a worker until it ends. In the mini board a dependency answers 2 s late for 1 s, and later it fails for 1 s. The default call timeout is 10 s, so each stuck call holds one of 32 workers for a long time. The workers fill up, and new requests wait in a buffer.
Retries add to the pile. The callers retry 4 times with no delay, so each failed request becomes five. The service gets a flood of work at the moment it tries to recover. Calls pile up after the dependency returns, and the order screen stalls twice.
Two changes fix most of it. A deadline of 100 ms frees a worker quickly, so a slow call cannot hold it. Zero retries stop the multiplication of work. Both together meet the limits in two edits.
A third edit adds a circuit breaker. It stops most calls during the outage. But it also rejects some calls after the recovery, while it is open, and availability falls by a point in this run. Both designs pass.
The rule is: bound the time of each call, and bound the work that a failure can create.
In the levels you find the causes of the pile-up and fix them in a few edits.