Retries
A retry sends a failed call again, because many faults last less than 1 s. Each retry also adds load to a service that can already be in trouble. A small retry budget with backoff and jitter recovers short faults without a new overload.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | No retries and no backoff | 2 retries with exponential backoff and jitter |
|---|---|---|---|
| Error rate | under 0.1% | 4% Fail | 0% Pass |
| p99 latency | under 1.5 s | 24 ms Pass | 332 ms Pass |
| CPU use | under 80% | 37.5% Pass | 52.5% Pass |
| Monthly cost | 16 USD/mo | 16 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Retries per failed call, default | 0 retries |
| Most retries a part can make | 10 retries |
| First wait before a retry, default | 100 ms |
| Random part of each wait, default | 0.2 fraction |
3 retries per call · no jitter · peak 6.0× normal load
How it works
Many faults are short: a lost packet, a restart, a link that fails for half a second. One more try after a short wait often succeeds.
Retries also add load. If each caller retries 3 times when a service is slow, the service gets up to 4 times its normal traffic at the moment when it is weakest. This is a retry storm. A retry budget limits the extra work, for example to 10 percent of normal calls. When the budget is empty, callers fail fast.
The wait between tries is important. Exponential backoff doubles the wait after each failure: 100 ms, then 200 ms, then 400 ms. This gives the fault time to clear. Jitter adds a random part to each wait, so that thousands of clients do not retry at the same moment.
Retry only the work that is safe to do again. A read is safe. A payment needs an idempotency key, or a retry can charge twice. Also count the retries in the total time: three tries with backoff can take longer than the caller is ready to wait.
In the levels you set the number of retries, the backoff and the jitter, so that a short fault recovers without a new overload.