Shared dependency cascade
A cascade spreads through shared parts. Search and checkout shared one database, so a 2x slowdown took both down. A separate API and replica for search keep checkout p99 near 30 ms.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Search and checkout share one database | Search has its own API and replica |
|---|---|---|---|
| p99 latency (Checkout) | under 150 ms | 10.1 s Fail | 29 ms Pass |
| Availability (Checkout) | at least 99.9% | 85.2% Fail | 100% Pass |
| Availability (Search) | at least 99% | 84.9% Fail | 100% Pass |
| Monthly cost | 24 USD/mo | 36 USD/mo | |
| Error rate | 14.8% | 0% |
Key numbers
| Number | Value |
|---|---|
| Reads per second, one database vCPU | 15,000 reads/s |
| Default call timeout | 10,000 ms |
| Node API call, median | 1 ms |
Two rows · 3 of 3 replicas up · load served
How it works
A failure rarely stays where it starts. A slow part makes the callers wait, the waiting callers hold their workers, and the load backs up to other callers. The harm travels along every shared part. This is a cascade.
In the mini board search and checkout share one database. Each search scans 50 times more rows than a checkout read, so search uses most of the database. At 0:20 the database runs at half speed for 40 s. The shared queue grows. Checkout requests wait in the same queue as search, time out, and fail. Checkout falls with search, though it needs less than 10 percent of the database.
A bulkhead cuts the shared path. Search gets its own API and its own replica. The slow database then serves only checkout. Checkout needs little, so even at half speed it keeps p99 near 30 ms and availability at 99.9 percent. Search still suffers, and the limit for search is lower, at 99 percent.
To find cascades, draw each shared part, and ask what happens to each caller when that part slows. Look for a heavy caller next to a critical one. Also check the timeouts: long timeouts let a slow part hold workers for a long time.
In the levels you separate the heavy path from the critical path.