Bulkheads
A bulkhead gives each kind of work its own capacity. When batch work rises from 600 to 2,400 requests/s, a shared pool delays checkout. Separate workers keep checkout p99 below 150 ms.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | One shared worker slot | Separate workers for batch |
|---|---|---|---|
| p99 latency (Checkout) | under 150 ms | 288 ms Fail | 24 ms Pass |
| Error rate (Checkout) | none | 4.7% Fail | 0% Pass |
| p99 latency (Batch) | under 300 ms | 336 ms Fail | 25 ms Pass |
| Error rate (Batch) | none | 9.2% Fail | 0% Pass |
| Monthly cost | 12 USD/mo | 12 USD/mo | |
| Level rules | none broken | Board is not complete Fail | Pass |
Key numbers
| Number | Value |
|---|---|
| Default worker pool of one service | 64 workers |
| Requests per second, one API server | 10,000 ops/s |
| CPU time for one API call | 0.3 ms |
Separate pools · batch at 2,400 requests/s holds 0% of the checkout workers
How it works
A ship has walls inside its hull. If one room floods, the water stays in that room. A bulkhead in software does the same for capacity. Each kind of work gets its own pool of workers, connections or queue slots, so a problem in one pool cannot spend the capacity of another.
In the mini board checkout and batch work share one worker slot. At 0:10 batch traffic rises from 600 to 2,400 requests/s for 20 s. The batch requests fill the slot, and checkout requests wait behind them. The checkout p99 passes its 150 ms limit, and some requests fail.
With a separate set of workers for batch, the surge stays in the batch pool. Batch p99 may rise, but it has a limit of 300 ms, and checkout stays under 150 ms. The two kinds of work no longer share a failure.
The price is idle capacity. Two pools that are each sized for their own peak hold more capacity than one shared pool. Share only what is safe to share, and give a separate pool to work that is critical, slow or likely to surge.
Bulkheads also work for a shared dependency. If search and checkout share one database, a slow database takes both down. Give search its own replica, and checkout can stay up.
In the levels you separate the work and check that the critical path stays inside its limits.