Load shedding
Load shedding drops low-value work first when demand passes capacity. Here browsing is capped at 300 requests/s during a sale, so checkout keeps all 600 requests/s with p99 under 50 ms. Equal limits for all traffic cannot protect the important requests.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Browsing and checkout share the queue | Browsing capped at 300 requests/s |
|---|---|---|---|
| p99 latency (Checkout) | under 300 ms | 10.1 s Fail | 45 ms Pass |
| Availability (Checkout) | at least 99.9% | 84.4% Fail | 100% Pass |
| Availability (Browse) | at least 45% | 82% Pass | 53.7% Pass |
| Monthly cost | 8 USD/mo | 9 USD/mo | |
| Error rate | 15.6% | 0% |
Key numbers
| Number | Value |
|---|---|
| Limiter check, p99 | 0.5 ms |
| Checks per second, one limiter | 10,000 ops/s |
| Default bucket depth | 1,000 tokens |
Buffer 1,000 jobs · last job waits 1.0 s · 17% turned away
How it works
A service has a fixed capacity. When demand is above it, something must wait or fail. If nothing decides, the queue grows, and every request gets slow, including the ones that matter most.
Load shedding decides in advance. It ranks the work by value and rejects the lowest first. A rejected request costs almost nothing, so the CPU goes to the work that stays.
In the mini board a sale starts at 0:20. Checkout rises to 600 requests/s and browsing to 1,000 requests/s. The shop API serves about 1,000 requests/s. Both request types share one queue, so the surge of browsing delays checkout too. A limiter on the browse path admits about 300 requests/s and rejects the rest. Checkout keeps all 600 requests/s, and the shop stays near 90 percent busy.
One limiter for both types does not work. An equal share large enough for checkout also lets too much browsing through. Put the limit on the class that can be shed.
Shedding is a trade. The shed users see errors. Choose what you may lose, and give those users a clear answer, for example a page that asks them to try again.
In the levels you cap the low-value class and keep the important class at full rate.