Graceful degradation
Graceful degradation keeps the main task working with less. In the mini board pages show a list that is up to 2.5 minutes old during a 2-minute crash. Availability stays at 99.9 percent instead of 33 percent.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Page fails when the service fails | Page shows the last good list |
|---|---|---|---|
| Availability | at least 99.5% | 33.3% Fail | 99.901% Pass |
| p99 latency | under 250 ms | 35 ms Pass | 25 ms Pass |
| Data age | under 180 s | 0 ms Pass | 149.9 s Pass |
| Monthly cost | 20 USD/mo | 24 USD/mo | |
| Error rate | 66.7% | 0.1% |
Key numbers
| Number | Value |
|---|---|
| Default stale window of a cache | 0 s |
| Default time to live of a cache entry | 60 s |
| Recovery time after a failure | 30 s |
Fallback cache · availability 99.9% · oldest list 2.5 min, limit 3 min
How it works
A page often depends on several services. If one of them fails, a simple page fails too, even when that service only adds a small part of the screen. Users lose the whole page for a feature that they could skip.
Graceful degradation decides in advance what the page can do without. In the mini board each product page asks a recommendation service for one of 40 category lists. The service crashes at 0:30 and returns at 2:30. Without a fallback, the page fails for the 2 minutes, and availability falls to 33 percent.
The fallback is a cache of the last good list for each category. The cache keeps each entry for 30 s and may serve it for up to 130 s more while it refreshes. During the crash the page shows a list that is up to 2.5 minutes old. Users see a slightly older list, and the page works. Availability stays at 99.9 percent.
The age limit is part of the design. The task allows a list that is up to 3 minutes old. Set the limit from what the user can accept, and do not serve data past it.
Test the fallback on purpose. A fallback that you never run is likely to fail when you need it.
In the levels you add the fallback and check availability and the age of the data.