Kill switch delivery
A kill switch must be fast and must not be a single point of failure. Reading the flag from the store for each request failed 20 percent of requests in a 20 s restart. A cache with pushed changes served every request and cut late replies from 150,000 to 600.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Every check reads the flag store | Local caches and pushed changes |
|---|---|---|---|
| Error rate | under 1% | 20% Fail | 0% Pass |
| Data mismatches | under 3,000 | 0 Pass | 600 Pass |
| p99 latency | under 100 ms | 27 ms Pass | 26 ms Pass |
| Monthly cost | 24 USD/mo | 34 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Local cache read, median | 0.1 ms |
| Change bus hop, median | 0.5 ms |
| Default time to live of a local cache | 60 s |
Canary · 5% of requests fail for 15 seconds · 750 failed requests
How it works
A feature flag turns a feature on or off without a new release. A kill switch is a flag that you flip when the feature is hurting users. Two things must be true. The flag must reach every server fast, and the flag system must not take the site down when it fails.
The simple design reads the flag from the flag store on every request. In the mini board three API instances check the flag on each of 3,000 requests/s. The flag store restarts for 20 s at 0:20. During that time every check fails, and 20 percent of requests fail with it.
A better design puts a cache in each instance. A check reads local memory, so it is fast, and it still works while the store is down. But a cache can hold an old value, so the switch must reach it. The flag store publishes each change on a bus, and each cache updates at once. At 0:50 you turn the flag off, and every instance stops the feature within about a second. In the mini board fewer than 3,000 replies run the feature after the switch, and late replies fall from 150,000 to 600.
Pick a safe default for a flag that cannot be read, and keep a time to live as a backup in case a message is lost.
In the levels you deliver the flag with a local copy and pushed changes.