Failure domains
A failure domain is a group of parts that one fault can stop at the same time. Two servers in one zone both stop when that zone fails, so 2 copies give no protection. Put copies in different domains, and size each domain to carry the full load alone.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Both servers in zone A | Servers in two zones with fast health checks |
|---|---|---|---|
| Availability | at least 97.5% | 26.7% Fail | 99.368% Pass |
| p99 latency | under 150 ms | 30 ms Pass | 34 ms Pass |
| Monthly cost | 44 USD/mo | 48 USD/mo | |
| Error rate | 73.3% | 0.6% |
Key numbers
| Number | Value |
|---|---|
| Round trip in one zone | 0.5 ms |
| Round trip between zones | 1.5 ms |
| Time between balancer health checks | 5 s |
| Failed checks before a server goes out | 3 checks |
Two rows · 3 of 3 replicas up · load served
How it works
A host, a rack, a power feed, a zone and a region are all failure domains. A fault in one of them stops every part inside it. Two copies of a server on the same host give no protection from a host fault. Two copies in the same zone give no protection from a zone fault.
To survive a fault, put copies in different domains. The balancer, the database and the servers behind them must all span the domains. One part that runs only in zone A can stop the whole system when zone A fails.
This has costs. A round trip between zones takes about 1.5 ms, against 0.5 ms inside one zone, and traffic between zones often costs money. Each zone must also carry the full load after the other zone fails. If two zones each run at 75 percent in normal use, the zone that survives must run at 150 percent, so it fails too. Plan for each zone to carry everything.
Detection time also matters. With a check every 5 s and 3 misses in a row, a balancer needs about 15 s to remove a dead zone.
In the levels you spread parts across hosts, racks and zones, and size each domain to run alone.