Rack and power spread
A rack switch and a power feed are different failure domains. Place 3 replicas so that any 1 failure takes at most 1 of them, and 2 stay up. The load needs 2 replicas, so the service stays inside its limits.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Two replicas in one rack | One replica in each rack |
|---|---|---|---|
| p99 latency | under 50 ms | 568 ms Fail | 25 ms Pass |
| Error rate | under 1% | 9.5% Fail | 0.5% Pass |
| Monthly cost | 229 USD/mo | 229 USD/mo | |
| Error rate | 9.5% | 0.5% |
Key numbers
| Number | Value |
|---|---|
| Reads per second, one database vCPU | 30,000 reads/s |
| Default health check interval | 5 s |
| Mean time between failures, one replica | 8,760 h |
Two rows · 3 of 3 replicas up · load served
How it works
A failure takes out everything that shares a cause. A rack has one switch, so a switch failure takes every server in that rack. A power feed can serve several racks, so a feed failure takes all of them. These are failure domains.
In the mini board three database replicas serve 80,000 lookups/s. Each replica serves about 60,000 lookups/s, so the load needs two. On the given board two replicas share rack 2. At 0:20 the switch of rack 2 fails and takes both replicas. One replica is left, and it cannot carry the load.
The better layout puts one replica in each rack. A switch failure then takes one replica. At 0:40 power feed A fails. Feed A powers only rack 1, because rack 2 also has feed B, so it takes at most one replica. Two of three replicas stay up the whole time, and p99 and errors stay inside their limits.
To plan this, list the shared causes: switch, power feed, cooling, zone. Then check each one: which replicas does it take? If any single cause takes more than the spare capacity, move a replica.
The same rule works in the cloud with zones and with accounts.
In the levels you place replicas on racks and feeds, and you check each failure in turn.