Warm standby region
A warm standby is a second region that already runs an API and holds a database replica. The failover needs 3 failed checks at 10 s plus the DNS lifetime: 330 s with a 300 s TTL, 90 s with a 60 s TTL. An async replica loses about 1 s of writes.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | One region only | Warm API and replica in a second region |
|---|---|---|---|
| Availability | at least 83.3% | 16.7% Fail | 95% Pass |
| p99 latency | under 250 ms | 35 ms Pass | 188 ms Pass |
| Lost writes | under 900 | 0 Pass | 15 Pass |
| Monthly cost | 17 USD/mo | 29 USD/mo | |
| Error rate | 83.3% | 5% |
Key numbers
| Number | Value |
|---|---|
| Default answer lifetime of GeoDNS | 300 s |
| Default health check interval of GeoDNS | 10 s |
| Time for a new API replica to boot | 90 s |
DNS TTL 60 s · users move after 90 s · limit 300 s
How it works
Recovery from the loss of a whole region has two numbers: how much time it takes, and how much data you lose. The first is the recovery time. The second is the recovery point. Both depend on what you keep ready before the loss.
In the mini board all traffic runs in eu-central. At 5:00 the whole region is lost for good. The system must recover in eu-west within 5 minutes, which is 83.3 percent availability over 30 minutes, and it may lose at most 1 minute of writes. With no second region, nothing serves orders after the loss.
A warm standby keeps a running API and a database replica in eu-west. The replica receives the writes with a lag of about 1 s, so a failover loses about 1 s of data. The API is already booted: a cold API replica would need 90 s before it serves.
Users move by DNS. GeoDNS keeps users on eu-central while it is healthy. After the loss it needs 3 failed checks at 10 s each, and then the TTL for old answers to expire. With a 300 s TTL that is 330 s, which is too slow. With a 60 s TTL it is 90 s, and the target is met.
A standby costs money all the time. Practice the failover, because a plan that you never run often fails.
In the levels you build the standby and tune the DNS lifetime to meet the recovery time.