Health check failover
A fallback is only as fast as the detection of failure. With checks every 5 s and 3 failed checks needed, a router needs 15 s to drop a dead provider. A 1 s check cut that to 3 s and kept 98.7 percent of chats.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | One provider, 5 s checks | Second provider, 1 s checks |
|---|---|---|---|
| Availability | at least 98% | 50.6% Fail | 98.7% Pass |
| p99 latency | under 5 s | 3.5 s Pass | 3.7 s Pass |
| Monthly cost | 12 USD/mo | 16 USD/mo | |
| Error rate | 49.4% | 1.3% |
Key numbers
| Number | Value |
|---|---|
| Default health check interval | 5 s |
| Default failed checks before removal | 3 checks |
| Hosted model call, median | 150 ms |
Checks every 1 s · 3 failed checks drop the provider · chats fail for 3 s
How it works
Redundancy helps only if traffic moves to the healthy copy. A router moves it, and the router learns about a failure from health checks. It sends a small probe to each backend. When a number of probes in a row fail, it removes the backend.
The time to detect a failure is the check interval times the number of failed checks. With the defaults of 5 s and 3 checks, a router needs 15 s. For those 15 s, a share of the requests goes to the dead backend and fails.
In the mini board chats go through a router to a hosted model provider. The provider goes down at 0:30 for 60 s. With one provider, every chat fails until it returns. With a second provider and a 1 s check interval, the router drops the dead one in about 3 s. Availability stays at 98.7 percent, and p99 stays under 5 s.
A short interval has costs. More probes mean more load, and a short blip can remove a healthy backend. Use a few failed checks in a row, and use a deeper check that tests a real call. Let callers retry once on a failure, so the requests in the window still work.
In the levels you add the second backend and set the check interval so that detection is fast.