Load balancing
A load balancer spreads requests across several servers and stops sending to a server that fails. With a check every 5 s and 3 misses to mark a server down, a dead server stays in the pool for about 15 s. During that time, some requests still go to it.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | No balancer in front of the servers | Layer 7 balancer in front of 3 servers |
|---|---|---|---|
| p99 latency | under 200 ms | 26 ms Pass | 25 ms Pass |
| Availability | at least 99.5% | 83.3% Fail | 99.913% Pass |
| Monthly cost | 12 USD/mo | 14 USD/mo | |
| Error rate | 16.7% | 0.087% | |
| Level rules | none broken | Board is not complete Fail | Pass |
Key numbers
| Number | Value |
|---|---|
| Requests per second, layer 7 balancer | 10,000 ops/s |
| Operations per second, layer 4 balancer | 1,000,000 ops/s |
| Time between health checks, default | 5 s |
| Failed checks before a server goes out | 3 checks |
| Price of one load balancer | 4 USD/mo |
5 servers · 72% busy · p99 66 ms
How it works
One server is a limit and a single point of failure. A load balancer gives clients one address and sends each request to one of several servers behind it. Round robin sends to each server in turn. Least connections sends to the server with the fewest open requests, which helps when some requests are slow. A hash of a key sends the same user or key to the same server each time.
The balancer also checks the health of each server. When a server misses enough checks in a row, the balancer stops using it. Fast checks find a failure sooner, but very fast checks can remove a server that is only busy for a moment.
A balancer adds no capacity of its own. When 1 of 3 servers fails, the other 2 must carry the full load, so each needs 50 percent more room than its normal share. A layer 7 balancer reads each HTTP request and can route by path or header. A layer 4 balancer only moves connections, so it does less work: in the catalog it handles 1,000,000 operations per second, against 10,000 for layer 7.
In the levels you put a balancer in front of servers, tune its health checks, and size each server so that the others can take over.