Least connections
A balancing rule decides where new work goes. Round robin gives each backend the same count of requests. Least connections sends each request to the backend with the fewest open ones, so it moves load away from a backend where requests take 50 ms and not 1 ms.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Round robin | Least connections |
|---|---|---|---|
| p99 latency | under 350 ms | 10.1 s Fail | 88 ms Pass |
| Error rate | under 0.1% | 40% Fail | 0% Pass |
| Monthly cost | 18 USD/mo | 18 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Requests per second, one HTTP balancer | 10,000 ops/s |
| Balancer hop, median | 0.3 ms |
| HTTP balancer price | 4 USD/mo |
Least connections · slow backend 6 requests/s · its CPU 15%
How it works
Equal counts are fair only when requests cost the same. Round robin sends the first request to backend A, the next to B, and so on. If a request on B takes 50 times as long as one on A, B fills with work while A sits idle.
In the mini board a balancer sends 300 requests/s to two backends with 2 vCPU each. Measured CPU for each request is 1 ms on one backend and 50 ms on the other. With round robin each backend gets 150 requests/s. The slow one then needs 375 percent of its CPU, so its queue grows without limit and p99 passes 350 ms.
Least connections looks at the open requests of each backend. The slow backend keeps requests open much longer, so it has more of them. New requests go to the fast backend. The load moves to where capacity is free, without any weights.
The rule is not magic. It needs the balancer to see the open requests, so it works best with an HTTP balancer. For long lived connections, such as streams, the count of connections can mislead. Keep the health checks on.
In the levels you choose the routing rule for backends with unequal work.