Round robin vs least connections
Round robin gives each backend the same number of requests. Least connections sends new work to the backend with the fewest requests in progress. When one backend needs 50 ms of CPU per request and the other 1 ms, only least connections keeps p99 under 350 ms.
Round robin · slow backend 150 requests/s · its CPU 375%
Least connections · slow backend 6 requests/s · its CPU 15%
Case 1
300 requests/s go to two backends with 2 vCPU each. One backend needs 50 ms of CPU per request, the other needs 1 ms.
| Measure | Goal | Round robin | Least connections |
|---|---|---|---|
| p99 latency | under 350 ms | 10.1 s Fail | 88 ms Pass |
| Error rate | under 0.1% | 40% Fail | 0% Pass |
| Monthly cost | 18 USD/mo | 18 USD/mo |
Case 2
The same two backends at 60 requests/s. The slow backend now has spare CPU, so both rules meet the target.
| Measure | Goal | Round robin | Least connections |
|---|---|---|---|
| p99 latency | under 350 ms | 272 ms Pass | 96 ms Pass |
| Error rate | under 0.1% | 0% Pass | 0% Pass |
| Monthly cost | 18 USD/mo | 18 USD/mo |