GPU vs CPU for boosted trees
A GPU pays off with large batches, and a batch adds waiting time. A tree model with 400 trees takes about 1 ms on one CPU core. At 20,000 scores/s, one GPU server with batches of 256 gives p99 44.5 ms for 552 USD/mo. Eight CPU replicas give p99 5.3 ms for 160 USD/mo.
Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s
1,500 req/s · Node API · 3 hosts · 18 USD/mo
Case 1
A fraud model with 400 boosted trees scores 20,000 payments/s. One score takes about 1 ms on one CPU core. p99 must stay below 25 ms and cost below 600 USD/mo.
| Measure | Goal | One GPU server, batches of 256 | 8 CPU replicas, 4 vCPU each |
|---|---|---|---|
| p99 latency | under 25 ms | 45 ms Fail | 5.3 ms Pass |
| Error rate | under 0.1% | 0% Pass | 0% Pass |
| Monthly cost | 552 USD/mo | 160 USD/mo |