CPU or GPU
Pick inference hardware from the work for each request. A 1 ms tree model fits on CPU: 32 vCPU serve 20,000 scores/s at p99 5 ms, for less than one third of the price of a 520 USD/mo GPU server.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | One GPU server, no batching | Eight CPU replicas |
|---|---|---|---|
| p99 latency | under 25 ms | 45 ms Fail | 5.3 ms Pass |
| Error rate | under 0.1% | 0% Pass | 0% Pass |
| Monthly cost | 552 USD/mo | 160 USD/mo |
Key numbers
| Number | Value |
|---|---|
| One L4 GPU host | 520 USD/mo |
| One 4 vCPU server | 24 USD/mo |
| Scores per second, one CPU model server | 10,000 ops/s |
Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s
How it works
A GPU is fast at large, regular work, such as matrix products. It is not faster at everything. A model of boosted trees makes many small decisions and does little arithmetic. It runs well on a CPU.
In the mini board a fraud model has 400 boosted trees and scores 20,000 payments/s. One score takes about 1 ms on one CPU core, so 20,000 scores/s need about 20 vCPU. Eight replicas with 4 vCPU each cost 128 USD/mo and keep p99 near 5 ms. The limit is 25 ms, and the budget 600 USD/mo.
A GPU host costs 520 USD/mo for the L4 class. It is efficient only when a batch is large, so many requests are scored together. Each request then waits until its batch fills, which adds latency. Without batches, the GPU does one small request at a time and falls behind. In the mini board the GPU design misses the p99 limit.
The decision rule: measure the work for each request. If one score takes about 1 ms on a CPU core, count the cores. If the model is a large neural network with billions of parameters, a CPU cannot reach the speed, and you need a GPU and batching.
Price both choices at your peak load, and include the idle time.
In the levels you pick the hardware from the work for each request and compare the cost.