Inference
Inference is the work of running a trained model to answer a request. One H100 GPU host costs 1,900 USD/mo, so each answer must use it well. A larger batch serves more requests per second, but each answer takes longer.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Batch size 4 | Batch size 24 |
|---|---|---|---|
| p99 latency | under 7 s | 10.1 s Fail | 6 s Pass |
| Throughput | at least 7 req/s | 4 req/s Fail | 8 req/s Pass |
| Monthly cost | 1,900 USD/mo | 1,900 USD/mo | |
| Error rate | 51.8% | 0.01% |
Key numbers
| Number | Value |
|---|---|
| Tokens per second, one model server | 2,500 tokens/s |
| Time to the first token, default | 150 ms |
| Batch size, default | 32 requests |
| Largest batch size, default | 64 requests |
| Price of one H100 GPU host | 1,900 USD/mo |
Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s
How it works
A language model writes its answer one token at a time. Each decode step reads all the model weights from GPU memory. For one request, most of that read gives only one token. A batch lets one step make a token for many requests, so the GPU does much more useful work for the same memory traffic.
Continuous batching adds new requests to the running batch at each step, and does not wait until the whole batch ends. This keeps the GPU busy when answers have different lengths.
Batch size is a trade-off between throughput and latency. A small batch gives fast answers but serves few requests, so a queue grows when traffic is high. A large batch serves more, but each request shares each step with more neighbors and ends later. Above some size, the time for each step grows fast and p99 goes over the target. GPU memory is a hard limit too: each request in the batch keeps its context in a cache that grows with each token.
Two numbers describe what the user feels: the time to the first token, and the rate of tokens after it. Measure both at p99.
In the levels you set the batch size so that one GPU meets both the throughput target and the latency target.