Model replicas
A model server completes a fixed number of tokens each second. Batching raises that number, and more GPU replicas multiply it. Here two H100 GPUs at batch size 32 serve 50 chats/s with p99 under 3 s.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Two GPUs, batch size 8 | Two GPUs, batch size 32 |
|---|---|---|---|
| p99 latency | under 3.5 s | 10.1 s Fail | 2.9 s Pass |
| Throughput | at least 45 req/s | 32 req/s Fail | 50 req/s Pass |
| Monthly cost | 3,808 USD/mo | 3,808 USD/mo | |
| Error rate | 36.4% | 0% |
Key numbers
| Number | Value |
|---|---|
| One H100 GPU host | 1,900 USD/mo |
| Default token rate, one model server | 2,500 tokens/s |
| Default batch size, one model server | 32 requests |
2 GPUs · batch 64 · job 1.5 h · 2.1 USD
How it works
A GPU serves a chat model at a steady token rate. Each chat needs input tokens and output tokens, so each GPU completes a fixed number of chats each second. When the demand is above that number, the queue grows, and p99 grows with it.
Two things raise the number. The first is batching. A GPU can process many requests in the same step, because the cost of loading the model weights is shared. In the mini board one H100 at batch size 8 completes about 18 chats/s, so two of them fall short of the 45 chats/s that the level needs, and p99 reaches 10 s in the run. At batch size 32 each GPU completes about 28 chats/s. A bigger batch also makes each request wait longer, so keep it inside the latency limit.
The second is replication. Another GPU holds another copy of the model and serves its own share of the requests. A balancer splits the chats evenly. In the mini board two H100 GPUs at batch size 32 carry 50 chats/s with spare room, and p99 stays under 3.5 s. One GPU cannot carry the load alone.
Replicas scale well because the requests are independent. They cost a full GPU each, here 1,900 USD/mo, so find the smallest number that meets the target.
In the levels you set the batch size and the number of replicas for the demand.