Batch size on one GPU

Agent 03
GPU server (H100) · 16 vCPU · 80 GB1,900 USD/moHTTP/2Users8 req/sAPI server1 vCPU · 1 GBLLM server1 vCPU · 1 GB
GPU server (H100)1,900 USD/moHTTP/2HTTP/2Users8 req/sAPI server1 vCPU · 1 GBLLM server1 vCPU · 1 GB

Tune the parts. Turn every goal green.

Traffic
8 req/s · reads only
Limit
API server answers about 3,300 req/s
Bill
  • GPU server (H100)1,900 USD/mo
1,900 of 2,000 USD/mo

Goals

Not tested yet

99 of 100 requests finish faster than this.Requests served each second.What the servers and services cost each month.

One H100 serves an 8B model. Chats arrive at 8/s. Each has 500 input and 200 output tokens. Set only the batch size. Keep p99 below 7 s and serve at least 7 chats/s.

A batch shares one read of the weightsDrag or use arrow keys to change the batch size

Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s