Batch size on one GPU

  • Tune
  • Agent 03

Difficulty 2 of 5

One H100 serves an 8B model. Chats arrive at 8/s. Each has 500 input and 200 output tokens. Set only the batch size. Keep p99 below 7 s and serve at least 7 chats/s.

Play level

GPU server (H100) · 16 vCPU · 80 GB1,900 USD/moHTTP/2Users8 req/sAPI server1 vCPU · 1 GBLLM server1 vCPU · 1 GB
GPU server (H100)1,900 USD/moHTTP/2HTTP/2Users8 req/sAPI server1 vCPU · 1 GBLLM server1 vCPU · 1 GB
The board at the start of the level
A batch shares one read of the weightsDrag or use arrow keys to change the batch size

Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s

Goals and limits

p99 latency
under 7 s
Throughput
at least 7 req/s
Monthly cost
under 2,000 USD/mo
Hosts
16 vCPU · 80 GB
Traffic
8 req/s

Concepts

How players did

Player results load from the server.

All levels