Serve one model on two GPUs

  • Build
  • Agent 05

Difficulty 3 of 5

Chats arrive at 50/s with 300 input and 60 output tokens. One H100 serves the 8B model at 16-bit with batch size 8. A second H100 is empty. Keep p99 below 3.5 s and serve 45 chats/s within 3,900 USD/month.

Play level

GPU server (H100)1,900 USD/moGPU server (H100)1,900 USD/moHTTP/2Users50 req/sAPI server1 vCPU · 1 GBLLM server1 vCPU · 1 GB
1,900 USD/mo1,900 USD/moHTTP/2HTTP/2Users50 req/sAPI serverLLM server
The board at the start of the level
A batch shares one read of the weightsDrag or use arrow keys to change the batch size

Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s

Goals and limits

p99 latency
under 3.5 s
Throughput
at least 45 req/s
Monthly cost
under 3,900 USD/mo
Parts you can add
LLM server, Load balancer (L7)
Part limit
3
Hosts
2 × 1 vCPU · 80 GB
Traffic
50 req/s

Concepts

How players did

Player results load from the server.

All levels