Serve one model on two GPUs

Agent 05
GPU server (H100)1,900 USD/moGPU server (H100)1,900 USD/moHTTP/2Users50 req/sAPI server1 vCPU · 1 GBLLM server1 vCPU · 1 GB
1,900 USD/mo1,900 USD/moHTTP/2HTTP/2Users50 req/sAPI serverLLM server

Turn every goal green.

Traffic
50 req/s · reads only
Limit
API server answers about 3,300 req/s
Bill
  • GPU server (H100) × 23,800 USD/mo
  • API server4 USD/mo
3,804 of 3,900 USD/mo

Goals

Not tested yet

99 of 100 requests finish faster than this.Requests served each second.What the servers and services cost each month.

Chats arrive at 50/s with 300 input and 60 output tokens. One H100 serves the 8B model at 16-bit with batch size 8. A second H100 is empty. Keep p99 below 3.5 s and serve 45 chats/s within 3,900 USD/month.

A batch shares one read of the weightsDrag or use arrow keys to change the batch size

Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s

Parts you can add: LLM server, Load balancer (L7). At most 3 parts. Budget: 3,900 USD/mo.

Tap a part to add it, or drag it onto a trace.Click a part to add it, or drag it onto a trace.