Protect a fixed GPU fleet at launch

Agent 24
1,900 USD/mo1,900 USD/moHTTP/2Users16 req/sAPI server1 vCPU · 1 GBLoad balanc…1 vCPU · 1 GBLLM server1 vCPU · 1 GBLLM server1 vCPU · 1 GB
1,900 USD/mo1,900 USD/moHTTP/2HTTP/2Users16 req/sAPI serverLoad balanc…LLM serverLLM server

Keep every goal green in each wave.

Traffic
16 req/s · up to 64 req/s · reads only
Limit
API server answers about 3,300 req/s
Bill
  • GPU server (H100) × 23,800 USD/mo
  • API server4 USD/mo
  • Load balancer (L7)4 USD/mo
3,808 of 3,812 USD/mo

Goals

Not tested yet

The share of time when every target was met.99 of 100 requests finish faster than this.The share of requests that fail or get no answer.What the servers and services cost each month.

A chat app launches on 2 H100 GPUs (about 24 chats/s). Demand rises from 16 to 64 chats/s in 4 waves, and the last wave has a 10 s burst at 6 times the rate. Keep p99 under 7 s and errors under 15 percent for 90 percent of the run. Budget: 3,812 USD/month, plus 4 per wave.

Parts you can add: Semantic cache, Redis cache, Rate limiter. At most 2 parts. Budget: 3,812 USD/mo.

Tap a part to add it, or drag it onto a trace.Click a part to add it, or drag it onto a trace.