Send each request type to its own model

Agent 07
520 USD/mo520 USD/mo1,900 USD/mo1,900 USD/moHTTP/2HTTP/2Users40 req/sUsers0.4 req/sAPI server1 vCPU · 1 GBLLM server1 vCPU · 1 GBLLM server1 vCPU · 1 GBLLM server1 vCPU · 1 GBLLM server1 vCPU · 1 GB
HTTP/2HTTP/2Users40 req/sUsersAPI ser…LLM ser…LLM ser…LLM ser…LLM ser…

Turn every goal green.

Traffic
Faq40 req/s · reads onlyAnalysis0 req/s · reads only
Limit
API server answers about 3,300 req/s
Bill
  • GPU server (L4) × 21,040 USD/mo
  • GPU server (H100) × 23,800 USD/mo
  • API server4 USD/mo
4,844 of 4,900 USD/mo

Goals

Not tested yet

99 of 100 requests finish faster than this.99 of 100 requests finish faster than this.The share of requests that fail or get no answer.What the servers and services cost each month.

Two entry points share one API. FAQ chats arrive at 40/s with short prompts. Analyses arrive at 0.4/s with 3,000-token prompts and need the 70B model. You have 2 small-model GPUs (2k-token context) and 2 large-model GPUs. FAQ p99 under 2.5 s, analysis p99 under 40 s.

A batch shares one read of the weightsDrag or use arrow keys to change the batch size

Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s

Parts you can add: API server, Load balancer (L7). At most 3 parts. Budget: 4,900 USD/mo.

Tap a part to add it, or drag it onto a trace.Click a part to add it, or drag it onto a trace.