Run evals off the serving path

Agent 21
1,900 USD/mo1,900 USD/moHTTP/2Users8 req/sAPI server1 vCPU · 1 GBLLM server1 vCPU · 1 GBPartner apps0 req/sWorker1 vCPU · 1 GBLLM API1 vCPU · 1 GB
HTTP/2Users8 req/sAPI ser…LLM ser…Partner…0 req/sWorkerLLM API

Turn every goal green.

Traffic
8 req/s · reads only
Limit
API server answers about 3,300 req/s
Bill
  • GPU server (H100) × 23,800 USD/mo
  • API server4 USD/mo
  • Worker4 USD/mo
  • LLM API4 USD/mo
3,812 of 3,900 USD/mo

Goals

Not tested yet

Time until all the work is done.99 of 100 requests finish faster than this.The share of requests that fail or get no answer.What the servers and services cost each month.

Each prompt change runs 800 eval cases. Each case needs one candidate answer and one judge score. The judge allows 9 calls/s. Chat p99 must stay under 6.5 s, and a run must end within 2.5 min. The GPU host gpu-b is free for the candidate.

One change feed keeps serving in stepTap or press Space to change the write paths

Two write paths · 4 of 16 served features differ from training

Parts you can add: Message broker, LLM server. At most 3 parts. Budget: 3,900 USD/mo.

Tap a part to add it, or drag it onto a trace.Click a part to add it, or drag it onto a trace.