Run evals off the serving path

  • Build
  • Agent 21

Difficulty 4 of 5

Each prompt change runs 800 eval cases. Each case needs one candidate answer and one judge score. The judge allows 9 calls/s. Chat p99 must stay under 6.5 s, and a run must end within 2.5 min. The GPU host gpu-b is free for the candidate.

Play level

1,900 USD/mo1,900 USD/moHTTP/2Users8 req/sAPI server1 vCPU · 1 GBLLM server1 vCPU · 1 GBPartner apps0 req/sWorker1 vCPU · 1 GBLLM API1 vCPU · 1 GB
HTTP/2Users8 req/sAPI ser…LLM ser…Partner…0 req/sWorkerLLM API
The board at the start of the level
One change feed keeps serving in stepTap or press Space to change the write paths

Two write paths · 4 of 16 served features differ from training

Goals and limits

Completion time
under 150 s
p99 latency (Chat)
under 6.5 s
Error rate (*)
under 1%
Monthly cost
under 3,900 USD/mo
Parts you can add
Message broker, LLM server
Part limit
3
Hosts
2 × 1 vCPU · 80 GB
Traffic
8 req/s

Concepts

How players did

Player results load from the server.

All levels