Offline evaluation
An offline eval runs a fixed set of cases through a candidate and scores each answer with a judge before release. 800 cases make 1,600 model calls. On their own GPU, with 64 in flight, a run took 130 s and chat p99 stayed near 5.7 s.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Eval runs on the serving GPU | Eval runs on its own GPU |
|---|---|---|---|
| Completion time | under 150 s | 129.72 s Pass | 129.72 s Pass |
| p99 latency (Chat) | under 6.5 s | 10.1 s Fail | 5.7 s Pass |
| Error rate (*) | under 1% | 68.5% Fail | 0.007% Pass |
| Monthly cost | 3,816 USD/mo | 3,816 USD/mo | |
| Error rate | 68.5% | 0.007% |
Key numbers
| Number | Value |
|---|---|
| One H100 GPU host | 1,900 USD/mo |
| Default token rate, one model server | 2,500 tokens/s |
| Default batch size, one model server | 32 requests |
Two write paths · 4 of 16 served features differ from training
How it works
A change to a prompt or a model can make answers better or worse. You cannot see the effect by trying a few examples. An offline eval gives a repeatable test. It holds a fixed set of cases. Each case runs through the candidate, and a judge scores the answer. You compare the score with the score of the current version before you release.
The eval needs model calls. In the mini board each prompt change runs 800 cases. Each case needs one candidate answer and one judge score, so a run makes 800 candidate calls and 800 judge calls. The judge allows 9 calls/s, which sets a floor of about 88 s for the run.
Where the calls run matters. If the eval runs on the production GPU, it takes capacity from users. Chat p99 goes past 10 s. A separate GPU host removes the clash. On its own host, with about 64 cases in flight, a run ends in 130 s, and chat p99 stays near 5.7 s. With only 32 in flight the run takes 248 s.
Keep the case set stable and versioned, or the scores cannot be compared. Add every bug that you meet to the set, so the same fault does not return.
Offline scores do not replace online tests. They catch obvious regressions early.
In the levels you run the eval off the serving path and finish within the time limit.