Batch inference
Batch inference computes predictions before anyone asks. Here one key-value read replaces 20 ms of ranking at 3,000 views/s. Use online inference only when the inputs change faster than the batch cycle.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Rank 20 items at each view | Read the stored picks |
|---|---|---|---|
| p99 latency | under 50 ms | 10.1 s Fail | 4.1 ms Pass |
| Error rate | under 0.1% | 88.4% Fail | 0% Pass |
| Monthly cost | 40 USD/mo | 24 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Stored row read, median | 0.3 ms |
| Reads per second, one key-value store | 30,000 ops/s |
| Scores per second, one CPU model server | 10,000 ops/s |
Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s
How it works
A model can run in two places in time. Online inference scores each request when it arrives. Batch inference scores all the cases ahead of time and stores the result. A request then reads a stored row.
In the mini board a shop shows 20 picks for each user on its home page. The picks use only yesterday's orders and a catalog that changes once each night. The inputs change once a night, so a score made at 02:00 is still correct at 20:00. Peak traffic is 3,000 views/s, and p99 must stay below 50 ms.
Online ranking scores 20 items for each view, which puts 20 ms of model time on the page path and repeats the same work for each visit. At 3,000 views/s that is a lot of CPU, and the budget of 60 USD/mo cannot pay for it. A nightly job scores each of 20 million users once, about 110 CPU hours, away from the page. Each view then reads one row, and p99 is small.
Batch has limits. A prediction can be a day old. It cannot use what the user did a minute ago, and it covers only the users and items that you scored. If the inputs change fast, such as a live session, online scoring is needed. Many systems mix both: batch for the base list, and a light online step for fresh context.
In the levels you decide when the prediction can be made and where it is stored.