Two-stage ranking
A two-stage recommender ranks a short list, not the whole catalog. Here retrieval cut ranker work from 200 calls to 1 call for each request, so 8 vCPU serve 2,000 requests/s. The first stage is cheap and broad, and the second is costly and precise.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Ranker scores the whole catalog | Retrieval picks 500, ranker scores them |
|---|---|---|---|
| p99 latency | under 80 ms | 62 ms Pass | 68 ms Pass |
| Error rate | under 0.1% | 99.326% Fail | 0% Pass |
| Monthly cost | 40 USD/mo | 48 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Searches per second, one vector index | 10,000 ops/s |
| Vector search, median | 0.3 ms |
| Default search width of the vector index | 64 candidates |
Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s
How it works
A neural ranker is precise and expensive. It can score about 500 items in 3 ms. A catalog of 100,000 items would need 200 such calls for one request. At 2,000 requests/s that is 400,000 calls/s, far more than the ranker can do.
A two-stage design splits the task. The first stage is retrieval. It uses a cheap method to find a few hundred items that may fit the user. A vector index does this well: it holds an embedding for each item and finds the items nearest to the user's taste vector in under 1 ms. The second stage is ranking. The costly model scores only the short list, in one call.
In the mini board the API sends the ranker the whole catalog in 200 calls for each request. about 99 percent of requests fail, and the limit is p99 80 ms. With retrieval first, the ranker gets about 500 items in 1 call. The work for each request falls by 200 times, 8 vCPU serve 2,000 requests/s, and the budget of 100 USD/mo is enough.
The weak point is recall. If retrieval does not find a good item, the ranker cannot recover it. Measure how often the best items are in the short list, and widen the search if they are not.
In the levels you add the retrieval step and check the calls to the ranker.