Retrieve then rank

  • Build
  • Model 12

Difficulty 3 of 5

Home recommendations come from a catalog of 100,000 items. The neural ranker scores 500 items in about 3 ms. Today the API sends it the whole catalog in 200 calls per request. Serve 2,000 requests/s with p99 under 80 ms. Budget 100 USD/mo.

Play level

HTTP/2gRPCUsers2,000 req/sAPI server2 vCPU · 2 GBModel (CPU)8 vCPU · 8 GB
HTTP/2gRPCUsers2,000 req/sAPI server2 vCPU · 2 GBModel (CPU)8 vCPU · 8 GB
The board at the start of the level
A batch shares one read of the weightsDrag or use arrow keys to change the batch size

Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s

Goals and limits

p99 latency
under 80 ms
Error rate
under 0.1%
Monthly cost
under 100 USD/mo
Parts you can add
Vector DB, Key-value store, Redis cache, Search index
Part limit
4
Traffic
2,000 req/s

Concepts

How players did

Player results load from the server.

All levels