Retrieve then rank

Model 12
HTTP/2gRPCUsers2,000 req/sAPI server2 vCPU · 2 GBModel (CPU)8 vCPU · 8 GB
HTTP/2gRPCUsers2,000 req/sAPI server2 vCPU · 2 GBModel (CPU)8 vCPU · 8 GB

Turn every goal green.

Traffic
2,000 req/s · reads only
Limit
API server answers about 6,700 req/s
Bill
  • API server8 USD/mo
  • Model (CPU)32 USD/mo
40 of 100 USD/mo

Goals

Not tested yet

99 of 100 requests finish faster than this.The share of requests that fail or get no answer.What the servers and services cost each month.

Home recommendations come from a catalog of 100,000 items. The neural ranker scores 500 items in about 3 ms. Today the API sends it the whole catalog in 200 calls per request. Serve 2,000 requests/s with p99 under 80 ms. Budget 100 USD/mo.

A batch shares one read of the weightsDrag or use arrow keys to change the batch size

Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s

Parts you can add: Vector DB, Key-value store, Redis cache, Search index. At most 4 parts. Budget: 100 USD/mo.

Tap a part to add it, or drag it onto a trace.Click a part to add it, or drag it onto a trace.