Batch size inside 50 ms

Model 11
1,900 USD/mogRPCUsers6,000 req/sAPI server4 vCPU · 4 GBFeature sto…1 vCPU · 4 GBModel (GPU)4 vCPU · 80 GB
1,900 USD/moHTTP/2gRPCUsers6,000 req/sAPI server4 vCPU · 4 GBFeature sto…1 vCPU · 4 GBModel (GPU)4 vCPU · 80 GB

Tune the parts. Turn every goal green.

Traffic
6,000 req/s · reads only
Limit
API server answers about 13,000 req/s
Bill
  • GPU server (H100)1,900 USD/mo
  • API server16 USD/mo
  • Feature store16 USD/mo
1,932 of 2,000 USD/mo

Goals

Not tested yet

99 of 100 requests finish faster than this.The share of requests that fail or get no answer.What the servers and services cost each month.

A ranking network on one GPU server must answer 6,000 requests/s. The whole request, with feature reads, must finish within 50 ms at p99. You can change only the batch size of the ranker.

A batch shares one read of the weightsDrag or use arrow keys to change the batch size

Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s