Batch size inside 50 ms

  • Tune
  • Model 11

Difficulty 3 of 5

A ranking network on one GPU server must answer 6,000 requests/s. The whole request, with feature reads, must finish within 50 ms at p99. You can change only the batch size of the ranker.

Play level

1,900 USD/mogRPCUsers6,000 req/sAPI server4 vCPU · 4 GBFeature sto…1 vCPU · 4 GBModel (GPU)4 vCPU · 80 GB
1,900 USD/moHTTP/2gRPCUsers6,000 req/sAPI server4 vCPU · 4 GBFeature sto…1 vCPU · 4 GBModel (GPU)4 vCPU · 80 GB
The board at the start of the level
A batch shares one read of the weightsDrag or use arrow keys to change the batch size

Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s

Goals and limits

p99 latency
under 50 ms
Error rate
under 0.1%
Monthly cost
under 2,000 USD/mo
Traffic
6,000 req/s

Concepts

How players did

Player results load from the server.

All levels