Batch embedding on GPUs

  • Build
  • Model 14

Difficulty 4 of 5

Every day, all 100 million catalog items get new embeddings, 256 tokens each. Today one L4 GPU server embeds them one at a time. Finish each run within 24 h and stay under 1,700 USD/mo. An L4 host costs 520 USD/mo and an H100 host 1,900 USD/mo.

Play level

520 USD/moAMQPgRPCgRPCUsers0 req/sMessage bro…2 vCPU · 8 GBKey-value s…2 vCPU · 8 GBEmbedding m…1 vCPU · 24 GB
AMQPgRPCgRPCUsers0 req/sMessage br…Key-value…Embedding…
The board at the start of the level
A batch shares one read of the weightsDrag or use arrow keys to change the batch size

Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s

Goals and limits

Completion time
under 86,400 s
Lost writes
none
Monthly cost
under 1,700 USD/mo
Parts you can add
Embedding model
Part limit
8

Concepts

How players did

Player results load from the server.

All levels