Batch embedding on GPUs

Model 14
520 USD/moAMQPgRPCgRPCUsers0 req/sMessage bro…2 vCPU · 8 GBKey-value s…2 vCPU · 8 GBEmbedding m…1 vCPU · 24 GB
AMQPgRPCgRPCUsers0 req/sMessage br…Key-value…Embedding…

Turn every goal green.

Limit
Key-value store answers about 20,000 writes/s
Bill
  • GPU server (L4)520 USD/mo
  • Message broker32 USD/mo
  • Key-value store32 USD/mo
584 of 1,700 USD/mo

Goals

Not tested yet

Time until all the work is done.Saved data that was lost later.What the servers and services cost each month.

Every day, all 100 million catalog items get new embeddings, 256 tokens each. Today one L4 GPU server embeds them one at a time. Finish each run within 24 h and stay under 1,700 USD/mo. An L4 host costs 520 USD/mo and an H100 host 1,900 USD/mo.

A batch shares one read of the weightsDrag or use arrow keys to change the batch size

Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s

Parts you can add: Embedding model. At most 8 parts. Budget: 1,700 USD/mo.

Tap a part to add it, or drag it onto a trace.Click a part to add it, or drag it onto a trace.