Weights plus KV cache
Difficulty 1 of 5
A 70B model with 16-bit weights is 140 GB. The KV cache holds 64 users x 8,192 tokens at 320 KiB per token. 4 H100 GPUs with 80 GB each split weights and cache evenly. One decode step reads every byte once at 3.35 TB/s per GPU. How many ms is one step?
The question
Answer with one number in ms.
Goals and limits
- Completion time
- under 50 ms
- Monthly cost
- under 10,000 USD/mo
Concepts
How players did
Player results load from the server.