Decode reads every weight
Difficulty 2 of 5
An 8B model with 16-bit weights is 16 GB. One user asks a question. At batch size 1, each new token reads all weights from HBM once. The H100 reads 3.35 TB/s. Compute takes about 16 µs per token. How many ms does a 100-token answer take?
The question
Answer with one number in ms.
Goals and limits
- Completion time
- under 1 s
- Monthly cost
- under 10,000 USD/mo