Decode reads every weight
Fabric 02Estimate
An 8B model with 16-bit weights is 16 GB. One user asks a question. At batch size 1, each new token reads all weights from HBM once. The H100 reads 3.35 TB/s. Compute takes about 16 µs per token. How many ms does a 100-token answer take?
1 ms10 ms100 ms1 s10 s100 s1,000 s
Type a number or drag the scale. Each mark is 10 times the mark before it.