Weights plus KV cache
Fabric 01Estimate
A 70B model with 16-bit weights is 140 GB. The KV cache holds 64 users x 8,192 tokens at 320 KiB per token. 4 H100 GPUs with 80 GB each split weights and cache evenly. One decode step reads every byte once at 3.35 TB/s per GPU. How many ms is one step?
100 µs1 ms10 ms100 ms1 s10 s100 s
Type a number or drag the scale. Each mark is 10 times the mark before it.