KV memory

Agent 04

Estimate

Each cached token has keys and values for 32 layers, 8 KV heads, and 128 values per head at 2 bytes/value. Each request has 8,192 tokens. A 24 GiB GPU reserves 12 GiB for weights and all other memory. How many full contexts fit in the rest?

1101001,00010K100K1M

Type a number or drag the scale. Each mark is 10 times the mark before it.

A batch shares one read of the weightsDrag or use arrow keys to change the batch size