KV memory
Difficulty 2 of 5
Each cached token has keys and values for 32 layers, 8 KV heads, and 128 values per head at 2 bytes/value. Each request has 8,192 tokens. A 24 GiB GPU reserves 12 GiB for weights and all other memory. How many full contexts fit in the rest?
The question
Answer with one number in requests.
Goals and limits
- p99 latency
- under 200 ms
- Monthly cost
- under 10 USD/mo
- Traffic
- 100 req/s
Concepts
How players did
Player results load from the server.