Cache the shared prompt prefix

  • Tune
  • Agent 06

Difficulty 3 of 5

A support bot gets 3 chats/s. Each prompt has 6,000 input tokens. The first 5,400 are the same system prompt for every chat. Output is 120 tokens. Set the prefix cache share and the batch size. Keep p99 below 2.5 s and serve 2.7 chats/s.

Play level

GPU server (H100)1,900 USD/moHTTP/2HTTP/2Users3 req/sAPI server1 vCPU · 1 GBLLM server1 vCPU · 1 GB
1,900 USD/moHTTP/2HTTP/2Users3 req/sAPI server1 vCPU · 1 GBLLM server1 vCPU · 1 GB
The board at the start of the level
Cache size and hit ratioDrag or use arrow keys to change the cache size

1,600 keys · hit 81% · mean 3.2 ms

Goals and limits

p99 latency
under 2.5 s
Throughput
at least 3 req/s
Monthly cost
under 2,000 USD/mo
Hosts
1 vCPU · 80 GB
Traffic
3 req/s

Concepts

How players did

Player results load from the server.

All levels