Prompt prefix cache
Prompt caching reuses the stored state of a shared prompt prefix. Here 5,400 shared tokens of 6,000 cut the GPU work for each chat by about 75 percent. Keep stable text first, because a change early in the prompt breaks the shared prefix.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Whole prompt processed for each chat | Shared prefix reused |
|---|---|---|---|
| p99 latency | under 2.5 s | 10.1 s Fail | 1.6 s Pass |
| Throughput | at least 3 req/s | 2 req/s Fail | 3 req/s Pass |
| Monthly cost | 1,904 USD/mo | 1,904 USD/mo | |
| Error rate | 43.6% | 0% |
Key numbers
| Number | Value |
|---|---|
| Default share of the prompt that is reused | 0 ratio |
| Default token rate, one model server | 2,500 tokens/s |
| Default time to the first token | 150 ms |
Prefix cache · the GPU reads 600 of 6,000 prompt tokens · work per chat 25%
How it works
A model reads the whole prompt before it writes a word. This first step, the prefill, costs GPU time in proportion to the number of input tokens. In many apps most of the prompt is the same for every user: the system prompt, the rules and a set of examples.
A prefix cache keeps the model's internal state for a prompt prefix. When a new request starts with the same prefix, the server loads the stored state, and it processes only the new tokens. The result is the same as before, with less work.
In the mini board a support bot gets 3 chats/s. Each prompt has 6,000 input tokens. The first 5,400 are the same system prompt for every chat, which is 90 percent of the input. The output is 120 tokens. Without the cache, the GPU processes all 6,000 tokens for each chat, and no batch size meets both targets: p99 below 2.5 s and 2.7 chats/s. With the cache, each chat needs about one quarter of the work, and a small batch keeps p99 near 1.6 s.
The cache only works for an exact prefix. Put the stable text first, and put the changing text, such as the user question, last. Even one changed word at the start breaks the match. The stored state also uses GPU memory, so it competes with the batch.
In the levels you set the share of the prompt that is cached and the batch size.