No prefix cache vs prefix cache
Without a prefix cache, the GPU reads the same 5,400 system prompt tokens again for every chat. A prefix cache keeps their computed state and reuses it. For 3 chats/s of 6,000 tokens, no cache serves 1.7 chats/s and p99 passes 10 s. With 90 percent cached, p99 is 1.6 s at the same cost.
No prefix cache · the GPU reads 6,000 of 6,000 prompt tokens · work per chat 100%
Prefix cache · the GPU reads 600 of 6,000 prompt tokens · work per chat 25%
Case 1
A support bot gets 3 chats/s of 6,000 input tokens. The first 5,400 tokens are the same in every chat. It must serve 2.7 chats/s with p99 below 2.5 s.
| Measure | Goal | No prefix cache, batch 8 | Prefix cache 90 percent, batch 4 |
|---|---|---|---|
| p99 latency | under 2.5 s | 10.1 s Fail | 1.6 s Pass |
| Throughput | at least 3 req/s | 2 req/s Fail | 3 req/s Pass |
| Monthly cost | 1,904 USD/mo | 1,904 USD/mo | |
| Error rate | 43.6% | 0% |