Semantic cache
A semantic cache answers a question from the stored answer to a similar one. Merging 8 phrasings of a topic into 1 entry gave a 66 percent hit ratio with a 150 s lifetime. Each entry is a model output, so its lifetime also caps how long an old answer can live.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Entries live 3,600 s | Entries live 150 s |
|---|---|---|---|
| p99 latency | under 6 s | 4.9 s Pass | 5.2 s Pass |
| Data age | under 180 s | 152.1 s Pass | 152.1 s Pass |
| Error rate | under 1% | 0% Pass | 0% Pass |
| Quality | at least 0.98 | 0.93 Fail | 0.99 Pass |
| Monthly cost | 3,812 USD/mo | 3,812 USD/mo | |
| Error rate | 0% | 0% |
Key numbers
| Number | Value |
|---|---|
| Default similarity needed for a hit | 0.95 ratio |
| Cache read, median | 0.1 ms |
| One H100 GPU host | 1,900 USD/mo |
Similar meaning · hit ratio 66% · 14 of 40 chats/s reach the GPUs, which serve about 24
How it works
People ask the same thing in many ways. A support bot gets 3,000 topics, and each topic comes in about 8 phrasings. An exact cache treats each phrasing as its own key. The hit ratio stays near 42 percent, and the GPUs, which serve about 24 chats/s, saturate at 40 requests/s.
A semantic cache compares the meaning of a question with the stored questions. If a stored question is similar enough, above a threshold, it returns that stored answer. The 8 phrasings of a topic merge into 1 entry. Far fewer requests reach the model.
The cached value is a model answer, and an answer can become wrong. In the mini board 2 requests/s are staff rewrites of answers. A cached answer that staff have edited must not be served for more than 3 minutes. So the lifetime of an entry is a trade. At 60 s the hit ratio is only 43 percent. At 300 s a stale answer lives 305 s, which is too long. At 150 s the hit ratio is 66 percent, and the oldest stale answer is 152 s.
The threshold is a second trade. A low threshold gives more hits, but it can return an answer to a question that is not the same. Test the threshold on real pairs of questions.
In the levels you tune the lifetime and check the hit ratio and the age of the oldest answer.