Index fits in RAM
Index memory follows what you store. Here 120 million messages need about 200 GB, while 12 million facts need 20 GB and recall 90 percent. The index host has 32 GB, so store distilled facts, not transcripts.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | 120 million messages in the index | 12 million distilled facts |
|---|---|---|---|
| p99 latency | under 50 ms | 10.1 s Fail | 26 ms Pass |
| Error rate | under 0.1% | 53.2% Fail | 0% Pass |
| Monthly cost | 164 USD/mo | 164 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Memory of an 8 vCPU server | 32 GB |
| Cache memory price | 5 USD/GB-mo |
| Vector search, median | 0.3 ms |
NVMe drive · 75 µs per read
How it works
Memory is fast and small. Disk is slower and larger. A lookup in memory takes about 90 nanoseconds. A read from a flash disk takes about 75 microseconds, and over the network it takes more. A search index is fast only while the parts that queries touch stay in memory. When it grows past RAM, the lookups fall to the slow tier.
So the first step of design is to count bytes. In the mini board a support agent recalls past chats. Over 90 days there are 120 million messages, 8 million sessions and 12 million customer facts. The index host has 32 GB. At the size of each vector, 120 million messages need about 200 GB, which is six times the RAM. The index cannot stay in memory, and lookups miss the p99 limit of 50 ms.
You can store less, and store it better. 12 million distilled facts need about 20 GB, which fits. In tests, recall is 93 percent for messages, 78 percent for summaries and 90 percent for facts, and the target is 85 percent. Facts meet the target and fit the host. Transcripts reach higher recall but do not fit.
The price also follows the tier. At 5 USD per GB-month for memory, 200 GB of index costs about 1,000 USD/mo, and 20 GB costs about 100 USD/mo.
In the levels you pick what to store so that the hot data fits in memory.