Semantic cache and entry lifetime

Agent 13
1,900 USD/mo1,900 USD/moHTTP/2Users40 req/sSemantic ca…1 vCPU · 1 GBAPI server1 vCPU · 1 GBLoad balanc…1 vCPU · 1 GBLLM server1 vCPU · 1 GBLLM server1 vCPU · 1 GB
1,900 USD/mo1,900 USD/moHTTP/2HTTP/2HTTP/2HTTP/2Users40 req/sSemantic c…API serverLoad balan…LLM serverLLM server

Tune the parts. Turn every goal green.

Traffic
40 req/s · 95% reads
Limit
API server answers about 3,300 req/s
Bill
  • GPU server (H100) × 23,800 USD/mo
  • Semantic cache4 USD/mo
  • API server4 USD/mo
  • Load balancer (L7)4 USD/mo
3,812 of 3,900 USD/mo

Goals

Not tested yet

99 of 100 requests finish faster than this.How old the data is when a reader gets it.The share of requests that fail or get no answer.What the servers and services cost each month.

A support bot gets 40 requests/s: 38 questions and 2 staff answer rewrites. Questions cover 3,000 topics in 8 phrasings each. Two H100 GPUs serve about 24 chats/s. Tune the cache: p99 under 6 s, at most 2 percent wrong answers, and no edited answer served for over 3 min.