Semantic cache and entry lifetime

  • Tune
  • Agent 13

Difficulty 3 of 5

A support bot gets 40 requests/s: 38 questions and 2 staff answer rewrites. Questions cover 3,000 topics in 8 phrasings each. Two H100 GPUs serve about 24 chats/s. Tune the cache: p99 under 6 s, at most 2 percent wrong answers, and no edited answer served for over 3 min.

Play level

1,900 USD/mo1,900 USD/moHTTP/2Users40 req/sSemantic ca…1 vCPU · 1 GBAPI server1 vCPU · 1 GBLoad balanc…1 vCPU · 1 GBLLM server1 vCPU · 1 GBLLM server1 vCPU · 1 GB
1,900 USD/mo1,900 USD/moHTTP/2HTTP/2HTTP/2HTTP/2Users40 req/sSemantic c…API serverLoad balan…LLM serverLLM server
The board at the start of the level
Cache size and hit ratioDrag or use arrow keys to change the cache size

1,600 keys · hit 81% · mean 3.2 ms

Goals and limits

p99 latency
under 6 s
Data age
under 180 s
Error rate
under 1%
Quality
at least 0.98
Monthly cost
under 3,900 USD/mo
Hosts
2 × 1 vCPU · 80 GB
Traffic
40 req/s

Concepts

How players did

Player results load from the server.

All levels