Cache the shared prompt prefix

Agent 06
GPU server (H100)1,900 USD/moHTTP/2HTTP/2Users3 req/sAPI server1 vCPU · 1 GBLLM server1 vCPU · 1 GB
1,900 USD/moHTTP/2HTTP/2Users3 req/sAPI server1 vCPU · 1 GBLLM server1 vCPU · 1 GB

Tune the parts. Turn every goal green.

Traffic
3 req/s · reads only
Limit
API server answers about 3,300 req/s
Bill
  • GPU server (H100)1,900 USD/mo
  • API server4 USD/mo
1,904 of 2,000 USD/mo

Goals

Not tested yet

99 of 100 requests finish faster than this.Requests served each second.What the servers and services cost each month.

A support bot gets 3 chats/s. Each prompt has 6,000 input tokens. The first 5,400 are the same system prompt for every chat. Output is 120 tokens. Set the prefix cache share and the batch size. Keep p99 below 2.5 s and serve 2.7 chats/s.