Send each request type to its own model
Difficulty 3 of 5
Two entry points share one API. FAQ chats arrive at 40/s with short prompts. Analyses arrive at 0.4/s with 3,000-token prompts and need the 70B model. You have 2 small-model GPUs (2k-token context) and 2 large-model GPUs. FAQ p99 under 2.5 s, analysis p99 under 40 s.
Batch 32 · 2,500 tokens/s · a 200 token answer takes 2.6 s
Goals and limits
- p99 latency (Faq)
- under 2.5 s
- p99 latency (Analysis)
- under 40 s
- Error rate (*)
- under 1%
- Monthly cost
- under 4,900 USD/mo
- Parts you can add
- API server, Load balancer (L7)
- Part limit
- 3
- Hosts
- 2 × 1 vCPU · 24 GB, 2 × 1 vCPU · 80 GB
- Traffic
- 40 req/s