Cost, latency and capacity of LLM serving
Chat assistant
- Pick a tier and cache the prefix
- Stream the first token
- Own GPUs for quality and cost
- Long documents and KV memory
- Router, fallback and a viral spike
- Final test
A chat assistant answers users with a language model. Start with a hosted model and its token bill, stream the answer, self-host on GPUs, add long documents, then survive a viral spike with a router and a fallback.
Final testA bigger viral spike: chats rise 3.5 times for 20 s at 5:00. Your final board runs once more with a new traffic seed.
Stage 5 of 5 · 3 parts given · 33 requests/s
Progress on this page stays on this device.