Model routing
Model routing sends each request type to the smallest model that can do the job. 40 FAQ chats/s fit on 2 small GPUs, but on the large model they would need about 15 GPUs. Only 0.4 analyses/s need the large pool.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | One front door for all four servers | Each request type has its own pool |
|---|---|---|---|
| p99 latency (Faq) | under 2.5 s | 10.1 s Fail | 1.6 s Pass |
| p99 latency (Analysis) | under 40 s | 60.4 s Fail | 26.1 s Pass |
| Error rate (*) | under 1% | 45.9% Fail | 0% Pass |
| Monthly cost | 4,844 USD/mo | 4,848 USD/mo | |
| Error rate | 45.9% | 0% |
Key numbers
| Number | Value |
|---|---|
| One L4 GPU host | 520 USD/mo |
| One H100 GPU host | 1,900 USD/mo |
| Small model, input tokens | 0.1 USD/M tokens |
| Large model, input tokens | 3 USD/M tokens |
1,500 req/s · Node API · 3 hosts · 18 USD/mo
How it works
A large model gives better answers, and it costs far more for each token. A large GPU host costs 1,900 USD/mo, and a small one costs 520 USD/mo. Many requests do not need the large model. A short FAQ question can get a good answer from a small one.
In the mini board two kinds of request share one API. FAQ chats arrive at 40/s with short prompts. Analyses arrive at 0.4/s with long prompts, and only the large model can do them. On the large model each FAQ chat takes about 0.4 GPU-seconds, so 40 chats/s would need about 15 GPUs.
One shared front door sends every request type to all four servers. Then FAQ chats fill the large servers, and the analyses wait behind them. The large pool saturates.
A router looks at the request type and picks the pool. FAQ chats go to the 2 small GPUs. Analyses go to the large pool, which then runs at about 56 percent. Both limits are met with the same four GPUs.
Routing needs a rule that is right. If the router sends a hard question to a small model, quality falls. Test the rule on real questions, and keep a way to move traffic to the large pool.
In the levels you route each request type to the right pool.