Model provider failover

Agent 08
HTTP/2Users10 req/sAPI server1 vCPU · 1 GBLoad balanc…1 vCPU · 1 GBLLM API1 vCPU · 1 GB
HTTP/2HTTP/2HTTP/2Users10 req/sAPI server1 vCPU · 1 GBLoad balanc…1 vCPU · 1 GBLLM API1 vCPU · 1 GB

Fix the board. Turn every goal green.

Traffic
10 req/s · reads only
Limit
API server answers about 3,300 req/s
Bill
  • API server4 USD/mo
  • Load balancer (L7)4 USD/mo
  • LLM API4 USD/mo
12 of 40 USD/mo

Goals

Not tested yet

The share of requests that get a correct answer.99 of 100 requests finish faster than this.What the servers and services cost each month.

Chats call one hosted model provider through a router. The provider goes down at 0:30 for 60 s, and every chat fails until it returns. Keep availability above 98 percent and p99 below 5 s. You may make 3 edits.

0:30. Chat errors jump from 0 to 100 percent. The model provider sends no answers.

Parts you can add: LLM API. At most 2 parts. Budget: 40 USD/mo.

Tap a part to add it, or drag it onto a trace.Click a part to add it, or drag it onto a trace.