Bounded queue under a token limit

Agent 09
HTTP/2Users1 req/sAPI server1 vCPU · 1 GBLLM API1 vCPU · 1 GB
HTTP/2HTTP/2Users1 req/sAPI server1 vCPU · 1 GBLLM API1 vCPU · 1 GB

Turn every goal green.

Traffic
1 req/s · up to 8 req/s · writes only
Limit
API server answers about 3,300 req/s
Bill
  • API server4 USD/mo
  • LLM API4 USD/mo
8 of 40 USD/mo

Goals

Not tested yet

99 of 100 requests finish faster than this.How long a message waits before a worker takes it.The share of requests that fail or get no answer.Saved data that was lost later.What the servers and services cost each month.

Each upload needs a summary: 4,000 input and 400 output tokens. The provider allows 600,000 tokens/min and rejects the rest. From 0:20 to 1:00, uploads jump from 1/s to 8/s. Keep upload p99 below 1 s, jobs under 60 s old, rejections at most 35 percent, and lose no job.

Parts you can add: Managed queue, Message broker, Worker. At most 2 parts. Budget: 40 USD/mo.

Tap a part to add it, or drag it onto a trace.Click a part to add it, or drag it onto a trace.