Bounded queue
A bounded queue holds only the work that a service can finish within the wait you allow. A limit of 600,000 tokens/min serves about 2.3 summaries/s, so a buffer near 100 jobs keeps accepted jobs under 60 s. The queue rejects the rest at once.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Direct calls, no buffer | Queue of about 100 jobs, rejects when full |
|---|---|---|---|
| p99 latency | under 1 s | 9.7 s Fail | 72 ms Pass |
| Queue lag | under 60 s | 0 s Pass | 44.3 s Pass |
| Error rate | under 35% | 45.8% Fail | 29.3% Pass |
| Lost writes | none | 0 Pass | 0 Pass |
| Monthly cost | 8 USD/mo | 12 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Default queue buffer | 10,000 messages |
| Messages per second, one queue | 20,000 ops/s |
| Hosted model call, median | 150 ms |
Buffer of 100 jobs · the service finishes 2.3 jobs/s · longest wait 43 seconds
How it works
A queue smooths a burst, but it cannot hide a service that is slower than the arrivals. If arrivals stay above the service rate, an unbounded queue grows without limit. Every job then waits longer, and old jobs are worth less.
Size the queue from two numbers: the service rate and the longest wait you allow. In the mini board the provider allows 600,000 tokens/min. Each summary needs 4,400 tokens, so the provider finishes about 2.3 summaries/s. In 60 s it finishes about 136 jobs. A buffer near 100 jobs keeps accepted jobs under 60 s.
When the buffer is full, the queue rejects new work at once. That is backpressure. The caller learns early and can retry later, and the work that was accepted still finishes on time. Direct calls to the provider have no such buffer. In the burst they are rejected by the provider, and retries add more load.
The caller must handle a rejection. Retry with a delay and jitter, or tell the user to wait. Do not retry at once.
In the levels you put a bounded queue in front of a limited service and size it from the rate and the wait.