Agent orchestration
More agents mean more model calls and more model servers. A planner with cheap executors made 1 large call and 6 small calls for each task and fit on 2 GPUs. A lead with 3 specialists needed 4 GPUs.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | One agent, all calls to the large model | Planner with cheap executors |
|---|---|---|---|
| p99 latency | under 30 s | 60.4 s Fail | 17.9 s Pass |
| Error rate | under 1% | 16.3% Fail | 0% Pass |
| Monthly cost | 5,712 USD/mo | 5,712 USD/mo |
Key numbers
| Number | Value |
|---|---|
| One H100 GPU host | 1,900 USD/mo |
| Default token rate, one model server | 2,500 tokens/s |
| Default batch size, one model server | 32 requests |
Fencing tokens · lease 30 s · late write with token 33: rejected
How it works
One agent is simple. It reads a task, calls tools and answers. When it is not good enough, a common move is to add agents: a planner that splits the task, executors that do the steps, specialists for hard parts. This is coordination, and it has a cost that is easy to forget.
Each agent is a stream of model calls. A planner calls a large model once to plan. Executors call a small model for each step. A lead with specialists calls the large model many times. Every call needs GPU time, and GPUs are the costly part of the system.
In the mini board a ticket agent must resolve 85 percent of tasks at 1 task/s on 3 GPUs. In tests one agent resolves 71 percent, which is too few. A planner with cheap executors resolves 88 percent. A lead with 3 specialists resolves 90 percent. The planner makes 1 large call and 6 small calls for each task, and it fits on 2 GPUs. The team needs 4 GPUs, so it does not fit. A single agent that sends every call to the large model is also too slow on 3 GPUs: its p99 reaches 60 s, and 16 percent of tasks fail.
Meet the target with the least coordination. Add roles only where the quality gain is worth the GPUs. Each handoff also adds latency.
In the levels you pick the design that meets the quality target and fits the GPUs.