Tool timeouts
Each tool needs a guard that fits how it fails. Here 2 retries cut CRM errors from 4 percent to 0.01 percent, while a cache and a 1.2 s timeout held search p99 near 1.2 s through a 40 s hang.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Every tool called directly | Timeout, retry, cache or queue for each tool |
|---|---|---|---|
| p99 latency (Search) | under 2 s | 10.1 s Fail | 1.2 s Pass |
| Error rate (Search) | under 5% | 33% Fail | 2.4% Pass |
| Error rate (Crm) | under 0.5% | 3.8% Fail | 0.01% Pass |
| p99 latency (Crm) | under 600 ms | 190 ms Pass | 256 ms Pass |
| p99 latency (Calc) | under 100 ms | 67 ms Pass | 67 ms Pass |
| p99 latency (Email) | under 300 ms | 2.4 s Fail | 52 ms Pass |
| Lost writes | none | 0 Pass | 0 Pass |
| Queue lag (Email) | under 60 s | 0 s Pass | 36.8 s Pass |
| Monthly cost | 16 USD/mo | 32 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Default call timeout | 10,000 ms |
| External API call, median | 150 ms |
| External API call, p99 | 300 ms |
Timeout 10.0 s · 64 of 64 workers wait in a fault · 0% of normal calls stop
How it works
An agent calls several tools, and each tool fails in its own way. A default timeout of 10 s is the same for all of them, and it is right for none.
In the mini board there are four kinds of call. Search hangs from 0:20 for 40 s. A call that waits 10 s holds a worker for 10 s, and many such calls fill the pool. A timeout of 1.2 s cuts the wait, and a cache of recent answers gives a result when the call fails. Search p99 stays near 1.2 s.
The CRM fails on 4 percent of calls, but each failure is brief. A timeout is not the answer. Two retries with backoff are: the second try nearly always works, and errors fall to 0.01 percent.
Email takes 1.2 s and is down for 20 s. The agent must not wait for it, and no email may be lost. Put the email on a durable queue, and let a worker send it. The agent call then returns at once.
The rule is: pick the guard from the failure mode. A hang needs a timeout and a fallback. A brief error needs a retry. A long outage needs a queue. A guard that does not match the failure only adds work.
In the levels you give each tool the guard that fits it and check the limits for each tool.