Distributed tracing
A trace lists the spans of one request: the time in each service and in each gap. The longest span is the first suspect. In the mini board a hop between regions added 115 ms each way, and one server at 92 percent CPU added long waits.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | No view of the time in each hop | Longest spans found and fixed |
|---|---|---|---|
| p99 latency (Orders) | under 300 ms | 624 ms Fail | 252 ms Pass |
| Error rate (Orders) | under 0.1% | 0% Pass | 0% Pass |
| Error rate (Cart views) | under 0.1% | 0% Pass | 0% Pass |
| Error rate (Tax quotes) | under 0.1% | 0% Pass | 0% Pass |
| Monthly cost | 56 USD/mo | 56 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Spans per second, one trace store | 10,000 spans/s |
| Round trip between continents, fast end | 70 ms |
| Round trip between continents, slow end | 160 ms |
One CPU graph · CPU 40% · disk no graph · the fault is hidden
How it works
A request through seven services is hard to understand from the outside. A metric says that p99 is 950 ms. It does not say where the time goes. Each service can say that it is fast, because each one measures only its own work.
A trace follows one request from the start to the end. Each service adds a span: when it started, when it ended, and which call it made. The spans form a tree. You can see the time in each service, and the time in the gaps between them, which is network and queue time.
In the mini board an order passes through seven services in a row. Order p99 is 950 ms and must be under 300 ms. The trace shows two large spans. One server runs at 92 percent CPU and builds long waits. And one call crosses between regions, which adds about 115 ms each way. Neither fault is visible in a service metric alone.
Fix the longest span first, then run again. The next longest span shows. Other teams also call the cart and tax services, so the fix must not break those calls: move the tax service to the same region, and add capacity where the CPU is high.
Traces cost storage and CPU. Sample them, and keep all traces that are slow or have errors.
In the levels you read the spans, find the longest one and fix it.