RED and USE methods
RED asks of each service how many requests, how many fail and how long. USE asks of each resource how busy, how much waits and what fails. RED finds the service that hurts, and USE finds the full resource: here disk throughput at 113 percent.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Add API servers | More disk throughput, fewer API servers |
|---|---|---|---|
| p99 latency | under 100 ms | 5.7 s Fail | 25 ms Pass |
| Error rate | under 0.1% | 10.3% Fail | 0% Pass |
| Monthly cost | 56 USD/mo | 40 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Disk throughput, document store | 500 MB/s |
| Requests per second, one API server | 10,000 ops/s |
| Points per second, one metrics store | 10,000 ops/s |
One CPU graph · CPU 40% · disk no graph · the fault is hidden
How it works
When users complain, you need to know where to look. Two short checklists help.
RED is for services. For each service, look at the rate of requests, the rate of errors and the duration of requests. It tells you which service hurts users.
USE is for resources, such as CPU, disk and network. For each resource, look at the utilization, the saturation and the errors. Utilization is how busy it is. Saturation is how much work waits. A resource with high utilization and a queue is the bottleneck.
In the mini board photo pages read 64 KB documents at 8,800 reads/s. Users see p99 5.7 s and 10 percent errors. The API CPU is at 33 percent, so the API is not the limit. In the document store, CPU is at 15 percent and disk operations are 18 percent busy. But disk throughput is at 113 percent of its 500 MB/s limit. That is the full resource.
The tempting fix is more API servers, because the API is where users see the pain. It does nothing: the disk is full. The right fix is more disk throughput, here 1,000 MB/s, and the API can even shrink from 6 to 4 servers.
Use RED to find the service and USE to find the resource inside it.
In the levels you read both views and pick the fix that matches the full resource.