Observability
Observability is the ability to see what a system does inside from the data it sends out: metrics, logs and traces. A database disk that allows 50,000 operations per second can be full while the CPU looks idle. Without a graph for each resource, you look in the wrong place.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Fixed size and one API CPU graph | Autoscaling from metrics and a second database copy |
|---|---|---|---|
| p99 latency | under 100 ms | 10.1 s Fail | 26 ms Pass |
| Error rate | under 0.1% | 16.1% Fail | 0% Pass |
| Monthly cost | 25 USD/mo | 25 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Disk operations per second, one database | 50,000 IOPS |
| Autoscaling target use, default | 0.7 fraction |
| Series one metrics store keeps, default | 100,000 series |
| Price to ingest one GB of logs or metrics | 0.12 USD/GB |
One CPU graph · CPU 40% · disk no graph · the fault is hidden
How it works
When a system is slow, the first question is where the time goes. Metrics tell how much and how often: requests per second, errors, latency and the use of each resource. Logs record single events with detail. Traces follow one request across all the services that it touches.
For each resource, watch three signals. Utilization shows how busy it is. Saturation shows how much work waits for it. Errors show how often it fails. A CPU at 40 percent tells nothing about a disk queue or a full connection pool. Watch the CPU, memory, disk, network and pools of each part, not only one of them.
Metrics also start actions. An autoscaler adds servers when a metric goes above a target, for example 70 percent CPU. If it reads the wrong metric, it never starts, and the system stays too small while users wait.
Observability has costs. Each value of a metric label makes a new series, and too many series overload the metrics store. Each log line costs ingest and storage. Sample traces, and keep detailed data only as long as you need it.
In the levels you add the metrics that show the real limit and let them start the scaling.