Shadow traffic
A shadow model scores real traffic, but its answers never reach users. In the mini board a queue kept user p99 at 14 ms while the new model was down for 10 s. Compare its scores with the live model before you switch.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | The API calls the new model directly | A queue feeds the new model |
|---|---|---|---|
| p99 latency | under 30 ms | 45 ms Fail | 14 ms Pass |
| Error rate | under 0.1% | 5.8% Fail | 0% Pass |
| Queue lag | under 30 s | 0 s Pass | 7.7 s Pass |
| Monthly cost | 72 USD/mo | 80 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Messages per second, one queue | 20,000 ops/s |
| Queue hop, median | 0.5 ms |
| Scores per second, one CPU model server | 10,000 ops/s |
Canary · 5% of requests fail for 15 seconds · 750 failed requests
How it works
A new version can pass every test and still fail on real traffic. A shadow deployment tests it on real traffic without risk. Each live request is also sent to the new version. The new version computes its answer, and the system keeps that answer for comparison. Users only get the answer of the live version.
The copy must not hurt the live path. In the mini board a new fraud model must score a copy of every live request for a week. It is slower than the live model and not yet stable. If the API called both models in the same request, a slow or failing shadow would slow or fail the user requests.
A queue separates them. The API sends each request to the live model, and it puts a copy on a durable queue. A consumer reads the queue and calls the shadow model at its own pace. If the shadow is down for 10 s, the queue holds the copies, and user p99 stays at 14 ms with no errors. The copies stay at most 30 s behind.
Then compare: score by score, find where the new model differs, and check the cases where they disagree. Switch only when the differences are understood.
A shadow doubles the load on the features and the data it reads. Plan the capacity for it.
In the levels you send the copy through a queue and check that user latency stays safe.