Model quality
Model quality is how often a model gives the right answer on real traffic, not only on test data. A feature that is 30 s old at serving time can differ from the value that the model saw in training. When training and serving compute features on different paths, quality falls without an error.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Separate writes for training and serving | One change feed for training and serving |
|---|---|---|---|
| Data mismatches (Lookups) | under 3,000 | 27,992 Fail | 740 Pass |
| Data age | under 10 s | 59.9 s Fail | 5.2 s Pass |
| Monthly cost | 47 USD/mo | 47 USD/mo | |
| p99 latency | 13 ms | 13 ms | |
| Error rate | 8.3% | 8.3% |
Key numbers
| Number | Value |
|---|---|
| Age of a served feature, default | 30 s |
| Time between feature updates, default | 10 s |
| Read one feature from the feature store | 2.5 ms |
| Time between saved change feed positions | 30 s |
Two write paths · 4 of 16 served features differ from training
How it works
A model learns from features: values such as the number of orders of a user in the last hour. At serving time it needs the same features, computed in the same way. When the two paths differ, the model gets inputs that it never learned. This is training-serving skew. The service stays up and latency stays low, but the answers get worse.
Skew has many causes. Two teams write the same feature in two languages. The online copy misses updates during an outage. Training uses values from after the event, which serving cannot know. The best fix is one definition and one pipeline for each feature, and both training and serving read its output.
Data also changes over time. User behavior, prices and fraud patterns move, so a model that was good last month can be weaker now. Measure quality on live data, compare live inputs with the training data, and train again on a schedule or when the difference grows.
Release models with care. Test a new model on a fixed test set, then on a small share of live traffic, and keep the old model ready.
In the levels you give training and serving one feature path, so that the values match.