Change data capture
Change data capture sends committed changes to derived stores. When the change worker stops for 5 s, it resumes from its saved position. The cache and the search index then catch up with no lost update.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Change feed not connected to the copies | Feed updates cache and search |
|---|---|---|---|
| Data mismatches (Reads) | under 2,000 | 22,922 Fail | 736 Pass |
| Data age | under 10 s | 59.9 s Fail | 5.2 s Pass |
| Monthly cost | 44 USD/mo | 44 USD/mo | |
| p99 latency | 25 ms | 25 ms | |
| Error rate | 0% | 0% |
Key numbers
| Number | Value |
|---|---|
| Changes per second, one feed | 10,000 ops/s |
| Default checkpoint interval | 30 s |
| Default refresh interval of a search index | 1 s |
2 workers · 1 TB moves in 1.1 h · 1,000 Mbps per card
How it works
A system often keeps copies of data: a cache for fast reads, a search index for text queries. Each copy must follow the database. A common mistake is a dual write: the application writes the database and then updates the cache and the index. If the app crashes between the writes, the copies disagree, and nothing repairs them.
Change data capture reads the database's own log of committed changes. Every change that the database accepted appears in the log, in order. A capture worker reads the log and sends each change to the cache and the search index. The application writes only to the database.
The worker saves its position in the log. In the mini board 1,000 accounts receive 100 updates/s for 20 s, and 500 reads/s run for one minute. The change worker stops from 0:10 to 0:15. When it restarts, it reads the log from its saved position. The cache and the search index catch up, and no update is lost. Mismatched reads stay below 2,000, and update age stays below 10 s.
Capture gives at-least-once delivery, so the consumers must handle a repeated change. Apply changes by version, and keep the saved position safe.
Watch the lag between the database and the copies. A growing lag means a stuck worker.
In the levels you connect durable changes to the cache and the search index.