Reconciliation
Reconciliation compares two records of the same money, such as your ledger and a provider file, and lists every difference. At 4 ms for each record, 2,000,000 records need 2.2 hours on one worker. Matching runs in parallel to finish in one hour.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | One consumer matches all records | 4 consumers and a dead-letter queue |
|---|---|---|---|
| Completion time | under 3,600 s | 4,000 s Fail | 2,947.12 s Pass |
| Lost writes | none | 10,000 Fail | 0 Pass |
| Monthly cost | 36 USD/mo | 28 USD/mo | |
| Error rate | 0% | 0% | |
| Level rules | none broken | Board is not complete Fail | Pass |
Key numbers
| Number | Value |
|---|---|
| Matches per second, one reconciler part | 10,000 ops/s |
| Reconciler call, p99 | 1 ms |
| Default delivery tries | 5 tries |
W 3 + R 3 of 5 · every read sees the last write
How it works
Two systems that handle the same money will disagree sometimes. A payment can be late, repeated or missing in one of them. Reconciliation is the check that finds these differences. It compares each record of one side with its partner on the other side.
The result is not a clean pass. Some records will not match, about 0.5 percent in the mini board. These unmatched items are the output. Keep every one for review. Do not drop them and do not guess. A person or a rule decides what to do with each one.
The job is big. At 02:00 the provider sends a file of 2,000,000 records, and the books close at 03:00. One match takes 4 ms, so one worker needs 2.2 hours. In the mini board a single consumer needs 4,000 s, which is after the books close at 3,600 s, and it drops the records that do not match.
Matching is easy to split, because each record is independent. Put the records on a queue, and let several consumers match them at the same time. With 4 consumers the batch fits in the hour. A record that fails to match after a few tries goes to a dead-letter queue for review, and none of the 10,000 unmatched items is lost.
Record totals as well as items. A match count that does not add up to the file is a warning.
In the levels you split the matching and keep every unmatched record.