Coordination
Coordination lets several machines agree on one fact, such as which runner owns a job. A lease gives the owner the right to act for a set time, 30 s by default here. A fencing token lets the resource reject an old owner that still acts after its lease ends.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Leases without fencing tokens | Fencing tokens on both runners |
|---|---|---|---|
| Duplicates | none | 100 Fail | 0 Pass |
| Split-brain writes | none | 100 Fail | 0 Pass |
| Lost writes | none | 0 Pass | 0 Pass |
| Completion time (Archive) | under 18 s | 11.02 s Pass | 13.92 s Pass |
| Completion time (Checks) | under 8 s | 5 s Pass | 5 s Pass |
| Monthly cost | 24 USD/mo | 24 USD/mo | |
| Error rate | 0% | 0% |
Key numbers
| Number | Value |
|---|---|
| Lease time for a job runner, default | 30 s |
| Round trip for a vote in one zone | 0.5 ms |
| Round trip for a vote between zones | 1.5 ms |
| Round trip between continents, up to | 160 ms |
Fencing tokens · lease 30 s · late write with token 33: rejected
How it works
Some work must have exactly one owner: one leader for a database, one runner for a job, one writer for a file. Machines do not share memory or a perfect clock, so they agree through messages, and messages can be late or lost.
A lease is a lock with an end time. The owner must renew it before it runs out. If the owner stops or loses contact, the lease ends and another machine can take it. A short lease gives a fast takeover. A long lease needs fewer renewals but makes the takeover slower.
A lease alone is not safe. An owner can pause, for example during a long network fault, and continue after its lease ended without knowing it. It then acts as if it still owns the job. A fencing token prevents this. Each new lease gets a larger number, and the resource rejects each request with a number smaller than the largest one it saw.
Agreement has a cost. A vote between machines needs at least one round trip, about 0.5 ms in one zone and up to 160 ms between continents, and a majority of the voters must be reachable.
In the levels you set leases and add fencing tokens so that an old owner cannot do harm.