Tamper-evident audit log
A tamper-evident log chains each entry to the hash of the entry before. Each chain needs one ordered writer. Keyed by actor, 4 partitions carry 15 MB/s with 0 events out of order.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Events spread over partitions with no key | Events keyed by actor |
|---|---|---|---|
| Out of order | none | 2,643 Fail | 0 Pass |
| Error rate | under 0.1% | 0% Pass | 0% Pass |
| Lost writes | none | 0 Pass | 0 Pass |
| Queue lag | under 2 s | 15.93 s Fail | 0 s Pass |
| p99 latency | under 50 ms | 35 ms Pass | 35 ms Pass |
| Monthly cost | 32 USD/mo | 32 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Events per second, one log | 100,000 ops/s |
| Log append, median | 2 ms |
| Log append, p99 | 10 ms |
One CPU graph · CPU 40% · disk no graph · the fault is hidden
How it works
An audit log answers who did what, and when. Its value depends on trust: if an attacker can change the log, the log proves nothing. A hash chain makes changes visible. Each entry contains the hash of the entry before it. If someone edits or removes an entry, every later hash stops matching.
A chain has one rule: its entries must be added in order. Entry 5 needs the hash of entry 4. So each chain needs one writer that sees the events in order.
In the mini board services send 5,000 audit events/s of 6 KB each, from 2,000 actors. Each actor has its own chain. A single log would be one chain for all events, and one partition cannot carry the rate: a log partition carries 10 MB/s. More partitions give the capacity, but events that go to any partition can arrive out of order.
The key fixes both. Key each event by actor. The log sends all events of one actor to the same partition, in order, and a chain writer reads each partition. Different actors run in parallel. Four partitions carry about 15 MB/s, and no event is out of order. p99 stays under 50 ms and lag under 2 s.
Store the head of each chain somewhere separate, so that deleting the last entries is also visible.
In the levels you key the log by actor and check the count of events out of order.