Watermark delay
A watermark sets how long a window waits for late events. Here a 25 s wait completes 98 percent of windows, but a 10 s wait completes only 93 percent. Each extra second of wait also delays each result by 1 s.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Windows wait 10 s | Windows wait 25 s |
|---|---|---|---|
| Error rate | under 3% | 7% Fail | 2% Pass |
| p99 latency | under 30 s | 10.1 s Pass | 25.1 s Pass |
| Monthly cost | 20 USD/mo | 20 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Events per second, one stream processor | 10,000 events/s |
| Log append, median | 2 ms |
| Default wait on a link | 10,000 ms |
2 workers · 1 TB moves in 1.1 h · 1,000 Mbps per card
How it works
Events do not arrive in the order in which they happen. A phone loses signal, and its events reach the log late. A stream job that counts events in one-minute windows must decide when a window is done. If it publishes at the end of the minute, it misses the late events. If it waits for ever, it never publishes.
A watermark is the rule in between. It says that the job has seen all events up to a time T once it has waited a set delay. A window closes when the watermark passes its end. Events that arrive after that are late.
The delay has two costs. A short delay publishes fast, but some windows are incomplete. A long delay completes more windows, but every result comes later. In the mini board 5 percent of windows get events 20 s late, and 2 percent get events 5 minutes late. The goal is 97 percent of windows complete, and 99 percent published within 30 s. A wait of 10 s completes only 93 percent. A wait of 25 s completes 98 percent and still publishes in time.
You cannot wait for the 5-minute stragglers without breaking the time limit. Handle them another way: update the result later, or send a correction.
Pick the delay from the lateness that you measured, not from a guess.
In the levels you set the wait and check the share of complete windows and the delay.