Data pipelines
A data pipeline moves data through steps such as extract, transform and load, often in large batches. A worker with a 1,000 Mbps network card moves at most about 125 MB/s, so 1 TB takes more than 2 hours on one worker. Split the input, and several workers share the time.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | One 8 vCPU batch worker | Three smaller workers in parallel |
|---|---|---|---|
| Completion time | under 9,000 s | 11,000 s Fail | 5,333.38 s Pass |
| Lost writes | none | 0 Pass | 0 Pass |
| Monthly cost | 79 USD/mo | 64 USD/mo | |
| Error rate | 0% | 0% | |
| Level rules | none broken | Board is not complete Fail | Pass |
Key numbers
| Number | Value |
|---|---|
| Network card of one host, default | 1,000 Mbps |
| Throughput of one batch task, default | 100 Mbps |
| Tasks one batch engine runs at once, default | 4 tasks |
| Reads per second for each object store prefix | 5,500 reads/s |
2 workers · 1 TB moves in 1.1 h · 1,000 Mbps per card
How it works
Pipelines carry data from where it is made to where it is used: from an app database to a warehouse, from logs to reports, from raw events to a model. Batch pipelines run on a schedule over a large set of data. Stream pipelines handle each event soon after it arrives.
A batch job ends when its slowest resource has moved all the data. Find that resource first. If the network card is the limit, more CPU on the same worker does nothing. Split the input into many pieces and give them to several workers, so that each worker moves only a part. The job time then falls with each worker that you add, until the source or the sink becomes the limit.
Pipelines fail part of the way through. Make each step safe to run again: write the output of each piece to a new place, record which pieces are done, and let a retry skip them. Check each batch for its size, its schema and missing values before the next step uses it.
Watch the time until the data is ready, not only the job time. A report that is due at 06:00 needs the job to start early enough, with time for one retry.
In the levels you split large jobs across workers and finish them before the deadline.