ETL or ELT
Difficulty 1 of 5
Raw JSON events arrive at 60 MB/s. Dashboards need only the 4 percent that survives cleaning. The warehouse loads at most 50 MB/s. The shop database has 8 vCPU. Choose where the transform runs. Keep shop p99 below 50 ms and lose no events.
Options
- Transform in a batch engine, then load
- Load raw data, transform in the warehouse
- Transform with SQL in the shop database
Goals and limits
- p99 latency (Shop)
- under 50 ms
- p99 latency (Raw)
- under 2 s
- Error rate (Raw)
- under 0.1%
- Monthly cost
- under 1,000 USD/mo
- Traffic
- 60 req/s
Concepts
How players did
Player results load from the server.