ETL or ELT

Flow 01

Choose

Raw JSON events arrive at 60 MB/s. Dashboards need only the 4 percent that survives cleaning. The warehouse loads at most 50 MB/s. The shop database has 8 vCPU. Choose where the transform runs. Keep shop p99 below 50 ms and lose no events.

Goals

Not tested yet

99 of 100 requests finish faster than this.99 of 100 requests finish faster than this.The share of requests that fail or get no answer.What the servers and services cost each month.
Choose a design
Transform in a batch engine, then load
Load raw data, transform in the warehouse
Transform with SQL in the shop database