Micro-batches fill the pipeline

Fabric 05
HostTCPTCPTCPTCPTCPTCPTCPTCPGPU (H100)GPU (H100)GPU (H100)GPU (H100)NVLink swit…

Tune the parts. Turn every goal green.

Goals

Not tested yet

Time until all the work is done.What the servers and services cost each month.

A 13B model runs as 4 pipeline stages on 4 H100 (80 GB each). A stage needs 100 ms for one micro-batch of 2,048 tokens. The run trains 8.2M tokens. Set the micro-batches per step so the run ends in under 505 s and each GPU still fits in memory.

A full batch keeps each GPU busyArrow keys change the GPUs. Tap or Space changes the batch

2 GPUs · batch 64 · job 1.5 h · 2.1 USD