Tensor parallel inside the server

Fabric 06
Data centerHostHostTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPGPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)NVLink swi…NVLink swi…InfiniBand…200 nsInfiniBand…200 nsInfiniBand…200 nsInfiniBand…200 nsInfiniBand…200 nsInfiniBand…200 nsInfiniBand…200 nsInfiniBand…200 ns

Turn every goal green.

Goals

Not tested yet

Half of the requests finish faster than this.What the servers and services cost each month.

Train a 30B model on 2 servers of 8 H100 (80 GB each). NVLink joins the GPUs of one server at 450 GB/s; each GPU has one 400 Gbit/s rail port to the other server. Pick the parallel layout so that the model fits and a training step takes less than 11 s.

A full batch keeps each GPU busyArrow keys change the GPUs. Tap or Space changes the batch

2 GPUs · batch 64 · job 1.5 h · 2.1 USD