Tensor parallel inside the server

  • Build
  • Fabric 06
  • Rack

Difficulty 3 of 5

Train a 30B model on 2 servers of 8 H100 (80 GB each). NVLink joins the GPUs of one server at 450 GB/s; each GPU has one 400 Gbit/s rail port to the other server. Pick the parallel layout so that the model fits and a training step takes less than 11 s.

Play level

Data centerHostHostTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPGPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)NVLink swi…NVLink swi…InfiniBand…200 nsInfiniBand…200 nsInfiniBand…200 nsInfiniBand…200 nsInfiniBand…200 nsInfiniBand…200 nsInfiniBand…200 nsInfiniBand…200 ns
The board at the start of the level
A full batch keeps each GPU busyArrow keys change the GPUs. Tap or Space changes the batch

2 GPUs · batch 64 · job 1.5 h · 2.1 USD

Goals and limits

p50 latency
under 11 s
Monthly cost
under 100,000 USD/mo

Concepts

How players did

Player results load from the server.

All levels