A week of training

Fabric 10
Data centerHostHostHostTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPGPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)NVLink sw…NVLink sw…NVLink sw…InfiniBan…200 nsInfiniBan…200 nsInfiniBan…200 nsInfiniBan…200 nsInfiniBan…200 nsInfiniBan…200 nsInfiniBan…200 nsInfiniBan…200 ns

Keep every goal green in each wave.

Goals

Not tested yet

How much recent data a failure can lose, in time.The share of requests that get a correct answer.What the servers and services cost each month.

16 H100 train a 13B model for a week; a third server waits idle. A GPU, a rail switch and a whole server will fail. Set the checkpoint plan and the spare pool before the run: the job must be productive 96.5 percent of the week and lose at most 40 min of work in a failure.

A backup and its restoreArrow keys change the backup interval. Tap to change the restore rate.

Backup every 60 minutes · RPO 60 minutes · RTO 11 hours at 100 Mbps