19. Scalability

This section covers how to scale machine learning and scientific workloads on the cluster, from the fundamentals of parallel computing to running across many GPUs. It introduces parallel computing concepts, how GPUs accelerate computation and how to profile them, the distributed training strategies (data, model, tensor, pipeline, and fully sharded parallelism), and parallel I/O for keeping the GPUs fed with data.