Why Distribution Is Necessary
Distributed Training
Purpose
Why does the linear logic of “more hardware = faster training” collapse at the bisection bandwidth wall?
Distributed training appears simple when framed as splitting the work across machines and combining the results. As the machine learning fleet grows, however, communication costs rise while computation per machine shrinks, until synchronization overhead dominates and adding hardware can degrade performance. When a job trains on a single accelerator, the design optimizes for arithmetic intensity; when it trains on 10,000 accelerators, the design optimizes for communication intensity. This scaling ceiling follows from the reliability gap and communication-computation ratio. Coordinating independent machines requires moving terabytes of state across networks that are orders of magnitude slower than on-chip memory. Distributed training manages this tension by partitioning work to reduce the coordination tax, overlapping communication with computation to hide latency, and choosing synchronization strategies that balance consistency against throughput. Without this understanding, organizations can waste millions on hardware that sits idle waiting for gradients to arrive or produce models that fail to converge because stale updates disrupt optimization. Distributed training puts the C³ taxonomy in motion, with each design choice trading parallelized compute against network-bound communication and coordination across workers.
Learning Objectives
- Apply the fleet law to diagnose compute-, memory-, communication-, or coordination-bound distributed training regimes
- Calculate scaling efficiency, critical batch limits, and synchronization overhead for data-parallel training jobs
- Design memory-sharding plans that trade accelerator capacity for additional communication
- Map tensor, pipeline, expert, and data parallelism onto hardware bandwidth tiers and model structure
- Construct microbatch pipeline schedules that minimize bubbles while preserving throughput and convergence stability
- Select synchronization and low-precision policies using staleness, straggler tolerance, numerical stability, and cluster heterogeneity
- Synthesize hybrid parallelism configurations for frontier, recommendation, and alignment training constraints
The accelerator hierarchy, network fabric, and storage pipeline form the physical foundation of the fleet. The remaining challenge is algorithmic: partitioning a single training job across thousands of resources without losing the semantics of one coherent optimization process. The universal scaling law (principle 9) explains why this pressure keeps increasing: frontier quality improvements demand disproportionately more compute, data, and parameters, so the training problem eventually outgrows any single machine.
A single accelerator with 100 terabytes of memory and an exaflop of compute would make distributed training unnecessary. Real systems instead impose finite high-bandwidth memory (HBM) capacity, finite interconnect bandwidth, and finite failure budgets, so training must be partitioned across many independent chips. In the fleet stack framework shown in The Fleet Stack, distributed training represents the distribution layer: the logic that partitions the mathematical workload across the physical fleet. The strategies defined here, including data, tensor, pipeline, and hybrid parallelism, create the traffic patterns that the interconnect must carry.
The physics of the cluster
Before optimizing algorithms, the physical constraints of the Machine Learning Fleet must be understood. The performance of any distributed training job is governed by the fleet law (principle 10), introduced in The fleet law, which decomposes the per-step time:
\[ T_{\text{step}}(N) = \frac{T_{\text{compute}}}{N} + T_{\text{comm}}(N) + T_{\text{sync}}(N) - T_{\text{overlap}} \]
The critical term is the Communication-Computation Ratio, \(\rho = T_{\text{comm}}(N)/(T_{\text{compute}}/N)\). This ratio determines whether a cluster behaves as a supercomputer or a collection of idling heaters.
Past a communication-to-compute threshold, scaling stops being ideal.
Two regimes follow from this ratio. A compute-bound cluster, where \(T_{\text{compute}}/N \gg T_{\text{comm}}(N)\), spends most of its time multiplying matrices; this is the ideal state, typical for large batch sizes on dense models such as ResNet. A communication-bound cluster, where \(T_{\text{comm}}(N) \approx T_{\text{compute}}/N\), spends significant time waiting for gradients or activations to arrive, the common state for large language models (LLMs) and Deep Learning Recommendation Model (DLRM)-style recommendation models, where parameter synchronization saturates the network. The bottleneck diagnostic table works this ratio into a diagnostic framework that classifies a workload as compute-bound, memory-bound, or communication-bound from its measured bandwidth and arithmetic intensity, turning these qualitative regimes into a repeatable test.
Multi-machine training requirements
Three concrete signals indicate when distributed training becomes necessary rather than merely beneficial. The first signal is Memory Exhaustion: model parameters, optimizer states, and activation storage exceed single-device capacity. For full mixed-precision training with Adaptive Moment Estimation (Adam), the parameter, gradient, and optimizer-state budget alone can exceed an 80 GB accelerator at roughly 5 billion parameters before activations; larger 10–20B models require sharding, offload, or other memory-saving techniques (Rajbhandari et al. 2020). Assumption Provenance records the 80 GB H100 and A100 capacity figures used in these budgets throughout the chapter, so the reader can trace every memory ceiling back to a single documented source.
Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. “Language Models Are Few-Shot Learners.” Advances in Neural Information Processing Systems 33: 1877–901. https://doi.org/10.48550/arxiv.2005.14165.
The second signal is unacceptable Training Duration. Even when a model fits, single-device training may require weeks or months to converge, making wall-clock time itself a systems constraint. GPT-3’s 175B-parameter training run used a cluster of V100 GPUs (Brown et al. 2020), illustrating why the calendar, not just HBM capacity, forces distribution at this scale.
The third signal is Dataset Scale. When training data reaches multiple terabytes, as occurs in large-scale vision or language modeling tasks, a single machine is no longer the natural unit of storage or input throughput. Distributed training then becomes a way to feed the model as much as a way to hold it.
Distributed training complexity trade-offs
Distribution changes the optimization problem by adding costs that do not exist inside one machine. Three complexity dimensions decide whether a parallelism strategy is viable, because any one of them can become the binding ceiling. Communication Overhead is the cost of synchronizing gradients: for a model with \(P\) parameters distributed across \(N\) devices, all-reduce operations must transfer approximately \(2P(N-1)/N\) gradient values per step, multiplied by the bytes per gradient element, and on commodity networks this can dominate computation time. Fault Tolerance grows harder as the cluster grows, because the expected number of failures per unit time rises linearly with cluster size while the probability that the entire cluster survives an interval without any failure decays exponentially; if a 100-node cluster has 99.9 percent per-node hourly survival, the cluster-level failure probability is 9.5 percent per hour, corresponding to a mean time between failures (MTBF) of about 10 hours. The MTBF cascade derives this cascade from per-node survival to cluster MTBF and works the same calculation through a full example, so the reader can reproduce why MTBF collapses as the cluster grows. Algorithmic Stability closes the set, because large batch sizes from data parallelism affect convergence behavior, requiring learning rate scaling and warmup strategies that single-machine training does not require (Goyal et al. 2017). Together, these costs explain why distributed training is a constraint satisfaction problem rather than a hardware multiplication exercise.
Single-machine to distributed transition
The systematic optimization methodology established for single-machine training extends to distributed environments with important adaptations. Profiling must capture inter-device communication patterns and synchronization overhead in addition to computation and memory metrics. The solution space expands to include data parallelism, model parallelism, pipeline parallelism, and hybrid approaches. Figure 1 visualizes this three-dimensional configuration space.
Figure 1 shows that the total accelerator count is \(N_{\text{total}} = d \times p \times t\). Each axis is independent, and training systems select a specific coordinate in this cube based on the memory, compute, and bandwidth constraints of the target model and cluster.
Engineering trade-offs: Selecting a parallelism strategy
Choosing the right parallelism strategy is not a matter of preference; it is a constraint satisfaction problem governed by parameter count \((P)\), batch size \((B)\), and interconnect bandwidth. Table 1 quantifies the parallelism communication costs for each strategy, revealing which approaches are physically feasible for a given hardware topology.
| Strategy | Communication Pattern | Comm. Volume | Hardware Constraint |
|---|---|---|---|
| Data Parallel (DP) | AllReduce Gradients | \(\propto M_{\text{grad}}\) (gradient bytes) | Requires high bisection BW |
| Tensor Parallel (TP) | AllReduce Activations | \(\propto B \times N_L\) (Layers) | Critical: Needs NVLink |
| Pipeline Parallel (PP) | Point-to-Point (P2P) | \(\propto B \times S \times d_{\text{model}}\) (Activations) | Low BW (Ethernet is sufficient) |
The bandwidth requirements impose a hard constraint on hardware placement: each parallelism strategy must be matched to the interconnect available between the GPUs that need to communicate, or the design fails before it leaves the whiteboard. A quick back-of-envelope bandwidth feasibility check catches that mismatch before any code is written.
Systems Perspective 1.1: Bandwidth feasibility check
Tensor parallelism across server racks connected by standard Ethernet will stall. The communication volume scales with batch size \(B\) and layer count \(N_L\). It requires the 600 GB/s–900 GB/s throughput of NVLink. For cross-rack scaling, the design must switch to pipeline or data parallelism to respect the physics of the network.
Figure 2 formalizes this constraint satisfaction process as a decision tree, showing how model size and hardware topology determine the viable parallelism strategies.
The decision tree reveals that parallelism strategy selection is not a preference but a consequence of physical constraints. This is a constraint-level preview: the communication-volume formulas in table 1 name what each strategy moves, but their full meaning depends on the mechanics of tensor, pipeline, and hybrid parallelism developed over the rest of the chapter. Section 1.8 completes the filter once those mechanics are in place. The next question is how these constraints shape the mechanics of a distributed training step on a real cluster.
The Distributed Training Step
Partitioning a training job across accelerators requires preserving the mathematical semantics of a single optimization step. The central challenge is ensuring that 1,024 GPUs, operating completely independently, agree on a single, mathematically rigorous set of updated weights at the end of each training iteration.
Definition 1.1: Distributed training
Distributed Training is a training methodology that partitions the optimization loop across multiple compute nodes (distributing either data, model layers, or individual tensor operations) and coordinates their outputs through synchronized communication primitives to produce a single coherent model.
- Significance: Distributed training becomes necessary when a model’s memory requirement exceeds a single accelerator’s capacity. GPT-3 (175B parameters) requires approximately 350 GB in BF16—more than 4\(\times\) the 80 GB capacity of a single H100. Training it requires at least 5 H100s for model sharding alone, while large runs may use hundreds or thousands of accelerators to reach tractable wall-clock time.
- Distinction: Unlike single-node multi-threading (which shares a coherent physical memory space across cores), distributed training partitions state across disjoint accelerator memories, requiring explicit collective communication primitives over interconnect fabrics to synchronize optimization state.
- Common pitfall: A frequent misconception is that distributed training scales linearly with node count. In practice, communication overhead grows with cluster size and the serial fraction of each step (Amdahl’s Law): with 30 percent of a step’s time spent on synchronization, the theoretical scaling ceiling is \(1/0.30 \approx 3\times\) regardless of how many accelerators are added.
A useful mental model frames these distributed strategies as loop transformations, the same conceptual toolkit that compilers use to optimize sequential code. If we view the training process as a massive loop over data and layers, distributed strategies are simply loop transformations applied by the cluster-level compiler. The logical training loop nests three iterators (epochs, batches, layers), and each parallelism strategy unrolls one of them across devices:
Data parallelism is the parallel for-loop, unrolling the outer loop (the batch dimension) across devices so that each device runs the same code body on different data indices. Tensor parallelism is vectorization, or single instruction, multiple data (SIMD), splitting the inner loops (matrix multiplication) across devices in a cluster-scale SIMD where NVLink acts as the vector register file. Pipeline parallelism is instruction pipelining, splitting the sequential operations (layers) across devices; just as a CPU pipeline stages fetch, decode, and execute, the cluster stages Layer 1, Layer 2, and Layer 3 to keep all compute engines busy.
Whichever loop a strategy unrolls, distributed training1 spreads the workload across machines that must coordinate to train a single model. Coordination here means keeping every model shard or replica on a compatible training step. Basic barriers can keep a small research run ordered, but long-running training jobs also need timeout, checkpoint, and recovery mechanisms so one failed worker does not waste days of compute. Fault Tolerance examines those reliability engineering challenges in depth.
1 Distributed Training: Google’s DistBelief (2012) was an early framework for training neural networks across thousands of machines, but its parameter server architecture created bandwidth bottlenecks at central nodes. This limitation drove the shift to decentralized AllReduce patterns in successors like Horovod and PyTorch DistributedDataParallel (DDP), where replicated data-parallel workers synchronize gradients at cost \(2(N-1)/N\) per worker rather than concentrating traffic at a single server (Dean et al. 2012; Sergeev and Balso 2018; Li et al. 2020; Patarasuk and Yuan 2009).
Dean, Jeffrey, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, et al. 2012. “Large Scale Distributed Deep Networks.” In Advances in Neural Information Processing Systems (NeurIPS), edited by Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, vol. 25. Curran Associates.
Sergeev, Alexander, and Mike Del Balso. 2018. “Horovod: Fast and Easy Distributed Deep Learning in TensorFlow.” CoRR abs/1802.05799.
Li, Shen, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, et al. 2020. “PyTorch Distributed: Experiences on Accelerating Data Parallel Training.” Proceedings of the VLDB Endowment 13 (12): 3005–18. https://doi.org/10.14778/3415478.3415530.
Patarasuk, Pitch, and Xin Yuan. 2009. “Bandwidth Optimal All-Reduce Algorithms for Clusters of Workstations.” Journal of Parallel and Distributed Computing 69 (2): 117–24. https://doi.org/10.1016/j.jpdc.2008.09.002.
2 NVLink: NVIDIA’s point-to-point GPU interconnect delivers 600 GB/s–900 GB/s bidirectional bandwidth, roughly 24×–36× InfiniBand HDR per port. This bandwidth gap is why tensor parallelism, which requires AllReduce on every layer, is confined to intra-node communication, while pipeline and data parallelism tolerate the slower inter-node fabric.
3 NCCL (NVIDIA Collective Communications Library): NCCL provides topology-aware inter-GPU collective communication primitives, including AllReduce, across PCIe, NVLink, InfiniBand, and IP networks. Current NCCL deployments expose ring, tree, and related algorithm families through runtime selection and tuning controls, so training frameworks can avoid naive mappings that route all traffic through the slowest inter-node path (Jeaugey 2017; NVIDIA 2026).
Jeaugey, Sylvain. 2017. NCCL 2.0. GPU Technology Conference presentation.
NVIDIA. 2026. NCCL User Guide.
Each rung of the scaling path inherits the previous one’s challenges. Single-GPU training needs only local memory management and forward/backward passes. Scaling to multiple GPUs within a node adds high-bandwidth communication, handled through NVLink2 or PCIe with NCCL3 optimization while preserving single-machine fault tolerance and scheduling.
The leap to multi-node training adds network communication overhead, fault tolerance requirements, and cluster orchestration. Because each stage compounds the previous one’s bottlenecks, single-GPU performance must be optimized before scaling out, so inefficiency does not multiply across the fleet. Although frameworks abstract away much of this through sharded data parallelism and communication libraries, implementing distributed training efficiently still demands careful network configuration (InfiniBand tuning, topology-aware routing), infrastructure management through cluster schedulers, and debugging of nonlocal issues such as synchronization hangs and communication bottlenecks.
Despite this complexity, the core workflow is mechanically straightforward; the engineering challenge is making it fast and reliable at scale. The recurring cost is the gradient synchronization that aggregates results across devices, an overhead that compounds as systems scale, as section 1.3 quantifies.
Four approaches address different constraint regimes. Data parallelism divides the training data across machines while each maintains a full model copy, making it the simplest approach for models that fit in single-device memory. Model parallelism splits the model itself across devices when parameters exceed single-device memory. Pipeline parallelism partitions models into sequential stages that process microbatches concurrently, improving utilization over naive model parallelism. Hybrid approaches integrate multiple strategies, enabling training at scales where any single approach would fail. Each strategy becomes necessary only after its predecessor reaches a physical ceiling.
Self-Check: Question
What structural requirement most fundamentally distinguishes distributed training from a stateless distributed web service that handles independent HTTP queries across multiple replicas?
- Distributed training must run on strictly more nodes than web services because neural networks cannot execute on small clusters
- Distributed training requires all workers to maintain a consistent mathematical view of mutable shared parameters via collective gradient synchronization
- Distributed training is always compute-bound while web serving is strictly latency-bound, preventing the workloads from sharing cluster hardware
- Distributed training cannot tolerate hardware failures because synchronization protocols prevent checkpoint-based recovery
Order the causal phases of a single synchronous data-parallel training iteration from earliest to latest:
- Update local model parameters using aggregated gradients via the optimizer
- Partition and assign a disjoint batch shard to each worker rank
- Synchronize local gradient tensors across all workers using AllReduce
- Compute forward and backward passes locally on the assigned batch shard
Framing distributed parallelism strategies as compiler loop transformations across the training loop’s iterators (batches, layers, and operations), which distributed strategy corresponds to vectorization (SIMD) of inner matrix multiplications?
- Tensor parallelism, which splits inner matrix-multiplication operations across devices with NVLink acting as a cluster-scale register fabric
- Data parallelism, which unrolls the outer batch loop across devices so each rank processes independent data indices
- Pipeline parallelism, which stages sequential layer operations across devices analogous to CPU instruction pipelining
- Model replication, which duplicates the entire forward and backward execution redundantly across machines
A 1,024-GPU Bulk Synchronous Parallel (BSP) training job reports a 180 ms average per-worker compute time, but the measured step time averages 340 ms with a p99 latency of 720 ms. Explain the mechanism behind this discrepancy and why it worsens as cluster size grows.
True or False: In a synchronous distributed training cluster using collective communication, if control-flow divergence causes worker Rank 0 to execute an AllReduce collective while Rank 1 skips it, the cluster will automatically recover by substituting stale gradients from the previous step.
Data Parallelism
The simplest approach gives each GPU a complete, identical copy of the model and assigns it a distinct slice of the data. Data parallelism is the natural starting point for distributed training because it requires minimal changes to the single-device training loop.
Definition 1.2: Data parallelism
Data Parallelism is a distributed training strategy in which each worker holds a complete replica of the model and processes an independent shard of the minibatch, then synchronizes gradient updates via AllReduce so all replicas apply identical parameter changes each step.
- Significance: With \(N\) workers each processing batch size \(B\), the effective global batch size is \(N \times B\), scaling throughput linearly while keeping per-worker memory constant. For a 1B-parameter model at 2 GB in BF16, 1,024 workers achieve 1,024\(\times\) the single-GPU throughput—until the gradient AllReduce (2 GB per step at ring-optimal \(2(N-1)/N\) per worker) exceeds backward compute time and creates the communication bottleneck.
- Distinction: Unlike model and pipeline parallelism (which partition the network’s layers or operations across accelerators), data parallelism replicates the entire model across all workers and partitions only the training dataset, requiring gradient synchronization after each backward pass.
- Common pitfall: A frequent misconception is that data parallelism scales indefinitely with worker count. Scaling the effective batch size \(B\) beyond the workload-dependent critical batch size degrades statistical efficiency, requiring more training steps to reach target loss and eroding the throughput gains from adding more workers (Shallue et al. 2019).
Each device trains a complete copy of the model using its assigned subset of the data. When training an image classification model on 1 million images using 4 GPUs, each GPU processes 250,000 images while maintaining an identical copy of the model architecture.
Data parallelism is most effective when the dataset size is large but the model size remains manageable, since each device must store a full copy of the model in memory. This method is widely used in image classification and natural language processing, where the dataset can be processed in parallel without dependencies between data samples. When training a ResNet model (He et al. 2016) on ImageNet, each GPU can independently process its portion of images because the classification of one image does not depend on the results of another.
He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. “Deep Residual Learning for Image Recognition.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–78. https://doi.org/10.1109/cvpr.2016.90.
The effectiveness of data parallelism stems from a property of stochastic gradient descent. Gradients computed on different minibatches can be averaged while preserving mathematical equivalence to single-device training. This property enables parallel computation across devices, with the mathematical foundation following directly from the linearity of expectation.
Consider a model with parameters \(\theta\) training on a dataset \(D\). The loss function for a single data point \(x_i\) is \(\mathcal{L}(\theta, x_i)\). In standard stochastic gradient descent (SGD) with batch size \(B\), the gradient update for a minibatch is: \[ g = \frac{1}{B} \sum_{i=1}^B \nabla_{\theta} \mathcal{L}(\theta, x_i) \]
In data parallelism with \(N\) devices, each device \(k\) computes gradients on its own minibatch \(B_k\): \[ g_k = \frac{1}{|B_k|} \sum_{x_i \in B_k} \nabla_{\theta} \mathcal{L}(\theta, x_i) \]
When all workers use the same local batch size, the global update averages these local gradients: \[ g_{\text{global}} = \frac{1}{N} \sum_{k=1}^N g_k \]
Under that equal-batch assumption, the averaging is mathematically equivalent to computing the gradient on the combined batch \(B_{\text{total}} = \bigcup_{k=1}^N B_k\): \[ g_{\text{global}} = \frac{1}{|B_{\text{total}}|} \sum_{x_i \in B_{\text{total}}} \nabla_{\theta} \mathcal{L}(\theta, x_i) \]
For unequal local batch sizes, the combined-batch gradient is the weighted average \(g_{\text{global}} = (1/|B_{\text{total}}|)\sum_k |B_k|g_k\). The equivalence shows why data parallelism maintains the statistical properties of SGD training: distributing distinct data subsets across devices, computing local gradients independently, and averaging them approximates the full-batch gradient. The averaging step itself is an AllReduce over the local gradient tensors; AllReduce derives the ring and tree AllReduce cost models that set the price of this synchronization, so the reader can predict when it overtakes the local compute saved by parallelism.
Checkpoint 1.1: Data parallelism mechanics
Verify your understanding of how data parallelism distributes work:
The method parallels gradient accumulation, where a single device accumulates gradients over multiple forward passes before updating parameters. Both techniques use the additive properties of gradients to process large batches efficiently. However, moving the same idea into a cluster introduces operational challenges beyond this theoretical equivalence. Communication overhead, node failures, and cost constraints each impose second-order effects that the single-machine derivation does not capture.
Data parallelism implementation
The implementation details matter because the SGD equivalence only holds when each phase preserves disjoint data, complete local gradients, and a single synchronized update. The concrete workflow therefore traces the path from distributing data subsets to synchronizing the computed gradients. Consider figure 3: it traces the complete workflow from dataset splitting through gradient aggregation, showing how each GPU processes its assigned batch before synchronization brings all gradients together for parameter updates.
The critical synchronization point is stage 4 (figure 3): AllReduce must complete before any GPU can update parameters, making gradient communication the dominant bottleneck as the device count grows.
Dataset splitting
Data splitting is the first place where the SGD equivalence can fail: each worker must see a unique, deterministic slice of the epoch. With a dataset of 100,000 training examples and 4 GPUs, each GPU receives 25,000 examples per epoch. The DistributedSampler must ensure no overlap between subsets to maintain gradient estimation validity: if two GPUs process the same example, the resulting gradient average would overweight that example, violating the unbiased gradient assumption that makes data parallelism mathematically equivalent to single-device training.
The sampler is therefore part of the training system, not only an input-loader convenience. Modern distributed training frameworks handle this distribution automatically through a distributed sampler that implements prefetching and caching mechanisms to keep accelerators fed without changing sample ownership. The sampler coordinates across workers using the process rank, the integer worker identifier assigned by the distributed runtime, to deterministically partition indices, ensuring reproducibility when the same random seed is used. For a 1.2 million example dataset distributed across 32 GPUs, each GPU processes approximately 37,500 examples per epoch, with the sampler padding the final batch to maintain consistent batch sizes across all workers.
Compute phase: Forward and backward passes
The defining feature of data parallelism is that the computation phase, both forward and backward, is embarrassingly parallel. Each GPU operates as an isolated island, executing an identical copy of the model on a unique micro-batch of data. Here, micro-batch means the per-GPU local slice used for activation accounting; pipeline microbatching uses the same word for a different mechanism, subdividing a global batch to keep pipeline stages occupied. For our 175B parameter reference model, this isolation is critical: during the forward pass, each GPU independently computes activations for its local batch (micro-batch size 4, sequence length 2048). Without optimization, storing these activations for backpropagation would consume roughly 1.1 TB of HBM, an order of magnitude beyond the capacity of even an H100 GPU. Activation checkpointing, which recomputes activations during the backward pass rather than storing them, becomes necessary in this scenario to suppress the footprint to ~19.3 GB.
The backward pass mirrors this independence but introduces the system’s primary bottleneck. As the GPU traverses the computation graph in reverse, it computes gradients for the parameters held by that replica or shard. Under pure data parallelism, a full 175B FP16 replica would imply a 350 GB gradient tensor per worker, which exceeds single-accelerator memory budgets; large runs therefore combine data parallelism with sharding, tensor parallelism, or pipeline parallelism. The computation itself requires zero communication, yet the resulting gradients represent a fractured view of the true loss surface, valid only for the local micro-batch. Before the optimizer step can occur, the corresponding local gradients or gradient shards must be aggregated across data-parallel workers to form a valid global update. The transition from isolated, high-throughput compute to synchronization defines the rhythm of data parallel training: long periods of silent, intense arithmetic punctuated by bursts of heavy network traffic.
Gradient synchronization
Gradient synchronization is where independent SGD estimates become one update, so its cost determines whether data parallelism still behaves like a scaling strategy rather than a network benchmark. The immediate training requirement is simple: every data-parallel replica must apply the same averaged gradient before it moves to the next step. AllReduce is the primitive that performs this operation for replicated tensors: each worker contributes its local gradient tensor, the fleet sums the tensors, and every worker receives the same reduced result. In sharded variants, ReduceScatter and AllGather move corresponding pieces rather than the full tensor on every device, but the same synchronization cost remains. Small tensors are dominated by synchronization latency because each communication round has a startup cost; large tensors are dominated by bandwidth because the relevant gradient payload must cross links. Collective Communication later derives the ring, tree, and hierarchical algorithms that implement this averaging operation on real fabrics.
When synchronization performance deviates from theoretical expectations, the fleet stack framework provides a structured approach to isolating the bottleneck.
Example 1.1: Debugging slow gradient synchronization
Scenario: An AllReduce of a 3 GB gradient tensor across 128 nodes (1,024 GPUs) takes 100 ms.
Diagnosis: Latency falls well below the 239.8 ms flat-ring bound, confirming NCCL uses a hierarchical algorithm. However, switch counters reveal inter-node InfiniBand links run at only 60 percent utilization due to uplink congestion.
Systems lesson: Debugging collective communication requires evaluating physical link utilization against algorithmic bounds. The gap between 48.5 ms theoretical and 100 ms observed latency stems from network fabric uplink contention, requiring rail-optimized topologies rather than algorithm changes.
Stepping back from that specific cluster to the design space, figure 4 contrasts three high-level synchronization topologies: the bandwidth-optimal ring AllReduce, the centralized parameter server, and the fully connected all-to-all mesh. In dense synchronous data-parallel settings, ring AllReduce can avoid a single reducer bottleneck by distributing traffic evenly across participating links, as Baidu’s implementation illustrates (Gibiansky 2017).
Gibiansky, A. 2017. Bringing HPC Techniques to Deep Learning. Baidu Research Technical Blog.
The trade-off visible in figure 4 is between bandwidth and latency: a ring spreads bandwidth evenly but adds a latency step per hop, while an all-to-all mesh collapses the latency path to constant rounds at the cost of a link count that grows quadratically with the node count. High-performance libraries such as NCCL select among these topologies automatically based on message size and cluster topology. Collective Communication formalizes the latency complexity of each topology and derives when a ring beats the alternatives.
Synchronization models
Distributed training systems operate under explicit synchronization models that govern when workers observe each other’s updates. The choice of model determines whether the system guarantees mathematical equivalence to single-device training or trades consistency for throughput. The baseline model, bulk synchronous parallel (BSP)4 (Valiant 1990), requires all workers to complete their local computation in forward and backward passes, synchronize gradients through a barrier with AllReduce, and then simultaneously update parameters.
4 Bulk Synchronous Parallel (BSP): Introduced by Valiant (1990) as a “bridging model” between hardware and software for parallel computation. BSP divides work into supersteps (compute, communicate, barrier), guaranteeing mathematical equivalence to sequential execution. The cost: iteration time equals the slowest worker’s time, and at 1,000 GPUs with 1 percent straggler probability per device, roughly 10 GPUs straggle every step, making the barrier increasingly expensive.
Valiant, Leslie G. 1990. “A Bridging Model for Parallel Computation.” Communications of the ACM 33 (8): 103–11. https://doi.org/10.1145/79173.79181.
BSP provides strong guarantees where every worker sees identical parameter values at each step, ensuring mathematical equivalence to single-device training. The cost is that the slowest worker determines iteration time, creating the straggler problem.
Stale synchronous parallel (SSP) relaxes this constraint by allowing workers to proceed up to \(s\) iterations ahead of the slowest worker before blocking. This bounds staleness while reducing synchronization delays. SSP requires careful learning rate tuning since workers compute gradients on slightly different parameter versions. The bounded staleness guarantee provides a middle ground between BSP’s strong consistency and fully asynchronous approaches (Ho et al. 2013).
Asynchronous SGD (ASP) eliminates synchronization barriers entirely as workers update parameters independently. This maximizes hardware utilization but introduces gradient staleness that can degrade convergence. The operational guarantee determines how much convergence risk the system accepts; section 1.3.4 develops the convergence rates, the staleness penalty, and the compensation techniques each model requires.
The key trade-offs across synchronization models are summarized in table 2, and figure 5 illustrates how each strategy schedules work across workers over time.
| Model | Consistency | Throughput | Convergence | Use Case |
|---|---|---|---|---|
| BSP | Strong | Bounded by slowest worker | Equivalent to single-GPU | Final training runs, reproducibility |
| SSP | Bounded staleness | Higher than BSP | Near-equivalent with tuning | Hyperparameter search |
| Async | Weak | Maximum | Degraded, requires compensation | Large heterogeneous clusters |
The same trade-off becomes clearer when the schedules are placed on a timeline.
The choice of synchronization model directly affects both system throughput and model convergence. Training teams often use BSP for final runs to preserve reproducibility, while exploring SSP or async approaches during hyperparameter search where exact reproducibility is less critical.
Barrier semantics and failure modes
AllReduce operations implement implicit barriers where no worker can proceed until all workers have contributed their gradients. This coupling creates failure modes absent from single-device training.
Worker failures during AllReduce cause all other workers to block indefinitely while waiting for the missing contribution. Without timeout mechanisms, the entire training job hangs rather than failing cleanly. Deployed systems often implement watchdog timers on the order of minutes to detect and terminate stuck jobs.
One missing worker stalls the entire AllReduce barrier.
Gradient mismatches occur when workers disagree on which tensors to synchronize due to conditional computation paths or dynamic batching. AllReduce operations may block waiting for tensors that some workers never send. This commonly occurs with variable-length sequences in natural language processing models, dynamic computation graphs, and mixture-of-experts with different routing decisions.
Straggler-induced delays arise because iteration time equals the slowest worker’s time plus synchronization overhead. A single slow worker, whether due to thermal throttling, network congestion, or OS jitter, delays all workers and reduces cluster utilization. At 1,000 GPUs with 1 percent probability of straggler per GPU per step, approximately 10 GPUs straggle every iteration.
Deployed systems address these issues through timeouts, heartbeat monitoring, and elastic training mechanisms. Fault Tolerance provides comprehensive coverage of failure detection, checkpointing strategies, and recovery mechanisms that enable training jobs to complete despite inevitable hardware failures.
Parameter updating
Parameter updating closes the data-parallel invariant: after aggregation, every device must apply the same optimizer update from the same gradient values. Each device independently updates model parameters using the chosen optimization algorithm such as SGD with momentum or Adam. This decentralized update strategy avoids a central coordination server because synchronization has already made the local gradients identical.
In a system with 8 GPUs training a ResNet model, each GPU computes local gradients based on its data subset. After gradient averaging via ring all-reduce (Patarasuk and Yuan 2009), every GPU has the same global gradient values. Each device then independently applies these gradients using the optimizer’s update rule. With SGD and learning rate 0.1, the update becomes weights = weights - 0.1 * gradients. The example shows why the update can remain decentralized without sacrificing mathematical equivalence to single-device training.
The cycle of splitting data, computing gradients, synchronizing results, and updating parameters repeats for each batch. Frameworks automate this cycle, but they cannot remove the ordering constraint that makes the replicas coherent: every worker must update only after the synchronized gradient is complete.
Trade-offs: The communication wall
Data parallelism is a common starting strategy for a reason: it scales throughput linearly with device count, provided the model fits in memory and communication is not the bottleneck. However, it hits a hard ceiling defined by the communication-computation ratio: once gradient exchange dominates useful computation, more workers mainly add synchronization work (Ben-Nun and Hoefler 2019).
Ben-Nun, Tal, and Torsten Hoefler. 2019. “Demystifying Parallel and Distributed Deep Learning: An in-Depth Concurrency Analysis.” ACM Computing Surveys 52 (4): 1–43. https://doi.org/10.1145/3320060.
Data parallelism offers three principal advantages. Throughput scales linearly for compute-bound models: scaling ResNet-50 on ImageNet from 1 to 256 GPUs yields near-linear speedup because the gradient exchange is small relative to the compute time. The model architecture also remains unchanged; the framework wraps the model in a data-parallel container that intercepts backward-pass hooks to trigger gradient synchronization automatically. Utilization stays high because, unlike model parallelism, there are no pipeline bubbles: all GPUs work on the forward and backward pass simultaneously.
Three hard ceilings limit these advantages. The memory wall requires every GPU to hold a full copy of the model parameters, gradients, and optimizer states; for a 175B-parameter model, this demands more than 1 TB of memory per GPU, exceeding per-device HBM budgets without ZeRO sharding. The bandwidth wall emerges as \(N\) grows: the AllReduce cost \(\frac{2(N-1)}{N} \times \frac{M}{\text{BW}_{\text{net}}}\) eventually dominates, and for large language models gradient synchronization can consume more than 50 percent of the step time, collapsing efficiency. The batch size trap compounds the problem because scaling to thousands of GPUs requires increasing the global batch size \((B_{\text{global}} = N \times B_{\text{local}})\), and eventually the critical batch size is reached, where adding more data per step yields diminishing returns in convergence.
Napkin Math 1.1: GPT-2 data parallel scaling: Single node
A GPT-2 scaling scenario makes the efficiency loss concrete by holding the model fixed and changing only the communication domain. The first case keeps all eight GPUs inside a single NVLink node.
Single GPU Baseline
- Batch size: 16 (with gradient checkpointing, fits in 32 GB)
- Time per step: 1.8 s
- Time to 50K steps: 25 hours
8 GPUs: Single Node with NVLink
- Per-GPU batch: 16, global batch: 128
- Gradient synchronization: 5.2 GB @ 450 GB/s (NVLink, per direction) \(\approx\) 11.7 ms
Performance results:
- Compute: 1800 ms per step
- Communication: 11.7 ms per step
- Total: 1811.7 ms per step
- Speedup (throughput): 8×
- Parallel efficiency: 99.4 percent
Training time: 25 hours ÷ 8 = 3.1 hours
Inside the node, NVLink keeps gradient exchange small relative to compute, so efficiency stays near 99.4 percent. The picture inverts once the same model must synchronize across nodes over a commodity network.
Napkin Math 1.2: GPT-2 data parallel scaling: Commodity scale-out
The second case scales the same GPT-2 run across multiple nodes, replacing the intra-node NVLink hop with inter-node Ethernet for the bulk of the AllReduce.
Commodity network configuration: 32 GPUs across 4 nodes
- Per-GPU batch: 16, global batch: 512
- Intra-node communication: 11.7 ms (NVLink)
- Inter-node communication: 5.8 GB @ 1.25 GB/s (10GbE) \(\approx\) 4650 ms
Performance results:
- Compute: 1800 ms (27.9 percent of time)
- Communication: 4661.7 ms (72.1 percent of time), so communication dominates and becomes the bottleneck.
- Total: 6461.7 ms per step
- Speedup (throughput): 8.9× faster → 2.8 hours
- Parallel efficiency: 27.9 percent
Gradient accumulation offers a direct remedy by keeping all communication within a single node’s NVLink domain while still training on an equivalently large effective batch.
Napkin Math 1.3: Gradient accumulation speedup
Problem: A GPT-2 run on a commodity 10G network is communication-bound at 32 GPUs, costing $1,436 for a fixed number of samples. Can a single 8-GPU node achieve the same effective batch size more efficiently using gradient accumulation?
Math:
- Effective batch size: 8 GPUs \(\times\) batch 16 \(\times\) 4 accumulation steps = 512.
- Communication overhead: With 4-step accumulation, we AllReduce once every 4 steps.
- Overhead = 11.7 ms / (4 \(\times\) 1800 ms) \(\approx\) \(0.162\%\).
- Training duration: Total time is 3.1 hours.
- Scale-out baseline cost: 4 nodes \(\times\) 2.8 hours \(\times\) $128/hr = $1,436.
- Accumulation cost: 3.1 hours \(\times\) $128/hr = $401.
Systems insight: Gradient accumulation saves $1,035 (72.1 percent) by concentrating computation where bandwidth is abundant (NVLink within the node) and minimizing the frequency of synchronization. When the network is slow, do not scale out—scale the batch size locally.
The calculation changes the scaling decision: when inter-node bandwidth is the binding constraint, gradient accumulation on a bandwidth-rich node can beat naive scale-out even if the wall-clock run becomes modestly longer. Four insights emerge. NVLink enables efficient scaling within single nodes (99.4 percent efficiency), while inter-node communication kills efficiency (dropping to 27.9 percent). Gradient accumulation beats naive scale-out for communication-bound runs when the scale-out network is slow, so the sweet spot for this GPT-2 scenario is 8 GPUs per node with gradient accumulation, not naive scaling to 32+ GPUs. OpenAI’s GPT-2 paper reports training on 32 V100s across 4 nodes using optimized communication (likely gradient accumulation combined with pipeline parallelism), not pure data parallelism.
Memory-efficient data parallelism: ZeRO and FSDP
The memory constraints of data parallelism motivate a family of techniques that shard memory state across workers while preserving the simplicity of data parallel training. The Zero Redundancy Optimizer (ZeRO)5 (Rajbhandari et al. 2020) and its PyTorch implementation, Fully Sharded Data Parallel (FSDP) (Zhao et al. 2023), enable training models that would otherwise require model parallelism.
5 ZeRO (Zero Redundancy Optimizer): Published by Microsoft Research in 2019, ZeRO partitions optimizer states, gradients, and optionally parameters across workers instead of replicating them. At ZeRO Stage 3 with 64 GPUs, per-device memory drops from 16 bytes/parameter (full replication) to 0.25 bytes/parameter, converting a 112 GB memory footprint into 1.75 GB. The trade-off: FSDP (PyTorch’s ZeRO-3 implementation) adds AllGather and ReduceScatter on every forward and backward layer, introducing 10–25 percent communication overhead that only pays off when memory pressure justifies it.
Rajbhandari, Samyam, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.” SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, 1–16. https://doi.org/10.1109/sc41405.2020.00024.
Zhao, Yanli, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, et al. 2023. “PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel.” Proceedings of the VLDB Endowment 16 (12): 3848–60. https://doi.org/10.14778/3611540.3611569.
Definition 1.3: Sharded data parallelism
Sharded Data Parallelism is the data-parallelism variant (implemented as the ZeRO stages and FSDP) that partitions optimizer state, gradients, and at the deepest stage the parameters themselves across the data-parallel workers, reconstructing each shard on demand through collectives so that per-worker memory falls toward \(1/N\) of the full training state while every worker still processes its own minibatch shard.
- Significance: Mixed-precision Adam training carries 16 bytes of state per parameter, so a 7B-parameter model requires 112 GB of training state, out of memory on any 80 GB accelerator even though the model fits comfortably for inference. ZeRO Stage 3 across 64 workers cuts the per-device state to 1.75 GB (0.25 bytes per parameter), converting the memory wall into a communication cost: the AllGather and ReduceScatter traffic that reassembles shards on demand adds 10–25 percent overhead per step.
- Distinction: Unlike vanilla data parallelism (which replicates the complete training state on every worker) and unlike model parallelism (which partitions the computation itself), sharded data parallelism partitions only the storage: every worker still executes the full forward and backward pass, gathering each layer’s parameters just in time and discarding them immediately after use.
- Common pitfall: A frequent misconception is that sharding is free memory. The on-demand AllGathers place parameter traffic on the critical path of every layer in every step; on slower interconnects or at small per-worker batch sizes, the exposed communication erodes throughput faster than the memory savings help. Capacity is purchased with bandwidth.
To understand the scale of memory savings ZeRO provides, consider the concrete memory budget for a large language model.
Napkin Math 1.4: ZeRO memory savings
Problem: Training a 7B parameter Llama-2 model in mixed precision requires 16 bytes per parameter for weights, gradients, and optimizer state. Does the full training state fit on a single A100-80 GB, and how does ZeRO Stage 3 with 64 GPUs change the per-device memory requirement?
Baseline: Standard DDP (Replicated State) Per-Parameter Memory Cost:
- Weights (FP16): 2 bytes
- Gradients (FP16): 2 bytes
- Optimizer state (FP32): 12 bytes (4 master weight + 4 momentum + 4 variance)
- Total: 16 bytes/parameter
Total Memory for 7B Model:
\[C_{\text{state,total}} = 7 \times 10^9 \times 16 \text{ bytes} \approx 112 \text{ GB}\]
Baseline outcome: Out-of-memory (OOM) on A100-80 GB.
Optimization: ZeRO-3 (Fully Sharded) With \(N =\) 64 GPUs, state is partitioned:
- Weights: \(2/N\) bytes
- Gradients: \(2/N\) bytes
- Optimizer: \(12/N\) bytes
- Total: 16\(/N =\) 0.25 bytes/parameter effective storage!
Per-GPU Memory:
\[C_{\text{state,ZeRO3}} = \frac{112 \text{ GB}}{64} \approx 1.75 \text{ GB}\]
Result: Fits easily, leaving ~78 GB for activations (batch size).
ZeRO addresses this redundancy through progressive sharding, as figure 6 illustrates and table 3 summarizes.
| Stage | What is Sharded | Memory Reduction | Communication Overhead |
|---|---|---|---|
| ZeRO-1 | Optimizer states only | ~4\(\times\) | None (same as DDP) |
| ZeRO-2 | + Gradients | ~8\(\times\) | ReduceScatter replaces AllReduce |
| ZeRO-3/FSDP | + Parameters | ~\(N\) (linear in workers) | AllGather before each layer |
ZeRO-1 shards optimizer states across GPUs. Each GPU stores only \(1/N\) of the Adam optimizer-related state. After gradient AllReduce, each GPU updates only its shard of parameters, then broadcasts updates to other GPUs. Under the 12-byte convention that counts FP32 master weights, momentum, and variance, memory savings reduce optimizer state from \(12N\) bytes/param to \(12\) bytes/param total across the cluster.
ZeRO-2 additionally shards gradients. Instead of AllReduce, which leaves full gradients on each GPU, ZeRO-2 uses ReduceScatter so each GPU receives \(1/N\) of the reduced gradients. Under the FP16-gradient convention used here, memory savings reduce gradients from \(2N\) bytes/param replicated across \(N\) workers to \(2\) bytes/param total, or \(2/N\) bytes/param per GPU.
ZeRO-3 and FSDP shard parameters themselves. Each GPU stores only \(1/N\) of the model. Before each layer’s forward pass, parameters are gathered via AllGather; after backward pass, gradients are reduced via ReduceScatter, then parameters are discarded. This achieves maximum memory efficiency at the cost of additional communication that FSDP introduces relative to standard DDP.
This sharding places communication on the critical path that DDP avoids. The forward pass needs an AllGather to reconstruct each layer’s parameters (\(M_{\text{layer}}\) bytes); the backward pass needs a second AllGather to reconstruct them when parameters are resharded after the forward pass (\(M_{\text{layer}}\) bytes), followed by a ReduceScatter for gradients (\(M_{\text{layer}}\) bytes). For a model with \(N_L\) layers, full-shard FSDP with resharding therefore performs about \(3N_L\) collective operations per training step against the single AllReduce that DDP needs, raising total communication volume to roughly \(3M_{\text{state}}\) bytes versus \(2M_{\text{state}}\) for DDP. The collectives are spread across more operations with overlap opportunities, however: while layer \(i\) computes, layer \(i+1\) can prefetch its parameters.
The choice between FSDP and DDP depends on model size and memory constraints. When the model fits in GPU memory with room for activations, DDP usually wins because it avoids the repeated AllGather work. As memory pressure rises, ZeRO-2 becomes attractive because it shards gradients and optimizer state while leaving parameters replicated; once parameters themselves exceed single-GPU memory, ZeRO-3/FSDP becomes necessary even though it puts AllGather on the critical path. For training 70B+ models on 80 GB, FSDP typically has to combine with tensor parallelism rather than replace it.
Memory-efficient data parallelism requires careful tuning of sharding strategy (by layer, by transformer block, or flat) and mixed precision settings. The sharding granularity determines the trade-off: finer sharding reduces per-GPU memory but increases communication frequency as more AllGather and ReduceScatter operations must execute per training step.
Eliminating memory as the bottleneck through ZeRO and FSDP makes it tempting to scale data parallelism to hundreds of GPUs. Doing so, however, changes the optimization landscape in ways the communication analysis alone does not predict. Large global batch sizes alter gradient noise statistics, and learning-rate schedules tuned for eight-GPU runs can diverge catastrophically at 256 GPUs. A landmark large-scale demonstration of this failure mode, and an engineering response that became widely influential, came from a single experiment.
War Story 1.1: The one-hour ImageNet run (2017)
Context: Facebook AI Research set out to train ResNet-50 on ImageNet in one hour using 256 GPUs, pushing data parallelism far beyond the batch sizes that earlier practice treated as safe (Goyal et al. 2017).
Mechanism: Scaling global batch size to 8,192 reduced gradient variance per step, causing standard constant-learning-rate SGD optimization to diverge during initial training epochs.
Impact: Without optimization adjustments, training accuracy collapsed, rendering multi-GPU scaling ineffective despite high hardware throughput.
Fix: The team introduced the linear learning-rate scaling rule paired with a 5-epoch gradual warmup schedule, stabilizing optimization at batch size 8,192.
Systems lesson: Distributed training is not just parallel hardware. Scaling changes the optimization regime, so the cluster, communication schedule, batch size, and learning-rate schedule must be tuned as one system.
Self-Check: Question
A 7B-parameter transformer in mixed precision requires 112 GB of static memory for standard replicated DDP (14 GB for FP16 weights, 14 GB for FP16 gradients, and 84 GB for FP32 Adam master weights, momentum, and variance). Under ZeRO Stage 3 (or FSDP Full Shard) across 64 GPUs, what is the resulting static training memory per GPU?
- \(112\text{ GB}\), because ZeRO-3 only reduces dynamic activation memory rather than model state
- \(28\text{ GB}\), because only optimizer states are sharded while parameters and gradients remain fully replicated
- \(1.75\text{ GB}\), because weights, gradients, and optimizer states are each partitioned equally across all 64 workers
- \(0.25\text{ GB}\), because INT4 quantization eliminates floating-point representation entirely
The chapter’s GPT-2 scaling case study shows that 8 GPUs on a single NVLink-connected node using gradient accumulation achieves higher cost efficiency and throughput than naive scale-out to 32 GPUs across a 10 Gb/s commodity Ethernet network. Explain the quantitative mechanism driving this outcome.
Order the memory sharding stages of data parallelism from least sharded (highest memory footprint) to deepest sharded (lowest memory footprint):
- ZeRO-2 (shards optimizer states and gradients using ReduceScatter)
- Standard DDP (full replication of parameters, gradients, and optimizer states)
- ZeRO-3 / FSDP (shards optimizer states, gradients, and model parameters with layer-wise AllGather/ReduceScatter)
- ZeRO-1 (shards optimizer states only, keeping parameters and gradients replicated)
If a model’s parameters, gradients, and activations already fit comfortably within single-GPU memory with high compute utilization under standard DDP, what is the expected throughput impact of enabling ZeRO-3 / FSDP Full Sharding?
- Throughput will increase linearly by \(N\times\) because sharding reduces per-GPU gradient memory
- Throughput will decrease because FSDP introduces additional AllGather collectives before forward and backward passes and ReduceScatter on the critical path
- Throughput will remain exactly identical because NCCL automatically hides all collective communication overhead
- Throughput will double because parameter sharding eliminates the backward pass entirely
Why did modern distributed training frameworks replace centralized parameter servers with decentralized AllReduce collective topologies for dense synchronous training?
- Parameter servers cannot execute stochastic gradient descent updates with mathematical validity
- Parameter servers are restricted to CPU hosts and cannot interface with GPU accelerators
- AllReduce eliminates all network traffic by updating weights locally without exchanging data
- Centralized parameter servers create an \(\mathcal{O}(N)\) network bottleneck at the server’s network interface, whereas AllReduce distributes communication symmetrically across all workers
In standard PyTorch DistributedDataParallel, each worker uses a ____ to partition dataset indices deterministically and prevent sample overlap across ranks based on process rank and epoch seed.
Scaling Efficiency and Convergence
When doubling the number of GPUs yields only 1.5\(\times\) speedup, communication overhead and synchronization barriers have consumed the missing 25 percent of compute budget. Data parallelism revealed the practical mechanics of gradient synchronization and memory sharding, but understanding why scaling efficiency degrades and how convergence changes with parallelism requires a quantitative framework. These metrics and convergence principles apply across all parallelism strategies (data, model, pipeline, and hybrid), governing the fundamental trade-offs among throughput, communication cost, and optimization quality.
The mathematics of scaling efficiency
Evaluating specific parallelism strategies requires first defining the metric that determines whether scaling from one device to many is worthwhile: scaling efficiency. If a model trains in time \(T_1\) on one device, ideal (linear) scaling would train it in time \(T_1/N\) on \(N\) devices. In practice, communication overhead, pipeline bubbles, and load imbalances reduce the speedup. Scaling efficiency is defined by equation 1:
\[\eta_{\text{scaling}} = \frac{T_1}{N \times T_N} \tag{1}\]
where \(T_N\) is the training time on \(N\) devices. An efficiency of 1.0 means perfect linear scaling; an efficiency of 0.5 means we achieve only half the expected speedup.
Definition 1.4: Scaling efficiency
Scaling Efficiency \((\eta_{\text{scaling}})\) is the ratio of actual training throughput to ideal linear throughput when increasing the number of ML compute devices (\(N\)).
- Significance: It is the most important metric for cluster productivity (\(\eta_{\text{scaling}} = \frac{T_1}{N \times T_N}\)). A scaling efficiency of 0.50 means that a 10,000-GPU cluster is delivering only the same useful work as a 5,000-GPU cluster, wasting 50 percent of the hardware investment.
- Distinction: Unlike single-node efficiency (which captures local bottlenecks like \(\text{BW}\)), scaling efficiency captures the cluster-level overhead of communication time \((T_{\text{comm}}(N))\) and synchronization.
- Common pitfall: A frequent misconception is that scaling efficiency is constant. In reality, it is a function of problem size: as \(N\) increases, the communication-to-compute ratio typically worsens (Amdahl’s Law), making it harder to maintain high efficiency for small models.
For data-parallel training of our 175B model, the communication cost per step is dominated by the AllReduce of 350 GB of gradients.
Using ring-AllReduce over InfiniBand at 50 GB/s effective bandwidth, the raw communication time is approximately \(2 \times (N-1)/N \times 350 / 50\), which for large \(N\) approaches \(2 \times 350 / 50\) seconds. With 75 percent overlap between gradient communication and the backward pass, the effective exposed communication time drops to \(T_{\text{comm}}(N) \approx 3.5\) seconds. Under the same assumptions, the compute term is \(T_{\text{compute}}/N \approx 2.1\) seconds, so the exposed step time is about 5.6 s and naive data-parallel scaling efficiency is 2.1 s/5.6 s \(\approx\) 37.5 percent. Well-configured systems can recover much of this loss by combining tensor parallelism, pipeline parallelism, topology-aware placement, and more effective communication overlap.
The scaling efficiency depends critically on computation per communicated byte. At fixed tokens per update, increasing parameter count raises both transformer computation and gradient bytes roughly proportionally, so model size alone does not guarantee better scaling. Larger batches raise computation per optimizer step without changing gradient size; the catch is that large batches can harm convergence and so require learning-rate tuning and warmup schedules. Network bandwidth enters directly because doubling InfiniBand bandwidth halves the bandwidth term, but efficiency changes according to the full step-time denominator rather than in direct proportion. The network fabric is therefore a first-order determinant of cluster productivity, not a secondary concern; its cost (10–15 percent of total system cost) is easily justified if it lifts scaling efficiency by even a few percentage points, because poor scaling efficiency wastes the other 85–90 percent of the investment.
Workloads with more computation per communicated byte are easier to scale.
These factors interact, so efficiency cannot be inferred from model size or GPU count alone. The following worked model fixes the parameter count, tokens per update, accelerator count, effective bandwidth, model FLOPs utilization (MFU), and overlap before calculating one efficiency point. Changing any of those inputs requires recomputing the full step-time model.
However, even for large models, scaling does not continue indefinitely. There exists a Scaling Cliff beyond which marginal speedup no longer justifies the added devices. Under the single explicit model used here (a 175B model, 2M per update, 1,024 GPUs, 50 percent MFU, and 75 percent communication overlap) naive data parallelism reaches only 37.5 percent. The example therefore does not imply a universal 1,024–4,096-GPU efficient region or an 8,192-GPU cliff. Evaluating another cluster size requires recomputing per-device compute time, collective latency and bandwidth cost, overlap, and topology. The model architecture and cluster size must be co-designed to keep those terms in balance.
Napkin Math 1.5: Scaling efficiency for a 175B model
Setup: Training a 175B model on a DGX H100 cluster with 400 Gb/s InfiniBand per GPU.
Compute per step (assuming batch size 2M tokens, 6 FLOPs per parameter per token): \(O_{\text{step}} = 6 \times 175 \times 10^9 \times 2 \times 10^6 \approx 2.1 \times 10^{18}\) FLOPs
Per-GPU compute time on 1,024 GPUs, each at 1979 TFLOP/s FP8 peak (50 percent utilization): \(T_{\text{compute}}/N \approx 2.1\) seconds
AllReduce Time for 350 GB of gradients using ring-AllReduce with overlap: \(T_{\text{comm}}(N) \approx 3.5\) seconds (raw transfer is about 14 seconds; this example assumes 75 percent overlap with the backward pass)
Scaling efficiency: \(\eta_{\text{scaling}} \approx \frac{T_{\text{compute}}/N}{T_{\text{compute}}/N + T_{\text{comm}}(N) + T_{\text{sync}}(N) - T_{\text{overlap}}} = 2.1/5.6 \approx 0.375\) in this simplified example, where synchronization is included in the AllReduce term and 75 percent communication overlap has already been applied.
The low efficiency (37.5 percent) shows why naive data parallelism at this scale is insufficient. Hierarchy-aware systems recover much of that lost efficiency by combining data parallelism with tensor parallelism, which communicates over NVLink, and pipeline parallelism, which overlaps computation with communication.
Parallelism-infrastructure interaction
The scaling efficiency analysis shows that the effective parallelism strategy is not determined by the model architecture alone but by the interaction between the model’s communication requirements and the infrastructure’s bandwidth hierarchy. Each combination of parallelism strategy and infrastructure topology produces a different scaling efficiency curve, and selecting the wrong combination can waste a significant fraction of the cluster’s capacity. The preceding napkin math showed pure data parallelism stalling near 37.5 percent efficiency at this scale; recovering that lost capacity means mapping each parallelism dimension onto the bandwidth tier that can carry its traffic. Section 1.6 works that combination through three concrete configurations and formalizes it as hierarchy-aware parallelism, once tensor and pipeline parallelism have been developed.
Selecting and combining parallelism strategies therefore leads directly to the traffic they create. Network Fabrics examined how topology is co-designed with the parallelism mapping to maximize scaling efficiency; here the next question is how the collective operations themselves shape the step time. AllReduce operations can consume 10–40 percent of total training time in data parallel systems, and this overhead grows with cluster size. BERT-Large on 128 GPUs can experience communication overhead reaching a large fraction of total runtime, while GPT-3-scale models require tensor, pipeline, and data parallelism plus communication overlap to avoid data-parallel gradient synchronization dominating the step.
AllReduce complexity depends on two components: latency \((\alpha)\) and bandwidth \((\beta)\). Ring AllReduce achieves bandwidth-efficient communication with \((N-1)/N\) utilization, while tree-based approaches offer lower latency at \(\mathcal{O}(\log N)\) steps. The choice depends on message size: tree algorithms win for latency-dominated small messages, ring algorithms win for bandwidth-dominated large gradients. High-performance implementations such as NCCL use hierarchical algorithms that combine tree latency within nodes and ring bandwidth between nodes. Collective Communication provides detailed algorithm analysis, including complexity formulas, hierarchical variants, and topology-aware optimizations for large-scale collective operations.
Interconnect selection determines whether large-scale deployments remain compute-bound or collapse into communication-bound regimes, and the bandwidth requirements for efficient distributed training are substantial, particularly for transformer models. Efficient systems often require 100–400 GB/s aggregate bandwidth per node for transformer architectures. BERT-Base (110M parameters) requires approximately 440 MB of gradient synchronization per iteration in FP32, while BERT-Large (340M parameters) requires approximately 1.4 GB. Across 64 GPUs, these synchronization demands require 100–200 GB/s sustained bandwidth for sub-50 ms synchronization latency. For 175B-parameter language models, exact bandwidth requirements depend on the 3D-parallel configuration, gradient accumulation, overlap, and interconnect topology rather than a single universal number.
Synchronization frequency presents a trade-off between communication efficiency and convergence behavior. Accumulating gradients for 4 microsteps reduces synchronization frequency by 75 percent, but the realized step-time reduction depends on the compute/communication mix and how much communication can overlap with backpropagation. In standard implementations, gradient accumulation reuses one resident gradient buffer and accumulates in place; memory pressure comes from keeping that buffer resident and from any larger microbatch or activation choices, not from storing 4 independent gradient tensors. Asynchronous methods eliminate synchronization costs entirely but introduce staleness that degrades convergence by 15–30 percent for large learning rates.
The physics of scaling: Amdahl’s Law with communication
Just as the Iron Law of Processor Performance governs single-thread execution, distributed training is governed by an extended version of Amdahl’s Law that explicitly accounts for communication overhead. The time to complete one training step on \(N\) devices is not simply \(T_{\text{single}} / N\), but is constrained by the sequential nature of synchronization. Amdahl's Law at fleet scale derives the speedup ceiling at fleet scale and works it through a concrete example; the essential consequence here is that a fixed synchronization fraction caps speedup no matter how many accelerators are added.
Figure 7 visualizes this divergence from ideal linear scaling as the Scaling Tax, a direct consequence of the scaling efficiency bound (principle 8). It shows how communication overhead \((r)\) acts as a drag on performance, creating a communication wall where adding more GPUs yields diminishing returns.
The fleet law (principle 10) maps directly onto the iron law’s variables, separating compute, bandwidth, and coordination into additive terms:
\[ T_{\text{step}}(N) = \underbrace{\frac{T_{\text{compute}}}{N}}_{\text{Compute-Time Term}} + \underbrace{T_{\text{comm}}(N)}_{\text{Bandwidth Term}} + \underbrace{T_{\text{sync}}(N)}_{\text{Coordination Term}} - T_{\text{overlap}} \]
The fleet-law terms separate step time into four distinct costs:
- Compute-Time Term \((T_{\text{compute}}/N)\): The total computation required for the batch after ideal partitioning across \(N\) devices.
- Bandwidth Term \((T_{\text{comm}}(N))\): The time spent moving data. This is governed by the iron law’s data-movement term, \(D_{\text{vol}}/\text{BW}\). For Ring AllReduce, this term is \(\frac{2(N-1)}{N} \times \frac{M}{\text{BW}_{\text{net}}}\), where \(M\) is the communicated gradient or model-state bytes and \(\text{BW}_{\text{net}}\) is network bandwidth.
- Coordination Term \((T_{\text{sync}}(N))\): The nonoverlapped cost of barriers, ordering, and straggler waiting.
- Overlap \((T_{\text{overlap}})\): The portion of communication hidden behind computation.
The fleet law leads to the scaling efficiency metric for fixed global work, where \(T_{\text{compute}}\) is the single-device compute time for that work:
\[ \eta_{\text{scaling}} = \frac{T_{\text{compute}}}{N \times T_{\text{step}}(N)} = \frac{1}{1 + \frac{N(T_{\text{comm}}(N) + T_{\text{sync}}(N) - T_{\text{overlap}})}{T_{\text{compute}}}} \]
This is the scaling efficiency bound (principle 8): perfect linear scaling \((\eta_{\text{scaling}} = 1.0)\) is a theoretical limit, not a practical target. Well-configured systems can achieve \(\eta_{\text{scaling}} = 0.85\)–\(0.95\) at moderate scale and degrade further as \(N\) grows. The gap between \(\eta_{\text{scaling}} = 1.0\) and the achieved efficiency is the communication tax and coordination tax: the price of distributed execution.
The equation reveals the Scaling Wall: as \(N\) increases, the compute term \((T_{\text{compute}}/N)\) shrinks, but the communication and synchronization terms can remain constant or grow. Eventually, the denominator is dominated by overhead, driving efficiency toward zero. Beyond wall-clock time, this communication overhead imposes an energy tax that scales with physical distance between devices.
Communication energy climbs from HBM out to the network.
Systems Perspective 1.2: The energy tax of scale
Distributed training is a race against energy as much as against time. In a single GPU, moving a byte from HBM to the cores costs roughly 1–2 pJ/bit. Moving that same byte across an NVLink interconnect costs 5–10 pJ/bit. Moving it across an InfiniBand network through switches costs 20–50 pJ/bit.
At the scale of 10,000 GPUs, the multiplication by aggregate bandwidth is what changes the engineering problem. A full H100 NVLink envelope is roughly 9 PB/s across the cluster; at 7.5 pJ/bit, that movement represents about 540 kW before cooling and switch overhead. The aggregate NDR InfiniBand envelope is smaller (500 TB/s), but at 35 pJ/bit it still represents about 140 kW. Communication-computation overlap is therefore necessary for wall-clock efficiency, but avoiding unnecessary movement is the direct way to reduce the power term itself.
Wall-clock efficiency and the energy tax degrade together as \(N\) grows, following predictable patterns across GPU counts. As a representative rule of thumb, systems in the linear scaling regime of 2–32 GPUs often achieve 85–95 percent parallel efficiency because communication overhead remains small. The communication-bound regime emerges at 64–256 GPUs, where efficiency can drop to 60–80 percent even with well-matched interconnects. Beyond 512 GPUs, coordination overhead can become dominant and limit efficiency to 40–60 percent due to collective operation latency.
Hardware selection critically impacts these scaling characteristics. NVIDIA DGX A100 systems provide 600 GB/s of bidirectional NVLink bandwidth per GPU, with aggregate NVSwitch bandwidth at the system level, enabling high parallel efficiency inside an 8-GPU node. Multi-node scaling requires a network fabric with enough bisection bandwidth; EDR-class 100 Gbps links can support smaller multi-node jobs, while HDR-class 200 Gbps links can support larger clusters when topology, placement, and overlap are well matched.
The efficiency metrics directly influence the choice of parallelism strategy. Data parallelism works well in the linear scaling regime but becomes communication bound at scale. Model parallelism addresses memory constraints but introduces sequential dependencies that limit efficiency. Pipeline parallelism reduces device idle time but introduces complexity in managing microbatches. The effective strategy depends on which constraint (memory, bandwidth, or synchronization) dominates the target workload.
Hardware efficiency metrics govern throughput, but convergence theory determines whether distributed training reaches the same solution quality as single-device training. Parallelism affects optimization convergence in three ways: convergence rate changes with batch size, adding workers yields diminishing returns beyond the critical batch size, and learning rates must scale with batch size.
Convergence rate for synchronous data parallel SGD
The fundamental convergence result for distributed SGD explains what synchronous averaging preserves. Let \(\mathcal{L}\) be lower bounded by \(\mathcal{L}_{\inf}\) and have \(L_s\)-Lipschitz gradients. Each worker computes gradients from \(b\) independent samples, the aggregated estimator \(g_k\) is unbiased, and its conditional variance satisfies \(\mathbb{E}[\|g_k-\nabla\mathcal{L}(\theta_k)\|^2\mid\theta_k]\leq\sigma^2/(Nb)\). These assumptions yield a nonconvex stationarity guarantee without asserting convergence to a global minimizer.
Theorem 1.1: Convergence rate for distributed SGD
For the update \(\theta_{k+1}=\theta_k-\eta g_k\) with constant step size \(0<\eta\leq 1/L_s\), after \(K\) synchronized updates,
\[ \frac{1}{K}\sum_{k=0}^{K-1}\mathbb{E}\!\left[\|\nabla\mathcal{L}(\theta_k)\|^2\right] \leq \underbrace{\frac{2\bigl(\mathcal{L}(\theta_0)-\mathcal{L}_{\inf}\bigr)}{\eta K}}_{\text{optimization term}} + \underbrace{\frac{\eta L_s\sigma^2}{Nb}}_{\text{stochastic-variance term}}. \]
Choosing \(\eta=\min\!\left\{1/L_s,\sqrt{2(\mathcal{L}(\theta_0)-\mathcal{L}_{\inf})Nb/(L_s\sigma^2K)}\right\}\) balances the two terms. Before the smoothness cap becomes active, the stochastic contribution scales as \(\mathcal{O}(1/\sqrt{NbK})\).
The theorem isolates the benefit that follows from the assumptions. With equal local batch size \(b\), synchronous data-parallel SGD computes the same minibatch estimator as one worker using batch size \(Nb\) for that update, and the estimator variance decreases as \(1/(Nb)\). It does not say that every increase in batch size proportionally reduces time to a target loss; learning-rate limits and critical-batch-size effects still apply. This distinction is the Statistical Efficiency of distributed training, separate from hardware efficiency.
The theorem assumes perfect synchronization (BSP). When workers proceed at different rates or use stale gradients, convergence guarantees degrade as staleness increases.
Staleness impact: BSP vs. SSP vs. ASP
The operational guarantees of BSP, SSP, and ASP established in section 1.2.1.4 have convergence consequences, but there is no universal additive staleness penalty independent of the objective, step-size schedule, worker count, and delay model. The staleness parameter \(\tau_{\text{stale}}\) quantifies how many updates occur between computing and applying a gradient, tying throughput to solution quality (Ho et al. 2013; Dutta et al. 2018).
Definition 1.5: Gradient staleness
Gradient Staleness (\(\tau_{\text{stale}}\)) is the number of parameter updates that occur between the time a gradient is computed and the time it is applied to the global model state.
- Significance: It represents the synchronization error in distributed optimization. Increasing \(\tau_{\text{stale}}\) can improve throughput by reducing barrier waits \((T_{\text{sync}}(N))\), but it typically degrades the rate of convergence, requiring more operations \((O)\) to reach the same accuracy.
- Distinction: Unlike network latency, which is a physical delay, staleness is an algorithmic offset that arises from the choice of synchronization protocol (e.g., ASP, SSP).
- Common pitfall: A frequent misconception is that staleness is “always bad.” In reality, it is a throughput-convergence trade-off: for some large-scale workloads, allowing bounded staleness is one practical way to keep thousands of GPUs in use.
The convergence behavior differs across these models in ways that directly affect training cost and solution quality. In BSP (\(\tau_{\text{stale}}=0\)), all workers compute on the same parameter version and aggregate before updating. Under the assumptions of theorem 1.1, the synchronized estimator inherits the \(Nb\) minibatch guarantee.
SSP (\(\tau_{\text{stale}}\leq s\)) allows workers to proceed up to \(s\) clocks ahead of the slowest worker. Ho et al. (2013) prove an average-regret rate \(\mathcal{O}(\sqrt{(s+1)N/T})\) for their SSP update sequence with \(N\) workers and \(T\) updates, assuming convex Lipschitz component functions, a bounded domain, and a decreasing step size. This result makes the staleness dependence explicit, but it is not interchangeable with the nonconvex stationarity theorem 1.1.
Dutta, Sanghamitra, Gauri Joshi, Soumyadip Ghosh, Parijat Dube, and Priya Nagpurkar. 2018. “Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-Offs in Distributed SGD.” Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics 84: 803–12.
Asynchronous SGD imposes no protocol-wide finite delay bound. Removing the barrier can improve update throughput, but a convergence rate requires additional assumptions about realized delays, the objective, and the step-size schedule. Dutta et al. (2018) analyze this as an error-runtime trade-off in which worker speed and gradient staleness jointly determine whether accepting a delayed update helps. Therefore neither a fixed square-of-average-delay penalty nor the complete loss of \(N\)-worker variance reduction follows from asynchronous execution alone.
Table 4 summarizes the convergence properties of each synchronization model.
| Model | Delay control | Scoped guarantee | Systems effect |
|---|---|---|---|
| BSP | \(\tau_{\text{stale}}=0\) | Synchronous \(Nb\)-minibatch theorem 1.1 | Waits for the slowest worker |
| SSP | \(\tau_{\text{stale}}\leq s\) | \(\mathcal{O}(\sqrt{(s+1)N/T})\) average regret under convex assumptions (Ho et al. 2013) | Bounds delay while reducing some waits |
| ASP | No protocol-wide finite bound | No universal rate without additional delay and objective assumptions | Removes the global barrier but applies stale updates |
Ho, Qirong, James Cipar, Henggang Cui, Seunghak Lee, Jin Kyu Kim, Phillip B. Gibbons, Garth A. Gibson, Greg Ganger, and Eric P. Xing. 2013. “More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server.” Advances in Neural Information Processing Systems (NeurIPS) 26.
Learning rate scaling rules
When increasing the effective batch size through data parallelism, the learning rate must be adjusted to maintain convergence quality. Two primary scaling rules have emerged from both theory and practice.
The linear scaling rule (Goyal et al. 2017) states that when the batch size is multiplied by \(k\), the learning rate should also be multiplied by \(k\):
\[ \eta_{\text{large}} = k \cdot \eta_{\text{base}} \]
This rule is justified by approximating one large-batch step at learning rate \(k\eta\) as \(k\) consecutive small-batch steps each at learning rate \(\eta\). With batch size \(kB\), the gradient variance decreases by factor \(k\), so the larger step is well-conditioned; the approximation holds when parameters do not move much during those \(k\) small-batch steps. The rule is empirical rather than universal: Goyal et al. (2017) validated it for ResNet-50/ImageNet at global batch size 8,192 with gradual warmup, while layer-wise adaptive rate scaling (LARS) and layer-wise adaptive moments based optimizer (LAMB) extend large-batch recipes through layer-wise adaptation for ImageNet and BERT-style pretraining (You et al. 2017; You et al. 2020). A Warmup Period that increases the learning rate linearly from \(\eta_{\text{base}}\) to \(k \cdot \eta_{\text{base}}\) over the first \(W\) iterations lets the model reach a region of the loss landscape where large learning rates are stable:
\[ \eta_t = \eta_{\text{base}} + \frac{t}{W}(k \cdot \eta_{\text{base}} - \eta_{\text{base}}) \quad \text{for } t < W \]
The square root scaling rule applies when batch sizes grow so large that linear scaling fails:
\[ \eta_{\text{large}} = \sqrt{k} \cdot \eta_{\text{base}} \]
The more conservative square root rule is motivated by the observation that gradient noise (not just magnitude) affects optimization dynamics. The square root rule better preserves the signal-to-noise ratio of gradient updates. Empirically, square root scaling becomes necessary when batch sizes exceed the critical batch-size regime.
For extreme batch sizes (32K–1M), LARS (You et al. 2017) and its Adam variant LAMB (You et al. 2020) automatically adjust learning rates per layer based on the ratio of weight norm to gradient norm:
\[ \eta_\ell = \eta_{\text{global}} \cdot \frac{\|w_\ell\|}{\|g_\ell\| + \lambda_{\text{wd}}\|w_\ell\|} \]
Here \(\lambda_{\text{wd}}\) is the optimizer’s weight-decay coefficient in the LARS/LAMB update, avoiding collision with the failure-rate parameter used in reliability analysis. Layer-wise scaling prevents layers with small weights from receiving disproportionately large updates. LAMB enabled BERT training with batch sizes up to 64K while maintaining convergence quality.
Critical batch size and diminishing returns
A fundamental property of distributed training is that adding more workers stops helping past a threshold. The critical batch size \(B^{*}\) marks the transition point beyond which increasing batch size yields diminishing returns in convergence per sample seen.
Definition 1.6: Critical batch size
Critical Batch Size \((B^{*})\) is the distributed-training batch size at which the gradient noise scale is large enough that further batch growth yields diminishing returns in sample efficiency (McCandlish et al. 2018; Shallue et al. 2019).
- Significance: It marks the transition point for parallel scaling efficiency. Below \(B^{*}\), increasing the batch size linearly improves the convergence per step. Above \(B^{*}\), larger batches yield diminishing returns, requiring proportionally more samples \((D)\) to reach the same loss.
- Distinction: Unlike the memory-limited batch size (determined by memory capacity and activation footprint), the critical batch size is an algorithmic property of the model and dataset.
- Common pitfall: A frequent misconception is that training can be sped up indefinitely by adding GPUs. In reality, \(B^{*}\) defines the physical ceiling for data parallelism: adding workers beyond this point wastes energy and compute \((O)\) without reducing total training time \((T)\).
The gradient-noise-scale proxy for the critical batch size can be estimated as:
\[ B^{*} \approx \frac{\text{tr}(\Sigma)}{\|\nabla \mathcal{L}(\theta)\|^2} \]
where \(\text{tr}(\Sigma)\) is the trace of the gradient covariance matrix (total gradient variance) and \(\|\nabla \mathcal{L}(\theta)\|^2\) is the squared gradient norm (signal strength). Intuitively, \(B^{*}\) is the batch size at which averaging reduces gradient variance to the level of the true gradient magnitude.
Published large-batch regimes illustrate the scale rather than providing fixed thresholds. On ImageNet, Goyal et al. (2017) stabilized ResNet-50 at global batch 8,192 with warmup, and LARS extends ImageNet large-batch training further in some settings (You et al. 2017). For BERT-Large pretraining, LAMB reports training at 32K–64K batch sizes with layer-wise adaptive updates (You et al. 2020). Across other domains, McCandlish et al. (2018) show useful batch-size limits ranging from tens of thousands in ImageNet-like settings to millions in reinforcement-learning workloads, so new frontier-language-model thresholds should be measured rather than copied.
You, Yang, Igor Gitman, and Boris Ginsburg. 2017. “Large Batch Training of Convolutional Networks.” arXiv Preprint arXiv:1708.03888.
You, Y., J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C.-J. Hsieh. 2020. “Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes.” International Conference on Learning Representations (ICLR).
The scaling law regime exhibits three distinct behaviors. Below the critical batch size \((B < B^{*})\), linear scaling holds: doubling the batch size halves the iterations needed to reach the target loss, and hardware efficiency determines throughput. At the critical point \((B \approx B^{*})\), samples-per-second efficiency is maximized, the optimal trade-off. Above it \((B > B^{*})\), returns diminish, because doubling the batch size requires more than double the total samples, so additional workers add throughput but not sample efficiency.
Figure 8 illustrates this relationship between batch size and training efficiency.
The critical batch size has important implications for distributed training system design. Adding workers beyond \(B^{*}/b\) (where \(b\) is the per-worker batch size) improves throughput but not sample efficiency, though it may still be worthwhile for cost if the marginal cost of additional workers is low. The learning rate schedule matters too, because above \(B^{*}\) aggressive warmup becomes essential, since the loss landscape near initialization may not support the large updates that linear scaling would produce. Communication trade-offs shift as well, because above \(B^{*}\) the reduced benefit of larger batches makes communication overhead relatively more costly, strengthening the case for gradient compression or asynchronous methods.
Checkpoint 1.2: Scaling decisions
Given a 7B-parameter model distributed across a cluster of 64 A100 GPUs (80 GB HBM2e each), determine the maximum useful batch size. Use a pilot trace with \(\text{tr}(\Sigma)=1.6 \times 10^6\), \(\|\mu\|^2=100\), per-GPU batch \(b=4\), and a proposed gradient-accumulation depth of 32 steps, giving \(B_{\text{global}} = 64 \times 4 \times 32 = 8192\) samples.
Worked example: Convergence comparison for 8 vs. 64 workers
To illustrate these concepts concretely, consider scaling from 8 to 64 workers when training a transformer language model with baseline batch size \(B = 32\) per worker.
Napkin Math 1.6: Scaling from 8 to 64 workers
Setup: Transformer model with 1.3B parameters, target perplexity 15, baseline training: 100K iterations on single GPU with \(b =\) 32.
Table 5: Scaling SGD across cluster sizes: Iteration count, communication overhead, wall-clock speedup, and sample efficiency for the same training job at 1, 8, and 64 GPUs.
8 Workers (BSP)
- Effective batch size: \(B =\) 8 \(\times\) 32 = 256
- Learning rate: \(\eta =\) 8 \(\times \eta_{\text{base}}\) (linear scaling with warmup)
- Expected iterations: 100K \(/\) 8 = 12.5K iterations
- Convergence: Reaches target perplexity in 12.8K iterations (98 percent efficiency)
- Communication overhead: 15 percent (NVLink intra-node)
- Wall-clock speedup: 100K \(\times\) 1 \(/\) (12.8K \(\times\) 1.15×) = 6.8×
64 Workers (BSP)
- Effective batch size: \(B =\) 64 \(\times\) 32 = 2,048
- Learning rate (\(N =\) 64 workers): \(\eta = N \times \eta_{\text{base}}\) (if \(B < B^*\)) or \(\eta = \sqrt{N} \times \eta_{\text{base}}\) (if \(B > B^*\))
- Assuming \(B^* \approx\) 4,000 (below critical): Linear scaling applies
- Expected iterations: 100K \(/\) 64 = 1.56K iterations
- Convergence: Reaches target perplexity in 1.72K iterations (91 percent efficiency)
- Communication overhead: 45 percent (InfiniBand inter-node, 8 nodes)
- Wall-clock speedup: 100K \(\times\) 1 \(/\) (1.72K \(\times\) 1.45×) = 40.1×
SSP workers: 64 workers (\(s =\) 4)
- Same effective batch size: \(B =\) 2,048
- Learning rate: \(\eta' = \eta_{\text{BSP}} / \sqrt{1 + s}\) with \(s =\) 4, giving \(\eta' \approx\) 0.45 \(\times \eta_{\text{BSP}}\)
- Expected iterations: Higher due to staleness penalty
- Convergence: Reaches target perplexity in 2.1K iterations (74 percent efficiency)
- Communication overhead: 25 percent (reduced synchronization)
- Wall-clock speedup: 100K \(\times\) 1 \(/\) (2.1K \(\times\) 1.25×) = 38.1×
Analysis (table 5)
| Configuration | Iterations | Comm. Overhead | Wall-clock Speedup | Sample Efficiency |
|---|---|---|---|---|
| 1 GPU (baseline) | 100,000 | 0% | 1\(\times\) | 100% |
| 8 GPU BSP | 12,800 | 15% | 6.8× | 98% |
| 64 GPU BSP | 1,720 | 45% | 40.1× | 91% |
| 64 GPU SSP | 2,100 | 25% | 38.1× | 74% |
The 64-GPU BSP configuration achieves 40× speedup despite only 91 percent sample efficiency because the communication overhead (45 percent) is offset by the massive parallelism. SSP provides comparable wall-clock time with lower communication overhead but requires more total samples.
Cost Analysis (assuming $3/GPU-hour):
- 8: 12.8K iters \(\times\) 0.4 s/iter \(\times\) 8/3600 \(\times\) $3/GPU-hour = $34
- 64 BSP: 1.72K iters \(\times\) 0.58 s/iter \(\times\) 64/3600 \(\times\) $3/GPU-hour = $53
- 64 SSP: 2.1K iters \(\times\) 0.50 s/iter \(\times\) 64/3600 \(\times\) $3/GPU-hour = $56
Despite higher parallelism, 64-GPU training costs more per run due to communication overhead and reduced sample efficiency. The 8-GPU configuration is more cost-efficient but takes 6\(\times\) longer wall-clock time. The choice depends on whether minimizing cost or minimizing time-to-result is the priority.
Trade-off: Communication cost vs. convergence speed
The fundamental trade-off in distributed training is between communication efficiency and convergence quality. Figure 9 visualizes this trade-off space.
Several techniques occupy different positions on this trade-off curve. Gradient compression reduces communication volume by transmitting compressed gradient information through quantization, sparsification, or low-rank approximation. Examples include sign-based updates (Bernstein et al. 2018), Deep Gradient Compression (Lin et al. 2018), PowerSGD (Vogels et al. 2019), quantized SGD with convergence analysis (Alistarh et al. 2017), and empirical sparse gradient dropping for neural machine translation (Aji and Heafield 2017).
Bernstein, Jeremy, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. 2018. “signSGD: Compressed Optimisation for Non-Convex Problems.” International Conference on Machine Learning (ICML) 80: 560–69.
Lin, Yujun, Song Han, Huizi Mao, Yu Wang, and William J Dally. 2018. “Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training.” International Conference on Learning Representations (ICLR).
Vogels, Thijs, Sai Praneeth Karimireddy, and Martin Jaggi. 2019. “PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization.” Advances in Neural Information Processing Systems (NeurIPS) 32.
Alistarh, Dan, Demjan Grubic, Jerry Z. Li, Ryota Tomioka, and Milan Vojnovic. 2017. “QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding.” Advances in Neural Information Processing Systems (NeurIPS) 30: 1709–20.
Aji, Alham Fikri, and Kenneth Heafield. 2017. “Sparse Communication for Distributed Gradient Descent.” Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 440–45. https://doi.org/10.18653/v1/d17-1045.
Stich, Sebastian U. 2019. “Local SGD Converges Fast and Communicates Little.” International Conference on Learning Representations (ICLR).
Local SGD takes a different approach: workers perform \(H_{\text{local}}\) local updates before synchronizing, reducing communication frequency by factor \(H_{\text{local}}\). Convergence analysis shows that for smooth, strongly convex objectives, Local SGD achieves the same asymptotic rate as synchronous SGD with appropriately tuned learning rates (Stich 2019).
Decentralized SGD restricts workers to communicating only with neighbors in a communication graph rather than performing global AllReduce. This reduces bandwidth requirements at the cost of slower mixing, making it suitable for geo-distributed training where global synchronization is expensive.
The choice among these methods depends on the specific bottleneck. When network bandwidth limits throughput, gradient compression provides the best trade-off. When synchronization latency dominates, Local SGD or SSP are preferred. When network topology constraints exist, decentralized approaches may be necessary.
Self-Check: Question
A team scales a data-parallel training run from 8 to 64 workers, increasing the effective global batch size from 512 to 4,096 samples. If the model operates comfortably below its critical batch size, which learning-rate adjustment rule is the standard default?
- Keep the learning rate unchanged, because large-batch variance reduction automatically stabilizes optimization
- Multiply the learning rate by 64 to match the total worker count rather than the batch expansion ratio
- Multiply the learning rate by \(\sqrt{8}\) from step zero without a warmup schedule
- Multiply the learning rate by \(8\times\) (matching the batch expansion ratio) and include a gradual linear warmup schedule
Explain the distinction between hardware efficiency and statistical efficiency in distributed training, and describe a concrete scenario where a cluster configuration achieves high hardware efficiency but poor statistical efficiency.
Why does Stale-Synchronous Parallel (SSP) training typically require a smaller effective learning rate than Bulk-Synchronous Parallel (BSP) at the same global batch size?
- Bounded parameter staleness introduces an optimization error term that destabilizes parameter updates unless damped by a smaller learning rate
- SSP workers compute exact full-batch gradients instead of stochastic minibatches
- SSP eliminates all inter-GPU network communication, which alters the loss landscape
- SSP is mathematically restricted to second-order quasi-Newton optimizers that require small step sizes
The threshold batch size \(B^* \approx \frac{\text{tr}(\Sigma)}{\|\nabla \mathcal{L}(\theta)\|^2}\), representing the point where gradient variance equals true gradient magnitude and past which larger batches yield diminishing returns in sample efficiency, is known as the ____.
True or False: Scaling data parallelism significantly beyond a model’s critical batch size (\(B \gg B^*\)) continues to reduce total GPU-hours to reach target loss because additional GPUs always increase processed samples per second.
Model Parallelism
When model state, activations, optimizer state, or an individual tensor operation exceeds the memory or communication budget of one accelerator group, data parallelism entirely collapses. This is the memory capacity gap (principle 3) in operational form: model parameter growth outpaces device memory growth, forcing the model’s computation and state to be partitioned. Data-parallel memory optimizations extend replication’s reach, but eventually the model itself must be partitioned.
Definition 1.7: Model parallelism
Model Parallelism is a distributed training strategy that partitions a single neural network’s parameters or operations across multiple devices so each device computes a distinct portion of the model.
- Significance: If a model with parameter state \(S_{\text{model}}\) is split across \(N_{\text{mp}}\) devices, the ideal per-device parameter footprint falls toward \(S_{\text{model}}/N_{\text{mp}}\), enabling training of models that exceed single-device memory. The cost is extra communication for activations and gradients plus idle time from sequential dependencies, so the memory gain must exceed the transfer and pipeline-bubble overhead introduced by the partition.
- Distinction: Unlike data parallelism, which replicates the full model on every worker and shards the minibatch, model parallelism shards the model itself. Pipeline parallelism and tensor parallelism are implementation forms of model parallelism: one partitions layers into stages, while the other partitions individual tensor operations.
- Common pitfall: A frequent misconception is that model parallelism automatically speeds up training. Its primary benefit is capacity, not throughput; without careful scheduling and overlap, downstream devices wait for upstream activations and utilization can fall below a single-device baseline.
Optimizer state, not weights, dominates the per-replica training budget.
Even with ZeRO-3 fully deployed, sharding optimizer states, gradients, and parameters across workers, some architectures remain intractable. Tensor parallelism, illustrated in figure 10, addresses this by partitioning individual weight matrices across devices; pipeline parallelism partitions layers across stages and is introduced later in section 1.4.2.2. For a 175B-parameter model, weights alone occupy 350 GB, or about 5.5 GB per GPU across 64 GPUs. Full mixed-precision Adam training state is much larger: 2,800 GB globally, or roughly 43.8 GB per GPU before activations. This distinction matters because optimizer sharding reduces static state, but it does not eliminate the activation and per-layer capacity constraints that force model parallelism.
For long-context transformers where activation memory dominates, a 2048-token sequence through 175B parameters generates on the order of a terabyte of intermediate activations at even a small micro-batch (roughly 1.1 TB at micro-batch 4, as computed earlier), and no amount of optimizer sharding addresses this constraint. Model parallelism addresses these limitations by splitting the model architecture itself across devices, rather than replicating it with sharded state.
Napkin Math 1.7: The memory wall of scale
Problem: Training a 175B-parameter model (like GPT-3) on NVIDIA A100s (80 GB). Can data parallelism with ZeRO-3 handle this?
Math:
- Parameter storage: 175B params \(\times\) 2 bytes (FP16) = 350 GB.
- Gradients and optimizer state: Gradients add 175B params \(\times\) 2 bytes = 350 GB, while Adam FP32 master weights, momentum, and variance add 175B params \(\times\) 12 bytes = 2,100 GB.
- Total static memory: 2,800 GB.
- ZeRO-3 sharding: With 64 GPUs, per-GPU static memory = 2,800 GB/64 GPUs \(\approx\) 43.8 GB.
- Activation memory: For sequence length 2048 and batch size 1, the canonical full-checkpointing strategy stores \(\approx\) 4.8 GB of activations per GPU.
Systems insight: 43.8 GB (static) + 4.8 GB (dynamic) = 48.6 GB. Under this simplified accounting, ZeRO-3 plus full activation checkpointing fits within the 80 GB A100 capacity, leaving material headroom for framework buffers and transient layer gathers. If those additional allocations exceed the remaining HBM, the design can reduce the micro-batch, offload state, increase the sharding degree, or add tensor or pipeline parallelism; tensor parallelism is not the only recourse.
Model parallelism addresses this limitation by distributing the model architecture itself across devices, but the placement decision is where to cut the model. Layer-based splitting assigns devices to sequential layer groups, such as layers 1–4 on one device and layers 5–8 on the next. Channel-based splitting divides the channels within a layer, such as 512 channels on one device and the remaining channels on another. Transformer architectures add attention-head splitting, where separate devices own different heads. Each cut changes a different cost: layer cuts save memory but serialize the pipeline, while channel and head cuts expose more parallel work but require faster intra-layer communication.
These cuts enable the large-scale cases that exceed a single device for different reasons. GPT-3, with 175B parameters, relies on model parallelism for training. Vision transformers processing high-resolution \(16k{\times}16k\) pixel images use model parallelism to manage activation and memory constraints. Mixture-of-experts architectures use this approach to distribute their conditional computation paths across hardware (Shazeer et al. 2017; Lepikhin et al. 2021; Fedus et al. 2022).
Device coordination proceeds in three phases. In the forward pass, data flows sequentially through model segments on different devices. The backward pass propagates gradients in reverse order through these segments. During parameter updates, each device modifies only its assigned portion of the model. The coordination ensures mathematical equivalence to training on a single device while enabling models that exceed individual device memory capacities. The cost is sequential dependency between stages: while one device computes its forward pass, downstream devices sit idle, creating the pipeline bubble that dominates utilization analysis for model-parallel systems.
Checkpoint 1.3: Model parallelism foundations
Verify your understanding of model sharding:
Model parallelism implementation
Once the model itself is the object being partitioned, the implementation problem is placement: every cut must save enough memory to justify the activation transfer and idle time it introduces. Figure 11 captures this bidirectional data flow: input data propagates forward through sequentially assigned model partitions while gradients flow backward to update parameters, with intermediate results transferring across device boundaries at each stage.
Consider our running example: the 175B-parameter model requires 350 GB of memory in FP16, exceeding the 80 GB capacity of a single A100 by a factor of four. Model parallelism addresses this Capacity Wall by partitioning model weights across multiple devices, effectively stitching them into a single super-accelerator. Unlike data parallelism, where every GPU holds a full replica of the model and processes a unique fraction of the global batch, model parallelism requires each GPU to hold a unique fraction of the model and process the same data stream sequentially. With 8-way partitioning on A100s, BF16 weights alone occupy approximately 44 GB per GPU. That is only the weights budget: full training still needs activations, gradients, and optimizer state, so pure 8-way model parallelism is not sufficient by itself. Training requires additional sharding or offload for optimizer and gradient state, plus pipeline or hybrid parallelism to keep activation memory within capacity.
In a typical pipeline parallel implementation, the training loop operates as a relay race. The forward pass initiates on GPU 1, which computes the initial transformer blocks and transmits the resulting intermediate activation tensor across the interconnect to GPU 2. For our 175B model with a hidden dimension of 12,288 and a micro-batch size of 4 sequences at 2,048 tokens each, this handoff involves moving approximately 200 MB of data per stage boundary per step. GPU 2 must wait for this payload before it can begin its computation, creating a strict dependency chain that propagates through all stages. The backward pass mirrors this path in reverse, propagating error gradients from the final layer back to the input, with each device computing gradients only for its local parameters.
The architecture fundamentally changes the optimization dynamics compared to data parallelism. Instead of a global AllReduce to average gradients across replicas, each GPU performs a local optimizer step (Adam (Kingma and Ba 2015), AdaFactor, or similar) on its specific slice of parameters. A device holding transformer layers 1–12 updates only those layers’ weights and biases, with no cross-device synchronization required during the optimization step. While this eliminates the bandwidth-heavy gradient synchronization of data parallelism, it trades one bottleneck for another: pipeline bubbles. If the layers assigned to GPU 1 are computationally heavier than those on GPU 2 (common when attention layers have different head counts or when embedding layers are unevenly sized), valuable compute cycles are lost to waiting. The primary engineering challenge thus shifts from maximizing arithmetic intensity to minimizing serialization latency and ensuring balanced load across the partitioned fleet (Rasley et al. 2020).
Kingma, Diederik P., and Jimmy Ba. 2015. “Adam: A Method for Stochastic Optimization.” 3rd International Conference on Learning Representations (ICLR).
Rasley, Jeff, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. “DeepSpeed: System Optimizations Enable Training Deep Learning Models with over 100 Billion Parameters.” Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 3505–6. https://doi.org/10.1145/3394486.3406703.
Parallelism variations
Partitioning strategy determines which cost the model pays after memory no longer fits on one device. Layer-wise partitioning spends idle time to reduce per-device memory; pipeline parallelism spends scheduling complexity to hide that idle time; tensor parallelism spends NVLink bandwidth to split a single layer. The following subsections separate these choices by the architecture and interconnect that make each viable.
Layer-wise partitioning
Layer-wise partitioning is the least invasive model cut: it saves memory by assigning consecutive layers to separate devices, but it preserves the model’s sequential dependence. In transformer architectures, this translates to specific devices managing defined sets of attention and feed-forward blocks. Figure 12 demonstrates this partitioning for a 16-layer transformer: four consecutive blocks reside on each of four devices, with forward activations flowing left-to-right and backward gradients propagating right-to-left across the device boundaries.
Sequential processing introduces device idle time, as each device must wait for the previous device to complete its computation before beginning work. While device 1 processes the initial blocks, devices 2, 3, and 4 remain inactive. Similarly, when device 2 begins its computation, device 1 sits idle. This pattern of waiting and idle time reduces hardware utilization efficiency compared to other parallelization strategies.
Pipeline parallelism
Pipeline parallelism extends layer-wise partitioning by introducing microbatching to minimize device idle time. Instead of waiting for an entire batch to sequentially pass through all devices, the computation is divided into smaller segments called microbatches, with overlapping execution across pipeline stages (figure 13). In this schedule, each row represents a device processing model layers across microbatches simultaneously, with forward activations flowing downstream and backward gradients propagating upstream to keep all devices active.
Definition 1.8: Pipeline parallelism
Pipeline Parallelism is a model parallelism technique that partitions a neural network’s layers into sequential stages assigned to different devices, passing activations forward and gradients backward between stages while overlapping computation across stages using micro-batches to maintain throughput.
- Significance: Inter-stage communication transmits only the activation tensor at each stage boundary, sized as \(B_{\mu} \times S \times d_{\text{model}} \times 2\) bytes at BF16, where \(B_{\mu}\) is the micro-batch size. For a hidden dimension of 8,192 with micro-batch size 1 and a 2,048-token sequence, this is approximately \(8{,}192 \times 2{,}048 \times 2 \approx 32\) MB per boundary, compared to the gigabytes required for gradient AllReduce in data parallelism. This low communication volume makes pipeline parallelism the primary technique for scaling model depth across nodes connected by 50 GB/s InfiniBand. The pipeline bubble wastes approximately \((p-1)/(m+p-1)\) of total compute, where \(p\) is the number of stages and \(m\) is the number of micro-batches—with \(p=8\) stages and \(m=32\) micro-batches, bubble overhead is about 18 percent.
- Distinction: Unlike tensor parallelism, which shards individual matrix multiplications within a single layer and requires AllReduce on every layer’s output (demanding NVLink-class bandwidth within each stage), pipeline parallelism shards at layer boundaries and requires only point-to-point activation transfers between stages—tolerating InfiniBand bandwidth between nodes.
- Common pitfall: A frequent misconception is that adding pipeline stages always improves throughput. Each additional stage increases the pipeline bubble fraction \((p-1)/(m+p-1)\) unless the number of microbatches \(m\) scales proportionally, causing hardware to idle and erasing the benefit of deeper partitioning.
More stages require enough microbatches or the bubble dominates utilization.
The same bubble term becomes concrete in the following worked example, where an 8-stage pipeline and 32 microbatches still leave a measurable idle-time tax.
Napkin Math 1.8: The cost of the pipeline bubble
Problem: A large-model training run uses pipeline parallelism across 8 stages. To hide the sequential delay, the batch is split into 32 microbatches. What is the “bubble tax”—the fraction of GPU cycles lost to idle waiting?
Math: In a synchronous pipeline (1F1B), the bubble fraction is determined by the ratio of stages to microbatches.
- Wait time: At the start and end of each batch, GPUs sit idle for \(p-1\) steps.
- Productive time: GPUs compute for \(m\) steps.
- Bubble fraction: \((p-1) / (p-1+m) =\) 7 \(/\) (7 \(+\) 32) \(\approx\) 17.9 percent.
Systems insight: Pipeline parallelism is a utilization-memory trade-off. Reducing the bubble from 18 percent to 5 percent requires more microbatches \((m)\), which consume additional activation memory on each GPU. In the machine learning fleet, depth does not scale for free; the cluster pays a 17.9 percent capacity tax just to keep the stages coordinated. This is why techniques like interleaved pipelining are essential: they chop the bubble into smaller pieces to recover that lost 18 percent of fleet capacity.
GPipe6 (Huang et al. 2019) introduced synchronous pipeline parallelism with micro-batch accumulation, while PipeDream (Narayanan et al. 2019) developed asynchronous approaches with weight stashing. Modern systems employ 1F1B (One-Forward-One-Backward)7 scheduling to reclaim activation memory earlier; that lower memory pressure makes larger microbatch counts practical, and the larger \(m\) is what shrinks the bubble term.
6 GPipe: Published by Google in 2019, GPipe introduced synchronous micro-batch pipelining that trained a 557M-parameter AmoebaNet across 8 Tensor Processing Units (TPUs) with near-linear scaling. The key trade-off GPipe exposed: the pipeline bubble fraction \((p-1)/(m+p-1)\) means that with \(p=4\) stages and \(m=4\) micro-batches, 43 percent of compute is wasted in idle time – driving systems design toward schedules that reduce activation residency and make larger micro-batch counts feasible.
Huang, Y., Y. Cheng, A. Bapna, O. Firat, D. Chen, M. X. Chen, H. Lee, et al. 2019. “GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism.” Advances in Neural Information Processing Systems (NeurIPS) 32: 103–12.
Narayanan, Deepak, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. 2019. “PipeDream: Generalized Pipeline Parallelism for DNN Training.” Proceedings of the 27th ACM Symposium on Operating Systems Principles, 1–15. https://doi.org/10.1145/3341301.3359646.
7 1F1B (One-Forward-One-Backward) Scheduling: Unlike GPipe (which processes all forward passes before any backward passes), 1F1B interleaves them. This reduces the peak activation memory footprint from \(\mathcal{O}(m \times p)\) to \(\mathcal{O}(p)\), where \(m\) is the number of micro-batches and \(p\) is the number of pipeline stages. This memory reclamation is what enables the massive micro-batch counts \((m \gg p)\) required to keep the pipeline bubble small.
At the schedule level, 1F1B is a warmup, steady-state, and drain loop. Algorithm 1 makes the contract explicit: each stage alternates forward work for newer microbatches with backward work for older microbatches once gradients arrive, so activations can be released before the whole batch has completed.
\begin{algorithm} \caption{One-forward-one-backward pipeline schedule} \begin{algorithmic} \Require $p$ ordered pipeline stages; $m$ microbatches; activation links $j\to j{+}1$ and gradient links $j{+}1\to j$ \Ensure one optimizer update after gradients accumulate across all $m$ microbatches \State split the global batch into $m$ microbatches; assign consecutive layers to the $p$ stages \State warm-up: launch forwards from stage 0, forwarding each activation downstream as produced \For{each stage in steady state} \State alternate one backward (oldest ready microbatch) with one forward (next microbatch) \State release a microbatch's activation as soon as that stage's backward consumes it \EndFor \State drain: finish the remaining backward passes in reverse stage order \State apply the optimizer update once every stage has accumulated gradients for all $m$ microbatches \end{algorithmic} \end{algorithm}
The warm-up and drain phases still leave a bubble of roughly \((p-1)/(m+p-1)\), but interleaving backward work in the steady state frees each activation as soon as its backward consumes it, so peak activation memory grows with the stage count rather than the microbatch count. That is the gain over GPipe-style all-forward-then-all-backward scheduling, and it lets the system raise \(m\) until activation memory, recomputation, or the global-batch constraint becomes the next limit. In a transformer model distributed across four devices, device 1 would process blocks 1-6 for microbatch \(i+1\) while device 2 computes blocks 7-12 for microbatch \(i\). Simultaneously, device 3 executes blocks 13-18 for microbatch \(i-1\), and device 4 processes blocks 19-24 for microbatch \(i-2\). Each device maintains its assigned transformer blocks but operates on a different microbatch, creating a continuous flow of computation.
The transfer of hidden states between devices occurs continuously rather than in distinct phases. When device 1 completes processing a microbatch, it immediately transfers the output tensor of shape \(B_{\mu} \times S \times d_{\text{model}}\) to device 2 and begins processing the next microbatch. This overlapping computation pattern maintains full hardware utilization while preserving the model’s mathematical properties.
One important variant targets the remaining bubble directly. GPUs at the beginning of the pipeline are idle while waiting for gradients to flow back from the end, and vice versa. These bubbles represent wasted compute. The 1F1B schedule in algorithm 1 keeps streaming multiprocessors busy during the steady state and reduces activation residency, but the fill and drain bubble remains unless the system increases \(m\) or schedules useful work into those otherwise idle slots.
Zero-Bubble Pipeline Schedules further reduce idle time by overlapping weight gradient computation with activation gradient communication. In a standard backward pass, the GPU computes \(\partial \mathcal{L} / \partial W\) (weight gradient) and \(\partial \mathcal{L} / \partial X\) (activation gradient, sent to the previous stage) together. Zero-bubble scheduling splits these into separate kernels: a \(B\) kernel that computes only the activation gradient \(\partial \mathcal{L} / \partial X\) and sends it to the previous stage, and a \(W\) kernel that computes the weight gradient \(\partial \mathcal{L} / \partial W\) locally. The \(B\) kernel must execute promptly (it is on the critical path), but the \(W\) kernel can be scheduled opportunistically to fill bubbles.
The scheduling freedom provided by this B/W split is substantial. In a 4-stage pipeline with 8 microbatches, the standard 1F1B schedule has a bubble fraction of approximately \((p-1)/(m+p-1)\) where \(p\) is the number of stages and \(m\) is the number of microbatches. For \(p=4, m=8\), this is \(3/11 \approx 27\%\) idle time. Zero-bubble scheduling can reduce this to near zero by filling the startup and teardown bubbles with W computations.
The trade-off is memory: zero-bubble scheduling requires storing intermediate activations for longer (because the W computation is deferred), increasing peak memory usage. Some implementations address this by combining zero-bubble scheduling with activation checkpointing, selectively recomputing certain activations rather than storing them. The interaction between these techniques creates a three-way trade-off among pipeline bubble size, memory consumption, and recomputation overhead, an example of the displacement of overhead (principle 13).
Tensor parallelism
Pipeline parallelism, examined in section 1.4.2.2, addresses device idle time by overlapping microbatch processing across stages. Each device holds complete layers and processes them sequentially, with communication only at stage boundaries when activations transfer between devices. This approach tolerates moderate interconnect bandwidth because communication occurs infrequently, once per layer boundary per microbatch. However, pipeline parallelism cannot help when individual layers themselves exceed device memory, or when the communication pattern within layers benefits from a different granularity than layer boundaries.
Tensor parallelism takes a fundamentally different approach: instead of assigning complete layers to devices, it splits the weight matrices within each layer. This operator-level parallelism (also called intra-layer parallelism) enables finer-grained distribution but requires high-bandwidth interconnects for the frequent intra-layer communication it introduces.
Definition 1.9: Tensor parallelism
Tensor Parallelism is a model parallelism technique that partitions individual tensor operations—primarily matrix multiplications—across multiple devices using column-parallel or row-parallel weight splits, typically requiring two AllReduce operations per transformer layer in Megatron-style transformer blocks to sum partial results from all participating devices.
- Significance: Megatron-LM style tensor parallelism places two AllReduce operations per transformer block—one after attention and one after the MLP—with each collective reducing an activation tensor of size \(B_{\text{batch}} \times S \times d_{\text{model}} \times 2\) bytes in BF16. A ring AllReduce moves roughly \(2(t-1)/t\) times that payload per GPU. At degree \(t=8\) on NVLink with 900 GB/s bandwidth, each AllReduce takes approximately 0.1–0.5 ms. The same operation over InfiniBand (50 GB/s) takes 2–10 ms per layer—with 96 transformer blocks, this adds 200–960 ms of pure synchronization overhead per step, collapsing MFU to single digits.
- Distinction: Unlike pipeline parallelism, which communicates only at layer boundaries and transfers small activation slices, tensor parallelism synchronizes within every layer via AllReduce, making it sensitive to the per-operation latency of the interconnect and restricting it to within-node NVLink fabrics in practice.
- Common pitfall: A frequent misconception is that tensor parallelism can simply be scaled across nodes over InfiniBand. NVLink delivers approximately 900 GB/s bidirectional, or 450 GB/s per direction; InfiniBand NDR delivers 50 GB/s per port, a 9× per-direction gap. Running tensor parallelism across InfiniBand typically collapses MFU to single-digit percentages, making the communication overhead larger than the compute benefit of distributing the matrix multiplication.
The interconnect topology dictates which form of model parallelism is viable at each level of the cluster hierarchy. Tensor parallelism’s per-layer synchronization demands NVLink-class bandwidth; pipeline parallelism’s boundary-only communication tolerates InfiniBand. This bandwidth pattern creates the design pressure for a hybrid: use tensor parallelism for bandwidth-intensive intra-layer splits and pipeline parallelism for coarser inter-layer splits.
Tensor parallelism needs NVLink bandwidth; pipeline parallelism tolerates slower fabric.
8 Megatron-LM: NVIDIA’s 2019 framework that trained an 8.3B-parameter transformer – 24\(\times\) BERT and 5.6\(\times\) GPT-2 at the time – by strategically placing only two AllReduce operations per transformer block (one after attention, one after the multilayer perceptron (MLP)). This column-then-row partitioning eliminates inter-GPU communication between the two linear layers within each block, achieving 76 percent scaling efficiency across 512 GPUs and establishing tensor parallelism patterns that remain influential in large-scale transformer training.
Shoeybi, Mohammad, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.” arXiv Preprint arXiv:1909.08053.
Megatron-style tensor parallelism8 (Shoeybi et al. 2019) partitions matrix multiplications in two ways. Examine figure 14: column-parallel splitting divides weight matrices along columns for QKV projections, allowing independent computation across GPUs, while row-parallel splitting divides along rows for output layers, requiring AllReduce to combine partial sums at the end of each block.
Column-parallel linear layers split weights along columns. For input \(X\) and weight matrix \(W = [W_1 | W_2]\) split across 2 GPUs: \[\mathbf{Y} = \mathbf{X}\mathbf{W} = \mathbf{X}[\mathbf{W}_1 | \mathbf{W}_2] = [\mathbf{X}\mathbf{W}_1 | \mathbf{X}\mathbf{W}_2]\] Each GPU computes its partition independently, and the outputs are concatenated without communication when the next operation is row-parallel. The row-parallel half of the pattern splits weights along rows, \(W = \begin{bmatrix} W_1 \\ W_2 \end{bmatrix}\): \[\mathbf{Y} = \mathbf{X}\mathbf{W} = \mathbf{X}_1 \mathbf{W}_1 + \mathbf{X}_2 \mathbf{W}_2\] Each GPU computes a partial sum, and the system pays one AllReduce to combine the partial outputs.
The column-then-row arrangement shown in figure 14 is the key design insight. Pairing a column-parallel layer with a row-parallel layer allows the intermediate activations to flow directly between them without communication, confining synchronization to the end of the pair. Megatron applies that pattern twice in each transformer block. The QKV projection is column-parallel and its attention output projection is row-parallel, then the first feed-forward layer is column-parallel and the second feed-forward layer is row-parallel. The design places AllReduce operations strategically, one after attention and one after the feed-forward network, for two AllReduce operations per transformer layer. Communication volume per transformer layer depends on sequence length \(S\), hidden dimension \(d_{\text{model}}\), tensor-parallel degree \(t\), and batch size \(B\). The activation payload is: \[M_{\text{act}} = B \times S \times d_{\text{model}} \times \text{sizeof(dtype)}\]
A ring AllReduce transfers approximately \(2(t-1)/t \times M_{\text{act}}\) bytes per GPU, which is close to \(2M_{\text{act}}\) for large \(t\). With \(S=2048\), \(d_{\text{model}}=4096\), \(B=4\), and FP16, the activation payload is \(4 \times 2048 \times 4096 \times 2 \approx 67\) MB. Applying the ring factor gives roughly 134 MB per AllReduce. For a 96-layer model with two AllReduce operations per layer, this totals approximately 25.7 GB per forward pass; including the symmetric backward AllReduces brings per-training-step communication to roughly 51 GB, requiring NVLink bandwidth to avoid becoming the bottleneck.
Tensor parallelism scaling degrades rapidly beyond 8-way parallelism because the same split that reduces memory also reduces the computation available to hide communication. As the tensor-parallel degree grows, per-GPU work falls while collective latency and aggregate traffic rise; per-GPU ring AllReduce bytes approach 2\(\times\) the activation tensor size rather than growing linearly without bound, but the smaller local matrix multiply leaves less useful work to cover that exchange. Once NVLink bandwidth saturates, adding another tensor-parallel shard mainly exposes more synchronization rather than more throughput. Published systems such as Llama 3 405B use TP=8 within NVLink-connected H100 nodes (Dubey et al. 2024), and similar node-local tensor parallelism is a common design pattern for large LLM training (Jiang et al. 2024).
Jiang, Ziheng, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, et al. 2024. “MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs.” 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), 745–60.
Liu, Hao, Matei Zaharia, and Pieter Abbeel. 2023. “Ring Attention with Blockwise Transformers for Near-Infinite Context.” arXiv Preprint arXiv:2310.01889, ahead of print. https://doi.org/10.48550/arXiv.2310.01889.
The same partitioning idea can apply to sequence length as well as matrix dimensions. While standard tensor parallelism tiles computation across the HBM-NVLink boundary within a node, Ring Attention partitions the attention state across GPUs rather than forcing every device to hold the full prefix (Liu et al. 2023). This becomes relevant for sequences that exceed a single GPU’s memory, such as million-token context windows. Each GPU owns a block of queries \(Q\) and circulates the key/value (\(K\)/\(V\)) blocks around a ring, computing attention against the block currently resident in memory and then forwarding that block to its neighbor.
The algorithm proceeds in \(N - 1\) communication rounds (where \(N\) is the number of GPUs). In each round, each GPU computes attention between its local \(Q\) block and the currently resident \(K\)/\(V\) block, sends that \(K\)/\(V\) block to its ring neighbor, and receives the next block from its other neighbor. The implementation overlaps this exchange with computation: while GPU \(i\) computes attention using \(K_j\)/\(V_j\), it simultaneously receives \(K_{j+1}\)/\(V_{j+1}\) from the ring.
If the compute time for one tile exceeds the communication time for transferring one tile over NVLink, the communication is fully hidden. On an H100 with 3.35 TB/s HBM bandwidth and 900 GB/s NVLink bandwidth, this overlap is achievable for typical tile sizes.
The practical impact of Ring Attention is measured in context length. Without it, a single GPU’s attention computation is limited by HBM capacity: the key/value state for a sequence of length \(S\) must fit entirely in one GPU’s memory. With Ring Attention across \(N\) GPUs, each GPU holds \(S/N\) tokens of that state, enabling context lengths of \(N \times S_{\text{single}}\). Performance Engineering later connects this sequence-level distribution to FlashAttention’s tile-level HBM reuse; here, the distributed-training lesson is that sequence length can become a partitioning dimension just like layers, tensors, or data.
Parameter servers and embedding sharding
While AllReduce dominates dense model training, the Parameter Server (PS) architecture, formalized by Li et al. (2014), remains common for recommendation systems and other sparse workloads. A parameter server architecture separates workers (who compute gradients) from servers (who store parameters and apply updates).
For dense models (like ResNet or BERT), the PS architecture creates a bottleneck: with \(N\) workers, the server’s inbound bandwidth must absorb \(N\) gradient streams simultaneously, saturating the server’s network bandwidth and making it the communication chokepoint. This “incast” problem drove the adoption of Ring AllReduce, where each worker sends and receives at its own link rate, achieving \(N\)-fold higher aggregate bandwidth by distributing the load across all nodes. For dense models beyond 4–8 GPUs, decentralized AllReduce wins decisively.
However, for Recommendation Systems (specifically DLRM) the model parameters are dominated by massive embedding tables (often 10 TB+) that cannot fit on any single GPU. Furthermore, the updates are sparse: a batch of users interacts with only a tiny fraction (e.g., 0.001 percent) of the items.
In this sparse regime, the parameter server avoids dense synchronization by moving only the rows that a batch touches:
- Embedding Sharding: The massive tables are partitioned across the PS fleet (often CPU nodes with massive DRAM).
- Sparse Lookups: Workers send a list of IDs to the PS.
- Sparse Pull: The PS returns only the requested embedding vectors, not the full table.
- Sparse Push: Workers send gradients only for the touched embedding rows.
The Sparse Pull/Sparse Push pattern avoids the bandwidth bottleneck of dense AllReduce. Implementations such as TorchRec or Meta’s hierarchical sharding place “hot” embeddings on GPUs and “cold” embeddings on CPU PS nodes, creating a tiered memory hierarchy for model parameters.
Expert parallelism (mixture of experts)
While tensor parallelism splits dense layers across devices, expert parallelism enables scaling model capacity (parameters) without increasing compute cost (FLOPs) by using conditional computation. In a mixture-of-experts (MoE) architecture (Shazeer et al. 2017), the feed-forward network of each transformer block is replaced by a set of \(E\) “experts” (independent feed-forward networks). For each token, a gating network selects a small subset (typically top-1 or top-2) of experts to process it.
Shazeer, Noam, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.” International Conference on Learning Representations (ICLR).
In a distributed setting, experts are partitioned across workers. If we have 8 GPUs and 8 experts, each GPU hosts one expert. The training process introduces a distinct communication pattern, as figure 15 illustrates:
- Gating: Each token determines its destination expert.
- All-to-All Dispatch: Tokens are routed across the network to the device hosting their selected expert.
- Computation: Experts process their assigned tokens.
- All-to-All Combine: Processed tokens are routed back to their original device to resume the sequence.
The primary advantage is decoupling model size from compute budget. A trillion-parameter MoE model might use only 10B parameters per token, enabling training on feasible hardware budgets. The constraint is the All-to-All communication, which is bandwidth-intensive and sensitive to load imbalance.
Definition 1.10: Expert parallelism
Expert Parallelism is the distribution strategy for mixture-of-experts models that places different experts on different devices and routes each token’s activations to the devices hosting its selected experts through All-to-All collectives, scaling total parameter count across the cluster without scaling the compute each token consumes.
- Significance: It decouples model capacity from compute budget: a trillion-parameter MoE model may activate on the order of 10B parameters per token, so per-token FLOPs remain near those of a 10B dense model while capacity grows roughly 100\(\times\). The price is communication structure: every MoE layer requires two All-to-All shuffles per pass (dispatch and combine) whose volume scales with tokens moved times hidden dimension, stressing the cluster’s bisection bandwidth rather than any single link.
- Distinction: Unlike tensor parallelism, which splits dense operations so that every device computes a fraction of every token, expert parallelism is conditional computation: each device computes only the tokens routed to its experts, making both the communication pattern and the load distribution data-dependent rather than fixed by the architecture.
- Common pitfall: A frequent misconception is that experts share work evenly. Natural token distributions are skewed: a hot expert can receive 3–5\(\times\) its fair share of traffic, overflowing its device’s activation buffer while sibling devices idle, which is why production MoE systems cap per-expert token budgets and penalize routing skew during training rather than trusting the router alone.
At the heart of expert parallelism lies the All-to-All communication primitive, which shuffles tokens across the cluster based on dynamic routing decisions. Consider a configuration with \(E=64\) experts distributed across 64 GPUs, processing a batch of \(B=4\) sequences at length \(S=2048\) with hidden dimension \(d_{\text{model}}=4096\). For every MoE layer, the system must dispatch \(B \times S\) tokens to their assigned experts. In FP16, this moves \(B \cdot S \cdot d_{\text{model}} \cdot 2\) bytes (approximately 67 MB) in a single direction. Since the processed embeddings must return to their original device for the residual connection, the total network overhead is roughly 134 MB per transformer block. While manageable in isolation, this latency accumulates rapidly in deep, sparse architectures like the Switch Transformer (Fedus et al. 2022) (up to 2,048 experts) or GShard (Lepikhin et al. 2021).
Fedus, William, Barret Zoph, and Noam Shazeer. 2022. “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.” Journal of Machine Learning Research 23 (120): 1–39.
Lepikhin, Dmitry, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding.” International Conference on Learning Representations (ICLR).
Network efficiency relies on the assumption of uniform token distribution, but natural language is inherently skewed: specific experts handling common syntax or connector words may receive 3–5\(\times\) their fair share of traffic. A hot expert is both a memory problem and a scheduling problem. If one expert receives too many tokens, its GPU runs out of activation buffer while other experts sit underused.
MoE systems therefore enforce a hard limit defined by the Capacity Factor \(C\), typically set between 1.25 and 1.5. This parameter caps the maximum number of tokens an expert processes at roughly \(C \cdot (B \times S)/E\) for the routing group. If the routing gate assigns more tokens than this buffer allows, the excess tokens are dropped, passing through the layer unprocessed via the residual connection. To reduce that data loss, training objectives include an Auxiliary Load Balancing Loss, weighted at 0.01–0.1 relative to the main cross-entropy loss, that penalizes the router for favoring specific experts. Models such as Mixtral 8x7B use top-2 routing across 8 experts, achieving a favorable balance between capacity scaling and routing stability.
The sparse communication pattern distinguishes recommendation and MoE workloads (Archetype B (DLRM at Scale)) from dense LLM training (Archetype A (GPT-4/Llama-3)) (Three systems archetypes).
Lighthouse 1.1: Archetype B (DLRM at Scale): DLRM vs. LLM scaling
Each scales through a different mechanism, bound by a different resource:
- LLMs (Dense): Scale via Tensor/Pipeline Parallelism. Constraint: compute and interconnect bandwidth (NVLink).
- DLRM-style models (Sparse): Scale via Embedding Sharding (Parameter Servers). Constraint: memory capacity and interconnect latency (Random Access).
The distinction dictates fundamentally different cluster designs: dense GPU pods for LLMs vs. memory-rich CPU/GPU hybrids for RecSys.
Trade-offs: The bubble vs. bandwidth dilemma
Model parallelism breaks the memory wall but introduces sequential dependencies that reduce hardware utilization. The engineering challenge is balancing pipeline bubbles (idle time) against all-to-all bandwidth (communication time).
Model parallelism offers three principal advantages. Memory scaling enables training of models that exceed single-device capacity: with 8-way tensor parallelism, the FP16 weight slice of a 175B model fits within A100 HBM before optimizer state and activation budgets are counted. Splitting the model also allows larger global batch sizes without out-of-memory errors, since each GPU processes a smaller parameter slice. The approach maps naturally to the physical structure of transformers, where attention heads split via tensor parallelism and layers split via pipeline parallelism.
The advantages come at the cost of three fundamental limitations. Pipeline bubbles cause GPUs to sit idle while filling and draining the pipeline; the bubble fraction is approximately \((p-1)/(m+p-1)\), where \(p\) is pipeline stages and \(m\) is microbatches, and achieving more than 90 percent efficiency requires \(m \gg p\), which increases activation memory. Communication intensity in tensor parallelism is equally constraining: two AllReduce operations execute per layer on the critical path, demanding extremely high-bandwidth, low-latency interconnects (NVLink) and typically preventing scaling beyond a single node (8 GPUs) before hitting the bandwidth wall. Implementation complexity rounds out the trade-off, requiring invasive changes to the model definition (replacing standard linear layers with column-parallel and row-parallel variants) unlike data parallelism, which wraps the model externally without modifying internals.
Self-Check: Question
In Megatron-LM style tensor parallelism for a transformer layer, how are the linear projections in the multi-head attention block arranged to minimize cross-device collective communication?
- Both QKV and Output projections are column-parallel, requiring an AllGather after each projection
- Both QKV and Output projections are row-parallel, requiring an AllReduce before each projection
- The QKV projection is column-parallel and the Output projection is row-parallel, requiring only a single AllReduce at the end of the attention block
- The QKV projection is partitioned across pipeline stages while the Output projection is replicated across data-parallel ranks
In a synchronous 1F1B pipeline-parallel training schedule with \(p=8\) stages and \(m=32\) microbatches, calculate the theoretical pipeline bubble fraction and explain how increasing the microbatch count \(m\) affects bubble overhead and activation memory.
Order the four sequential phases of token routing and execution in Expert Parallelism (Mixture of Experts) across distributed workers:
- All-to-All Combine: Processed token embeddings are routed back to their original source devices for residual connections
- Computation: Sharded expert feed-forward networks process their assigned token batches
- Gating: A router network evaluates tokens and selects top-k destination experts
- All-to-All Dispatch: Tokens are shuffled across the network fabric to the devices hosting their selected experts
In a Mixture of Experts (MoE) distributed training system, what occurs when a hot expert receives more tokens than its allocated buffer capacity determined by the Capacity Factor \(C\)?
- Excess tokens are dropped and pass through the MoE layer unprocessed via the residual connection
- The expert GPU dynamically allocates host CPU memory over PCIe, halting computation on other ranks
- The All-to-All collective automatically pauses until all sibling experts process matching token counts
- The extra tokens are rerouted to a random cold expert regardless of gating network weights
Contrast the communication patterns and hardware interconnect requirements of dense LLM training (Megatron-style Tensor Parallelism) with large-scale Recommendation Systems (DLRM embedding sharding).
True or False: Ring Attention scales context length beyond single-GPU memory capacity by circulating key/value blocks in an accelerator ring across \(N-1\) communication rounds while overlapping transfer with block-level attention computation.
FP8 for Distributed Training
Numerical precision is itself a distribution lever: halving the bytes per value halves the gradient and activation payloads that the AllReduce and tensor-parallel collectives move, shrinking the \(T_{\text{comm}}(N)\) term that dominates step time at scale. The 8-bit floating point (FP8) format exploits this lever by reducing the numerical precision of computation while trying to preserve enough range for stable optimization.
FP8 matters to distributed training when its narrower formats reduce communication without destabilizing optimization. Traditional mixed-precision training uses FP32 master weights with FP16 forward and backward passes. FP8-capable accelerators such as the NVIDIA H100 add support for FP8, offering two formats optimized for different phases of training (Micikevicius et al. 2022).
Micikevicius, Paulius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, et al. 2022. “FP8 Formats for Deep Learning.” arXiv Preprint arXiv:2209.05433.
The choice between the two FP8 formats is therefore not an instruction-set detail; it decides which tensors can be narrowed without breaking the optimizer. E4M3 (4-bit exponent, 3-bit mantissa) provides a range of approximately \(\pm 448\) with moderate precision, making it suitable for weights and activations in the forward pass where values cluster in predictable distributions. E5M2 (5-bit exponent, 2-bit mantissa) provides a much larger range of approximately \(\pm 57344\) but coarser precision, which is why it is used for gradients that can span many orders of magnitude during backpropagation. Using E4M3 for gradients would cause frequent overflow and underflow, while E5M2 captures the full gradient distribution at the cost of slightly noisier updates. Read table 6 as a distribution constraint map: each row changes the balance among range, precision, memory traffic, and trainability.
| Format | Exponent | Mantissa | Range | Precision | Use Case |
|---|---|---|---|---|---|
| FP32 | 8 bits | 23 bits | \(\pm 3.4 \times 10^{38}\) | Very high | Master weights |
| FP16 | 5 bits | 10 bits | \(\pm 65504\) | High | Mixed-precision |
| BF16 | 8 bits | 7 bits | \(\pm 3.4 \times 10^{38}\) | Moderate | Training |
| E4M3 | 4 bits | 3 bits | \(\pm 448\) | Low | FP8 forward pass |
| E5M2 | 5 bits | 2 bits | \(\pm 57344\) | Very low | FP8 gradients |
The critical engineering challenge in FP8 training across these precision formats (table 6) is dynamic scaling. FP8’s narrow dynamic range means that a fixed scale factor will cause either overflow or underflow. Per-tensor scaling multiplies each tensor by a scale factor before casting to FP8, then divides by that factor after the FP8 computation. The scale factor is adjusted dynamically, typically by tracking the running maximum absolute value of each tensor and choosing a scale that maps this maximum to near the FP8 maximum representable value.
The three-precision approach (FP32 master weights, FP8 general matrix multiply (GEMM) operations, FP16 accumulation) can achieve near-FP16 training quality while improving effective throughput on FP8-capable hardware for suitable workloads. The systems implication is concrete but hardware-dependent: FP8 halves the bytes moved relative to FP16/BF16 tensors, while operation throughput and energy gains depend on the accelerator implementation. The throughput gain is therefore a performance expression of narrower movement and hardware support, not an additional multiplicative factor.
For distributed training, FP8 earns its place only when smaller payloads reduce communication while dynamic scaling protects convergence. It changes the payload size, not the parallelization axis. If precision reduction is still insufficient, the system must decide how data, tensor, and pipeline parallelism map onto the hardware hierarchy.
Self-Check: Question
Why does standard FP8 mixed-precision training pair the E4M3 format for forward-pass activations and weights with the E5M2 format for backward-pass gradients?
- E4M3 has a wider numerical range than E5M2, which prevents gradient underflow during backpropagation
- E4M3 provides higher precision (3 mantissa bits, range \(\pm 448\)) suitable for bounded activations, while E5M2 provides a wider dynamic range (5 exponent bits, range \(\pm 57344\)) necessary for gradients spanning multiple orders of magnitude
- E4M3 is an integer format while E5M2 is a floating-point format supported exclusively on CPU hosts
- E5M2 requires half the memory bandwidth of E4M3, accelerating the backward pass AllReduce
Explain why per-tensor dynamic scaling is necessary in FP8 distributed training and describe how the scaling factor is computed during execution.
True or False: Converting training tensors to FP8 eliminates the need for tensor and pipeline parallelism when training 100B+ parameter models because FP8 changes the geometric parallelization axes.
Hybrid Parallelism
Training a large model when data parallelism runs out of memory and model parallelism runs out of network bandwidth requires orchestrating both strategies simultaneously across three dimensions. The analysis in section 1.2 and section 1.4 revealed a fundamental tension: data parallelism scales throughput but demands massive memory, while model parallelism enables large models but starves the compute.
Hybrid Parallelism resolves this tension by applying both strategies orthogonally: model parallelism splits the architecture to fit available memory, while data parallelism scales throughput across multiple model replicas. Training a 175B parameter language model on a dataset of 300 billion tokens demonstrates this approach in practice. The neural network layers distribute across multiple GPUs through model parallelism, while data parallelism enables different GPU groups to process separate batches. This dual strategy addresses both memory constraints from model size and computational demands from dataset scale simultaneously, and it is precisely this combination that defines Archetype A training at large-model scale.
Lighthouse 1.2: Archetype A (GPT-4/Llama-3): Physics of 3D parallelism
Archetype A (GPT-4/Llama-3) is the primary driver for hybrid parallelism. Because the model parameters \((P)\) exceed the memory of any single accelerator \((C_{\text{mem,device}})\), and the training dataset \((D)\) requires massive throughput, we must split the problem along three orthogonal axes:
- Tensor parallelism: Splits individual layers to fit \(P\) within a node’s memory.
- Pipeline parallelism: Splits layers across nodes when the parameter footprint exceeds a single node.
- Data parallelism: Replicates the entire split-model pipeline to scale throughput on \(D\).
Only by combining all three can we train Archetype A systems efficiently.
To see why the combination matters, consider three parallelism configurations for training the 175B model on a 1,024-GPU cluster organized as 128 nodes of 8 GPUs each. The progression from pure data parallelism through tensor and pipeline parallelism makes the efficiency gain concrete.
Configuration A: Pure Data Parallelism (DP-1024) is a communication counterfactual rather than a feasible placement for this model. The FP16 weights alone occupy 350 GB, so an A100 or H100 cannot replicate them; sharding only optimizer states does not change that fact. If replication were possible, all 1,024 ranks would form one data-parallel group, and each rank would participate in an AllReduce of the 350 GB gradient tensor across the InfiniBand fabric. The 37.5 percent calculation in section 1.3 isolates that communication cost.
Configuration B: TP-8, DP-128. Within each of 128 model replicas, 8 GPUs use tensor parallelism over NVLink. The corresponding tensor shard from every replica forms one 128-rank data-parallel group, so the layout contains 8 such groups. Each rank AllReduces its 1/8 gradient shard, 43.75 GB rather than 350 GB, and TP communication remains within the fast NVLink domain. The inter-node AllReduce time drops from 14 seconds to approximately 1.75 seconds. If the compute time is still 2.1 seconds, the efficiency improves to \(2.1 / (2.1 + 1.75) \approx 54.5\) percent, and with communication-computation overlap, practical efficiency reaches 75–85 percent.
Configuration C: TP-8, PP-4, DP-32. Each of 32 model replicas uses 8-way tensor parallelism within a stage and 4 pipeline stages across nodes. For each of the \(8\times4=32\) tensor-and-pipeline shard positions, the corresponding ranks from all replicas form one 32-rank data-parallel group. Each rank therefore AllReduces a \(350/(8\times4)\approx10.94\) GB gradient shard. Pipeline communication forwards activations between stages and has lower volume than a full-model AllReduce. The trade-off is the pipeline bubble: at the beginning and end of each microbatch, some pipeline stages are idle while waiting for activations from earlier stages or gradients from later stages. The bubble fraction is approximately \((p-1)/(m+p-1)\), so with 4 pipeline stages and 32 microbatches per training step, the bubble wastes roughly 9 percent of compute (\(3/35 \approx 8.6\%\)).
The three-way comparison shows that configurations that keep high-bandwidth communication (tensor parallelism) within the fast NVLink domain and push only low-bandwidth communication (data-parallel AllReduce of smaller gradient shards) onto the slower InfiniBand fabric achieve the highest efficiency. The infrastructure hierarchy dictates the parallelism hierarchy. The resulting principle is called hierarchy-aware parallelism, and it is a common approach for large-scale training systems.
Definition 1.11: Hierarchy-aware parallelism
Hierarchy-Aware Parallelism is the strategy of mapping different parallel execution modes to the physical bandwidth tiers of the cluster.
- Significance: It ensures that high-frequency synchronization (for example, tensor parallelism) stays on the fastest links (NVLink), while lower-frequency tasks (for example, data parallelism) use slower tiers (InfiniBand). This alignment maximizes scaling efficiency \((\eta_{\text{scaling}})\) by reducing exposed communication time \((T_{\text{comm}}(N))\) and latency \((L_{\text{lat}})\).
- Distinction: Unlike uniform parallelism, which treats all node-to-node links as equal, hierarchy-aware strategies respect the bandwidth cliffs between die, node, and rack boundaries.
- Common pitfall: A frequent misconception is that any model can be sharded across any number of nodes. In reality, if the hierarchy mapping is wrong (for example, sharding a large tensor across a slow inter-rack link), the communication time will dwarf the compute time, making the scale-out useless.
The interaction also flows in the reverse direction: the choice of parallelism strategy influences infrastructure design. A training system that uses TP-8, PP-4, DP-32 generates a communication pattern where the most bandwidth-intensive traffic (tensor-parallel AllReduce) is confined to within each node, the moderate-bandwidth traffic (pipeline stage communication) flows between groups of 4 neighboring nodes, and the lowest-bandwidth traffic (data-parallel AllReduce of reduced gradient shards) flows across the full cluster.
The layered communication pattern favors a hierarchical network topology where nearby nodes have higher bandwidth between them (a “locality-aware” topology) over a flat topology where all node pairs have equal bandwidth (a uniform fat-tree). Rail-optimized and hierarchical fat-tree designs exploit this locality, placing the nodes that communicate most frequently on the same switch or in the same rack, minimizing the number of switch hops for the most bandwidth-intensive traffic.
The practical implication for infrastructure procurement is that the network topology must be co-designed with the parallelism strategy, not selected independently. An organization that purchases a flat fat-tree fabric (optimized for any-to-any communication) but trains exclusively with hierarchy-aware parallelism (where most traffic is local) has over-provisioned the network’s global bandwidth while potentially under-provisioning local bandwidth. Conversely, an organization that purchases a rail-optimized fabric (optimized for local communication) but later needs to run mixture-of-experts models with all-to-all communication (which requires global bandwidth) will find the fabric inadequate. The network fabric, which represents 10–15 percent of total system cost, must be matched to the anticipated workload mix, and changing the fabric after deployment is prohibitively expensive and disruptive.
The 3D training loop
Training a 175B-parameter model requires 3D Parallelism: the coordinated composition of data, pipeline, and tensor parallelism across thousands of devices.9 This approach does not merely sum the benefits of individual parallelism strategies; it composes them geometrically to match the physical topology of the hardware.
9 3D Parallelism: Named after the three orthogonal axes of decomposition: (1) Data Parallelism (batch), (2) Pipeline Parallelism (depth), and (3) Tensor Parallelism (layer width). Organizations visualize their training fleets as a 3D grid \((d, p, t)\), where the product \(N_{\text{total}} = d \times p \times t\) equals the total GPU count. This geometric perspective is essential for balancing the tiered bandwidth constraints of high-bandwidth clusters.
Consider a training fleet configured with tensor parallelism (TP) of 8 GPUs, pipeline parallelism (PP) of 16, and data parallelism (DP) of 128. This configuration uses 16,384 GPUs (8 GPUs \(\times\) 16 \(\times\) 128) organized into a hierarchy of bandwidth domains.
The training step begins at the data-parallel level. Each of the 128 model replicas receives a distinct slice of the global batch. Within each replica, the model is split across 16 pipeline stages (nodes), with micro-batches flowing sequentially from the embedding layer on Node 0 to the loss calculation on Node 15. At the finest granularity, within each node, the 8 GPUs fuse into a single “super-accelerator” via TP. Every matrix multiplication in the forward pass is fractured across these devices, which must exchange partial results via high-bandwidth NVLink after every operation. For a 175B-scale hidden dimension, the activation payload is roughly 201.3 MB; the resulting tensor-parallel traffic is about 4.2 GB per pipeline stage per microbatch in the forward pass, or 8.5 GB including the backward pass. Across the full 96-layer replica, the forward tensor-parallel traffic is about 67.6 GB. This traffic is the highest-intensity in the system, but its exposed latency can remain small relative to compute because the chips communicate over 600 GB/s–900 GB/s local bandwidth.
The backward pass inverts this flow and exposes the critical dependencies between parallelism dimensions. As gradients flow backward through the pipeline, nodes exchange activation gradients point-to-point. This traffic is relatively light (roughly 201.3 MB per stage boundary) allowing it to traverse slower inter-node InfiniBand links without stalling the pipeline. The true bottleneck emerges at the end of the step: data-parallel synchronization. The full model’s gradient state is still about 350 GB, but in a 3D-parallel layout each data-parallel group synchronizes the corresponding parameter shards for its tensor- and pipeline-parallel replica rather than every GPU moving the full tensor. Gradient bucketing groups ready layer-gradient tensors into larger messages as backward computation proceeds; when the bucket schedule and cross-sectional bandwidth line up, synchronization runs under useful computation instead of extending the critical path.
The architectural imperative is bandwidth matching: the communication volume of each algorithm must map inversely to the latency of the hardware interconnects. Chatty, blocking TP communication stays within the NVLink domain (600 GB/s or higher). Serialized, point-to-point PP transfers traverse the cluster spine at InfiniBand speeds. The massive but infrequent DP synchronization amortizes across the full training step. Attempting to run TP across racks, or DP without gradient accumulation, can violate this hierarchy and leave the 16,000-GPU fleet waiting for data to traverse the wire. This bandwidth-matching principle completes the feasibility check introduced in section 1.0.5.
Hybrid-parallelism worked example
Applying this bandwidth-matching principle to physical infrastructure transforms cluster design into a placement problem: each parallelism dimension must sit on the network tier that can tolerate its traffic. Tensor parallelism is the most latency-sensitive because it launches frequent AllReduce operations inside transformer blocks, so it belongs on the intra-node NVLink domain (600 GB/s–900 GB/s). Pipeline parallelism moves activation tensors only at stage boundaries, so it can span nearby nodes over InfiniBand (50 GB/s–100 GB/s). Data parallelism produces the largest payload, the gradient synchronization, but it occurs once per step and can be overlapped with backward computation. Memory capacity per device (80 GB–80 GB) sets the hard limit for every placement.
For a DGX A100-style deployment, this reasoning leads to a concrete layout. Fix tensor parallelism at \(t=8\) so the bandwidth-heavy matrix-multiplication collectives stay within one 8-GPU node. Map pipeline parallelism across nodes in the same high-bandwidth rack or island, often \(p=8\) or \(p=16\) depending on the memory footprint. Use data parallelism for the remaining scale-out dimension across pods. The result is not a generic rule but a consequence of matching each traffic pattern to the cheapest fabric tier that can carry it.
With the placement fixed, the next question is whether the static model state fits inside each accelerator’s HBM budget.
The memory budget for training a 175B-parameter model is dominated by model states and requires aggressive sharding to fit within the 80 GB HBM capacity of H100-class accelerators. The FP16 weights alone consume approximately 350 GB (175 \(\times 10^9 \times 2\) bytes). If we relied solely on tensor parallelism with 8-way sharding, each GPU would hold a 43.75 GB slice of the weights. However, the optimizer state presents a larger hurdle. In the simplified optimizer-state convention used for this 3D-parallelism budget, Adam’s FP32 momentum and variance consume 8 bytes per parameter, adding about 1.4 TB globally; the fuller 12-byte accounting that also includes FP32 master weights would be about 2.1 TB. Even with 8-way tensor parallelism, the simplified combined weight and optimizer state would exceed 218.8 GB per GPU, causing an out-of-memory (OOM) error. Consequently, we must employ pipeline parallelism (\(p\)) to further partition the model layers. With a hybrid configuration of tensor parallelism 8 and pipeline parallelism 16, the simplified static memory footprint drops to 13.7 GB per GPU (often budgeted as roughly 15 GB after framework buffers), leaving the remaining HBM available for the dynamic activation memory \((A)\) generated during the forward pass, which scales linearly with micro-batch size and sequence length. Under the fuller 12-byte Adam convention, the same 8-way tensor, 16-stage pipeline split would be 19.1 GB per GPU.
Once the configuration fits in memory, the same layout has to satisfy the communication hierarchy that motivated it. Each parallelism dimension imposes a distinct traffic profile on the network. Tensor parallelism is the most chatty, requiring two AllReduce operations for every transformer block (one for the attention projection, one for the MLP) in both the forward and backward passes. These messages are relatively small but occur thousands of times per step, making them strictly latency-bound and necessitating NVLink. Pipeline parallelism, in contrast, involves point-to-point transfers of activation tensors (size \(B_{\mu} \times S \times d_{\text{model}}\)) only at the boundaries of the pipeline stages. While these messages are moderate in size, they occur less frequently, making them manageable over standard InfiniBand links. Data parallelism generates the largest burst of traffic, requiring a global AllReduce of the entire 350 GB gradient buffer. However, this communication occurs only once per global batch update. Using gradient bucketing to overlap this transmission with the compute-intensive backward pass hides the effective cost of DP communication, provided the cluster maintains sufficient cross-sectional bandwidth.
The last cost is not bandwidth but scheduling: a pipelined model is only efficient when enough micro-batches are in flight to keep all stages busy. This remaining cost is the pipeline bubble developed in section 1.4.2.2. Applying its bubble fraction \(\frac{p-1}{m + p - 1}\) to this hybrid layout, a GPT-175B configuration with \(p=16\) stages and \(m=32\) micro-batches wastes \(\frac{15}{47} \approx 31.9\%\) of theoretical compute capacity, nearly one-third. The 1F1B scheduling that reclaims activation memory, and the larger micro-batch counts it makes practical, drive that fraction down asymptotically.
Blackwell-class scaling example
The Blackwell architecture provides a concrete example of how accelerator packaging changes the 3D parallelism trade-offs. First, NVLink 5 provides 1.8 TB/s of bidirectional bandwidth per GPU, doubling the intra-node capacity of the Hopper generation. This allows for larger tensor parallelism \((t)\) groups (for example, \(t=\) 16 or \(t=\) 32 across multiple nodes) with lower latency overhead. Second, Blackwell adds FP4 Tensor Core support, improving low-precision throughput and memory efficiency for supported workloads; whether FP4 is used for training depends on the numerical recipe and software stack.
For a 1 trillion-parameter model, a Blackwell-class configuration can use \(t=\) 16 (spanning two 8-GPU nodes via NVLink Switch) and \(p=\) 8. Relative to the Hopper reference’s \(t=8\) and \(p=16\), this keeps the model-partition product at 128 while halving the number of pipeline stages and its associated bubble term. The 10 TB/s die-to-die interconnect within the Blackwell GPU further collapses the distinction between intra-chip and intra-package communication, allowing the two reticle-limited dies to function as a single high-bandwidth tensor parallel unit. The general systems lesson is that improving local bandwidth shifts pressure outward: once intra-node communication is less binding, rail-optimized topologies (Rail-optimized topology) and All-to-All optimization become more important for larger Machine Learning Fleet configurations.
The same placement constraints can be made operational through design-space search.
Example 1.2: Automated design space search (Tier 3 optimizer)
Scenario: An engineering team schedules a 175B-parameter model across 8,192 H100 GPUs (80 GB HBM each), navigating the multi-dimensional 3D-parallelism search space (TP \(\times\) PP \(\times\) DP).
Diagnosis: Manual trial-and-error risks memory overflow (exceeding 80 GB HBM) or saturating NVLink. The ParallelismOptimizer evaluates all factorizations, selecting TP=8 GPUs, PP=8, and DP=128 to maximize MFU without exceeding HBM capacity.
Systems lesson: Automated design-space search formalizes empirical heuristics: tensor parallelism should be confined within single-node NVLink bounds (TP=8 GPUs), while pipeline parallelism (PP=8) should be minimized to avoid pipeline bubble overhead while satisfying memory constraints, achieving 25.9 percent projected MFU.
Adjusting any one dimension shifts the pressure onto the other two, making the factorization TP \(\times\) PP \(\times\) DP a tightly coupled system rather than three independent knobs.
Checkpoint 1.4: Hybrid 3D parallelism
Verify your understanding of how parallelism strategies combine:
The MFU values need a historical utilization baseline to show how published systems approached 50 percent utilization. Figure 16 traces the evolution of MFU across published training systems from 2020 to 2024. The progression from GPT-3’s 21 percent MFU to PaLM’s 46 percent MFU reflects not improvements in raw hardware speed but advances in the parallelism strategies, communication overlap techniques, and scheduling optimizations discussed throughout this chapter. The plateau near 40–46 percent reveals that the theoretical ceiling imposed by communication overhead, pipeline bubbles, and memory management remains formidable even in highly optimized hybrid-parallel systems. Notably, Meta’s Llama 3 training at 16,384 H100 GPUs achieved slightly lower MFU (41 percent) than the same model at 8,192 GPUs (43 percent), confirming that the scaling tax described in section 1.3.2 is not merely theoretical but measurable in large published runs.
Chowdhery, Aakanksha, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, et al. 2022. “PaLM: Scaling Language Modeling with Pathways.” arXiv Preprint arXiv:2204.02311.
Dubey, Abhimanyu, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783.
Self-Check: Question
In a 3D parallelism layout (\(N_{\text{total}} = d \times p \times t\)), what hardware-matching principle dictates mapping tensor parallelism intra-node while pipeline parallelism spans inter-node links?
- Pipeline parallelism requires more memory bandwidth than NVLink can provide, forcing it onto InfiniBand
- Tensor parallelism requires only point-to-point transfers, making it insensitive to link latency
- Hierarchy-aware parallelism (bandwidth matching): TP launches high-frequency per-layer AllReduce collectives demanding intra-node NVLink bandwidth (>900 GB/s), whereas PP transmits boundary activations tolerating inter-node InfiniBand
- Data parallelism cannot execute across nodes that use tensor parallelism internally
A 175B-parameter model is trained using 3D hybrid parallelism with \(t=8\) (tensor parallel) and \(p=16\) (pipeline parallel). Using 350 GB for FP16 weights and 1,400 GB for 8-byte Adam optimizer states, calculate the static memory footprint per GPU and explain why pipeline parallelism is necessary in addition to tensor parallelism.
Order the three primary communication patterns in hybrid 3D parallelism from highest communication frequency (most frequent) to lowest communication frequency (least frequent):
- Data-parallel AllReduce across replica groups
- Tensor-parallel intra-node AllReduce within transformer blocks
- Pipeline-parallel point-to-point activation transfers across stage boundaries
In a 16,384-GPU cluster configured with \(\text{TP}=8\), \(\text{PP}=16\), and \(\text{DP}=128\), how many GPUs participate in each individual data-parallel AllReduce communicator group?
- \(16{,}384\text{ GPUs}\), because all accelerators in the cluster must synchronize in a single global collective
- \(128\text{ GPUs}\), because each rank communicates only with its corresponding tensor-and-pipeline shard position across the 128 model replicas
- \(8\text{ GPUs}\), matching the tensor-parallel degree within each local node
- \(16\text{ GPUs}\), matching the pipeline stage depth across the cluster spine
True or False: A uniform fat-tree network topology providing equal bisection bandwidth between all node pairs is always more cost-effective than a rail-optimized topology when training large language models with hierarchy-aware 3D parallelism.
Multi-Model Training: RLHF and Alignment
The parallelism strategies examined so far assume a single model being trained on a single objective. Reinforcement learning from human feedback (RLHF) and its variants break this assumption by requiring multiple models to coordinate within a single training loop, each with different memory footprints, compute profiles, and gradient requirements. At a high level, RLHF generates model outputs, scores them with preference-derived reward signals, regularizes updates against a reference model, and then updates the policy; proximal policy optimization (PPO) implements this online loop with separate policy, reference, reward, and value models, while direct preference optimization (DPO)-style methods remove much of that rollout and value-model machinery. This creates a heterogeneous fleet management problem that cannot be solved by any single parallelism strategy and represents a qualitatively different distributed systems challenge from standard pretraining.
The multi-model coordination problem
Standard pretraining involves one model, one loss function, and one gradient stream. RLHF alignment, by contrast, orchestrates a system of models that interact during every training step. In PPO (Schulman et al. 2017), as used in InstructGPT-style RLHF systems (Ouyang et al. 2022), four distinct models must operate in concert. Table 7 separates those models by role, execution mode, and memory burden.
Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. “Proximal Policy Optimization Algorithms.” arXiv Preprint arXiv:1707.06347, ahead of print. https://doi.org/10.48550/arXiv.1707.06347.
Ouyang, L., J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, et al. 2022. “Training Language Models to Follow Instructions with Human Feedback.” Advances in Neural Information Processing Systems (NeurIPS) 35: 27730–44. https://doi.org/10.52202/068431-2011.
| Model | Role in PPO-style RLHF | Execution mode | Memory burden |
|---|---|---|---|
| Policy Model | The model being aligned and optimized. | Training mode with gradients, optimizer moments, and activations. | For a 70B-parameter model in mixed precision with Adam, memory is approximately 70B \(\times\) (2 \(+\) 2 \(+\) 12) = 1,120 GB, the same budget as standard pretraining. |
| Reference Model | A frozen copy of the pretrained policy that supplies the KL-divergence penalty. | Inference mode only. | FP16/BF16 parameters without gradients or optimizer state; a 70B model needs about 70B \(\times\) 2 = 140 GB, roughly 8\(\times\) less than the training configuration. |
| Reward Model | A preference-trained scorer that produces scalar rewards for generated sequences. | Inference mode only. | Often smaller than the policy; a 13B reward model requires approximately 26 GB in FP16, but it must process every generated sequence. |
| Value Model | The PPO critic that estimates expected future reward. | Training mode, either separate or sharing a policy backbone. | A full-size separate critic can add another 1,120 GB; smaller models or shared-backbone designs reduce this to 200–400 GB. |
The aggregate memory demand of the four-model PPO system dwarfs standard pretraining. A naive co-location of a 70B policy, 70B reference, 13B reward, and 13B value model requires approximately 1,494 GB of accelerator memory, before accounting for the KV caches and intermediate activations generated during sequence generation. On H100 GPUs with 80 GB of HBM each, this system requires a minimum of 19 GPUs for parameter storage alone. Once generation-phase KV caches (which grow linearly with output sequence length) and training-phase activations are included, the practical minimum rises to 64–128 GPUs for a single RLHF training instance.
Infrastructure asymmetry: Training vs. inference models
The defining infrastructure challenge of RLHF is not the total memory footprint but the asymmetry between the models’ compute profiles. The policy and value models require full backward passes with gradient computation, activation checkpointing, and optimizer updates, compute-intensive operations that benefit from tensor parallelism and high arithmetic intensity. The reference and reward models, by contrast, perform only forward passes: they are inference workloads embedded within a training loop, with memory access patterns dominated by KV cache management rather than gradient accumulation.
The asymmetry creates a placement dilemma. Co-locating training and inference models on the same GPUs wastes compute during the generation phase (when the training models sit idle) and wastes memory during the gradient phase (when the inference models’ parameter storage could be reclaimed for activations). Separating them onto dedicated GPU pools eliminates waste but introduces network latency for reward queries, each generated token batch must traverse the interconnect to reach the reward model and return a scalar signal before the policy gradient can be computed.
The generation phase itself introduces a sequential bottleneck absent from standard pretraining. RLHF requires the policy model to generate complete sequences (typically 256–2,048 tokens) autoregressively before computing rewards and policy gradients. Autoregressive generation is memory bandwidth bound, not compute bound: each token requires a full forward pass through the model to produce a single output token. For a Llama-2-70B-style grouped-query attention policy, the KV cache grows by approximately \(2 \times N_L \times H_{\text{KV}} \times d_{\text{head}} \times 2\) bytes per generated token per sequence. With 80 layers, 8 KV heads, 128-dimensional heads, 1,024 generated tokens, and a batch of 256 prompts, the cache consumes 85.9 GB: \(2 \times 80 \times 8 \times 128 \times 1024 \times 256 \times 2\) bytes. A dense multi-head attention model using the full 8,192-wide hidden state for K and V would yield 687.2 GB. Even with grouped-query attention, the cache uses about one H100 GPU’s worth of HBM for intermediate attention state that is discarded after reward computation.
RLHF systems address this asymmetry through temporal multiplexing. During the generation phase, the cluster behaves like an inference system: the policy model generates sequences while the reference model computes log-probabilities, and the dominant resource is HBM reserved for token-by-token attention state. During the training phase, the same fleet switches back to training mode: gradients are computed through the policy and value models using the 3D parallelism configuration developed in section 1.6. Serving systems handle that phase with request batching and KV-cache placement; here, the important point is the mode switch. RLHF is not one steady training loop, but an alternation between inference-shaped work and training-shaped work, and standard training frameworks rarely manage that transition by themselves.
DPO: Simplifying the fleet
DPO (Rafailov et al. 2023) eliminates the reward model and value model entirely by reformulating the alignment objective as a classification loss over preference pairs. Instead of generating sequences, computing rewards, and estimating advantages, DPO directly optimizes the policy to assign higher log-probability to preferred responses over dispreferred ones, using the reference model only to compute a KL-divergence regularization term.
Rafailov, R., A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. 2023. “Direct Preference Optimization: Your Language Model Is Secretly a Reward Model.” Advances in Neural Information Processing Systems (NeurIPS) 36: 53728–41. https://doi.org/10.52202/075280-2338.
The infrastructure implications are substantial. DPO reduces the multi-model system from four models to two: the policy model (training mode) and the reference model (inference mode). For the 70B policy with 13B reward and value models quantified in section 1.7.4, static parameter memory drops from about 1,494 GB to 1,260 GB, a 16 percent reduction. If the PPO value model is policy-sized, the reduction is much larger: about 2,406 GB to 1,260 GB, or roughly 48 percent. DPO also eliminates the autoregressive generation phase entirely. Training operates on a fixed dataset of (prompt, preferred response, dispreferred response) triples, restoring the standard pretraining data pipeline: fixed-length sequences, deterministic batching, and no sequential token-by-token generation. The training loop becomes a standard supervised learning step with a modified loss function, amenable to the same 3D parallelism, gradient accumulation, and communication overlap techniques used for pretraining.
The trade-off is capability. DPO operates on a static preference dataset, meaning the policy cannot explore new responses and receive feedback during training. PPO’s online generation allows the policy to improve iteratively on its own outputs, potentially discovering better strategies that the static dataset does not contain. For deployment scenarios where the preference data comprehensively covers the target distribution, DPO’s infrastructure simplification dominates. For scenarios requiring adaptive exploration (training models to solve novel reasoning tasks, for example), PPO’s online feedback loop may justify the 2\(\times\) infrastructure overhead.
Quantitative analysis: PPO vs. DPO resource requirements
To make the infrastructure trade-off concrete, consider aligning a 70B policy model on a cluster of 256 H100 GPUs (80 GB HBM each).
Napkin Math 1.9: RLHF infrastructure budget: PPO vs. DPO
Problem: Aligning a 70B policy model requires co-locating it with a 70B reference model and, for PPO, a 13B reward model and a 13B value model. Under 3D parallelism (TP=8 GPUs, PP=4, DP=8), how much per-GPU memory does each approach consume, and how much headroom does DPO free by eliminating the reward model, value model, and generation-phase KV cache?
Table 8: PPO vs. DPO Memory and MFU: Per-GPU memory and effective training MFU for PPO and DPO on the same accelerator budget.
PPO memory budget (per-GPU, with TP=8 GPUs, PP=4, DP=8 for policy)
The policy model under 3D parallelism with TP=8 GPUs and PP=4 distributes its 1,120 GB training state across 32 GPUs, yielding 35 GB per GPU. The reference model, requiring only 140 GB for inference, can be sharded across a separate pool of 8 GPUs at 17.5 GB each, or co-located with the policy GPUs at an additional 4.4 GB per GPU (140 GB / 32 GPUs). The 13B reward model adds 26 GB shared across its pool. The 13B value model in training mode adds approximately 208 GB (13B \(\times\) 16 bytes/param) across its pool.
Total static memory (co-located policy + reference on 32 GPUs): 35 GB + 4.4 GB \(\approx\) 39.4 GB per GPU, leaving \(\sim\) 40.6 GB for activations and KV cache. With generation-phase KV caches for a rollout batch containing 256 sequences and 1,024 tokens inside one \(TP{\times}PP\) policy replica, each policy GPU must reserve an additional 85.9 GB/32 GPUs \(\approx\) 2.7 GB for its KV cache shard. The remaining \(\sim\) 37.9 GB constrains the micro-batch size during the training phase.
DPO Memory Budget (same parallelism configuration)
The policy model uses the same 35 GB per GPU. The reference model adds 4.4 GB per GPU (co-located). No reward model, no value model, no KV cache for generation.
On the policy GPUs, total static memory is 35 GB + 4.4 GB = 39.4 GB per GPU, identical to PPO’s co-located policy/reference footprint. Cluster-wide, however, DPO avoids the PPO reward and value model memory, and it also removes the generation-phase KV cache burden. The full \(\sim\) 40.6 GB remaining on each policy GPU is available for training activations, permitting modestly larger micro-batches (about 1.07× under this co-located policy/reference memory budget) or reducing the need for activation checkpointing that PPO requires.
Throughput Comparison
PPO’s two-phase design (generate then train) introduces a fundamental throughput penalty. If generation consumes 60 percent of the step time (typical for autoregressive decoding of long sequences), the training hardware achieves only 40 percent utilization during an RLHF step. DPO, operating as a standard training loop, achieves the same 40 percent–55 percent MFU as pretraining. Table 8 compares the two regimes side by side.
| Metric | PPO (70B policy) | DPO (70B policy) |
|---|---|---|
| Models required | 4 (policy, ref, reward, value) | 2 (policy, reference) |
| Total parameter memory | \(\sim\) 1,494 GB | \(\sim\) 1,260 GB |
| Minimum GPUs (memory) | 64–128 GPUs | 32–64 GPUs |
| Generation phase | Yes (sequential, BW-bound) | None |
| Effective training MFU | 15–25% | 40–55% |
| Data pipeline | Online generation | Static preference pairs |
Systems insight: In this 13B reward/value-model scenario, DPO reduces static parameter memory by 15.7 percent and doubles the effective compute utilization by eliminating the reward model, value model, and autoregressive generation phase. The choice between PPO and DPO is not only a modeling preference but an infrastructure constraint: organizations with limited GPU budgets may prefer DPO because it restores the workload to a fixed-data training loop. If the PPO value model is policy-sized rather than 13B, the memory reduction approaches the larger 2,406 GB to 1,260 GB comparison discussed in section 1.7.3.
Sequence length variance and batching challenges
Both PPO and DPO face a data engineering challenge absent from standard pretraining: extreme variance in sequence length. Pretraining typically uses fixed-length sequences (2,048 or 4,096 tokens, padded or packed to fill each position), enabling uniform batch shapes and predictable memory consumption. RLHF training data consists of variable-length prompts (10–500 tokens) concatenated with variable-length completions (50–2,048 tokens), producing sequence lengths with 10–50\(\times\) variance within a single batch.
Length variance creates two interacting problems. Fixed-size batching pads all sequences to the maximum length in the batch, wasting compute on padding tokens. A batch containing one 2,048-token sequence and fifteen 128-token sequences wastes 88 percent of the compute on padding. Dynamic batching groups sequences by similar length to minimize padding, but introduces load imbalance across data-parallel workers: one worker may receive a batch of long sequences consuming 60 GB of activation memory while another processes short sequences using only 8 GB, causing the short-sequence worker to stall at the synchronization barrier while the long-sequence worker completes.
RLHF systems mitigate this through a combination of sequence packing (concatenating multiple short sequences into a single fixed-length input with attention masking to prevent cross-contamination) and adaptive micro-batching (dynamically adjusting the number of sequences per micro-batch based on the aggregate token count rather than the sequence count). These techniques can recover much of the compute lost to padding variance but add complexity to the data pipeline that standard pretraining frameworks do not provide. The interaction between variable-length data and distributed synchronization remains an active area of systems engineering research, because every worker must process the same number of tokens per step to maintain gradient consistency.
RLHF is therefore not a detour from parallelism strategy; it is the case that shows why no strategy is sufficient by itself. The policy model may need tensor and pipeline parallelism, the reference and reward models behave like inference services, the value model may require training state, and the rollout data pipeline changes shape from step to step. With that stress test in view, the final comparison can return to the general engineering question: identify the binding constraint first, then choose the parallelism pattern that moves the least data while fitting the required state.
Self-Check: Question
Why does Proximal Policy Optimization (PPO) in multi-model RLHF create an infrastructure asymmetry challenge that does not exist in standard pretraining?
- It combines compute-intensive training models (Policy and Value models requiring backward passes and optimizer state) with memory-bandwidth-bound inference workloads (Reference and Reward models requiring autoregressive rollout and KV cache storage)
- All four models in PPO must be trained with identical optimizer hyperparameters on the same GPU
- PPO requires all models to be converted to INT4 precision to enable cross-node gradient synchronization
- PPO eliminates backward passes entirely, turning distributed training into a stateless inference service
Explain how Direct Preference Optimization (DPO) simplifies cluster infrastructure compared to PPO, detailing which models and computational phases are eliminated.
During the autoregressive generation phase of PPO alignment for a 70B-parameter model, what resource represents the primary memory consumer and system performance bottleneck?
- FP32 Adam momentum and variance buffers stored on CPU hosts
- Key-Value (KV) cache memory in GPU HBM, bounded by memory bandwidth during token-by-token generation
- AllReduce gradient communication buffers across InfiniBand switches
- Host filesystem NVMe write bandwidth for checkpoint dumping
The scheduling approach in RLHF distributed systems where the accelerator fleet alternates between an autoregressive rollout phase dominated by KV cache memory and a backward training phase dominated by matrix GEMM compute is known as ____.
True or False: Because Direct Preference Optimization (DPO) eliminates reward and value models and avoids online sequence generation, it is strictly superior to PPO for all model alignment and exploration tasks.
Parallelism Strategy Comparison
A parallelism strategy is useful only if it moves the binding constraint without creating a larger one. The feasibility filter previewed by the decision tree in section 1.0.5 named those constraints; now that tensor, pipeline, and hybrid parallelism have been developed, the same filter can be stated as engineering signals rather than a generic ranking. For a new 50-billion-parameter model, the decisive questions are which memory state fits, how often communication occurs, where the pipeline bubble appears, and which abstraction cost the engineering team can tolerate. Table 9 turns those questions into engineering signals.
| Strategy | Binding constraint | Primary movement | Placement decision | Validation signal |
|---|---|---|---|---|
| DDP | Model state fits on one accelerator | AllReduce gradients every step | Replicate full model across data-parallel workers | Step time dominated by compute, not synchronization |
| FSDP/ZeRO | Model state exceeds one accelerator | Shard and gather parameters or optimizer state | Place shards to minimize gather and reduce-scatter cost | Peak memory remains below device budget |
| Tensor parallelism | One layer or tensor operation exceeds local capacity | AllReduce or AllGather within layer blocks | Keep tightly coupled partitions on fast intra-node links | Layer latency stays below activation handoff cost |
| Pipeline Parallelism | Layer stack fits only when split by stage | Activations and gradients between stages | Balance stages so bubbles are smaller than useful work | Bubble fraction falls as microbatch count increases |
| Hybrid parallelism | More than one constraint binds at once | Multiple collectives on different axes | Align data, tensor, pipeline, and shard groups to topology | MFU improves without exceeding memory or network budget |
Figure 17 converts the same filter into a decision tree. It begins with memory fit because capacity failure is nonnegotiable, then asks whether data scale alone justifies replication. The tree is intentionally simplified; hardware heterogeneity, communication bandwidth, and workload imbalance enter as second-order checks after the binding constraint has been identified.
The table and flowchart are intentionally coarse, but their purpose is precise: identify the binding constraint before choosing the implementation stack.
Self-Check: Question
According to the chapter’s parallelism decision framework, which condition indicates that standard DistributedDataParallel (DDP) is the optimal default strategy over FSDP, TP, or PP?
- The model’s single layer weight matrices exceed the HBM capacity of an individual GPU
- The cluster is connected by slow 10 Gb/s commodity Ethernet across 1,024 nodes
- The entire model state (parameters, gradients, and optimizer state) fits comfortably within single-device memory with sufficient headroom for activations
- The model architecture contains sparse Mixture-of-Experts routing layers
An engineering team observes that an 8-GPU training job using pure pipeline parallelism achieves only 35 percent Model Flops Utilization (MFU), while interconnect links are largely idle. Diagnosing the issue with the bubble formula \(F_{\text{bubble}} = \frac{p-1}{m+p-1}\), explain the root cause and propose two concrete adjustments to recover throughput.
Order the following parallelism strategies by their structural granularity from coarsest (whole model / batch level) to finest (intra-operator level):
- Tensor Parallelism (Megatron column/row splits)
- Standard Data Parallelism (batch dimension splitting)
- Pipeline Parallelism (inter-layer stage splitting)
From Principles to Systems
Choosing DDP, FSDP/ZeRO-style sharding, tensor parallelism, or pipeline parallelism is a decision about which constraints the runtime owns and which constraints the engineer must manage directly. The parallelism strategies examined throughout this chapter (gradient averaging, AllReduce synchronization, tensor splitting, pipeline scheduling) translate into deployed training systems through this layered abstraction hierarchy.
At the data-parallel layer, the simplest distributed training abstraction wraps a model so that the framework automatically replicates it across available accelerators, splits each batch, and averages gradients after the backward pass. Historically, parameter-server systems organized this work with server nodes that store shared parameters while workers push and pull updates (Li et al. 2014). As section 1.4.3 established, that design concentrates gradient bandwidth at the server tier, so dense synchronous training favors decentralized collectives.
Li, M., D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su. 2014. “Scaling Distributed Machine Learning with the Parameter Server.” Proceedings of the 2014 International Conference on Big Data Science and Computing, 583–98. https://doi.org/10.1145/2640087.2644155.
Large-scale data parallelism eliminates this bottleneck by replacing the central server with decentralized AllReduce. Each worker participates symmetrically in the reduction: every device both sends and receives gradient chunks at its own link rate, distributing the bandwidth load across all nodes rather than concentrating it. The framework initializes a process group that maps workers to the physical topology, selects an appropriate collective algorithm (ring, tree, or hierarchical) based on the detected interconnect, and inserts gradient synchronization hooks into the backward pass automatically. Gradient bucketing further improves efficiency by grouping small tensors into larger messages before transmission, and computation-communication overlap allows the AllReduce for early layers to proceed while later layers are still computing gradients. These optimizations can reach high parallel efficiency at moderate scale, not because the API is simple, but because the underlying runtime makes topology-aware decisions that a manual implementation would otherwise need to reproduce.
Model and pipeline parallelism require a fundamentally different abstraction because the framework must manage cross-device tensor placement and the sequential data flow between partitions. The core principle is explicit device assignment: the engineer specifies which layers reside on which devices, and the framework handles the activation transfers between devices during the forward pass and the gradient flow in reverse during backpropagation. Explicit device assignment makes the sequential dependencies of model parallelism visible, since each downstream device must wait for its upstream neighbor to complete. The engineer must reason about pipeline bubbles and load balance at the architecture level rather than hiding them behind an opaque wrapper.
For large-scale model parallelism, the key architectural insight is that tensor splitting and pipeline scheduling require different levels of framework support. Tensor parallelism replaces standard linear layers with column-parallel and row-parallel variants that automatically insert AllReduce operations at the correct points in the transformer computation graph, as described in section 1.4.2.3. Pipeline parallelism adds microbatch scheduling logic that interleaves forward and backward passes across stages to minimize bubble overhead. Memory-efficient sharding integrates ZeRO-3 style parameter partitioning by wrapping model layers with automatic AllGather operations before each forward pass and ReduceScatter after each backward pass. The critical design decision is which abstraction levels to compose: pure data parallelism suffices when the model fits in memory, memory-efficient sharding extends data parallelism to memory-constrained regimes, and full tensor or pipeline parallelism becomes necessary when individual layers or the full model depth exceed single-device capacity.
Across these abstraction layers, distributed training ultimately reduces to a small set of collective communication primitives. Rather than exploring the network mechanics of how these primitives physically route data across the cluster, which is extensively covered in Collective Communication, framework abstractions focus on when and why these primitives are invoked.
The primitives compose into the communication patterns that define each parallelism strategy. Data parallelism uses one AllReduce per training step. Full-shard FSDP with resharding uses about \(3N_L\) collectives per step (two AllGathers and one ReduceScatter for each of \(N_L\) layers). Tensor parallelism uses 2 AllReduce operations per transformer block on the critical path. The choice of primitive, its message size, and its frequency relative to computation determine whether the system operates in the compute-bound or communication-bound regime. No framework abstraction changes the underlying physics: the efficiency of distributed training ultimately depends on physical interconnect bandwidth, memory capacity, and synchronization latency.
Self-Check: Question
In distributed training framework runtimes (such as PyTorch DDP), what mechanism combines multiple small layer gradient tensors into contiguous memory buffers before launching collective communication?
- Activation checkpointing
- ZeRO Stage 3 parameter gathering
- Temporal multiplexing
- Gradient bucketing
Contrast the collective communication primitive invocation frequency of standard DDP against full-shard FSDP with resharding for a model with \(N_L\) layers during a single training step.
Explain why framework optimizations such as CUDA graphs and asynchronous communication streams cannot eliminate the physical scaling penalty of distributed training as the cluster size \(N\) becomes very large.
Fallacies and Pitfalls
Many engineering teams who scale cluster capacity by \(4\times\) find that their training iterations take longer to complete, reflecting a fundamental misunderstanding of distributed training physics. The following misconceptions capture errors that waste compute resources and delay research.
Fallacy: Linear speedup is achievable with sufficient engineering effort.
Amdahl’s Law establishes hard limits: any sequential component bounds maximum speedup regardless of parallelism. In distributed training, gradient synchronization is inherently sequential since all gradients must be collected before any update proceeds. As section 1.3 demonstrates, the scaling efficiency equation \(\eta_{\text{scaling}} = 1/(1 + N(T_{\text{comm}}(N) + T_{\text{sync}}(N) - T_{\text{overlap}})/T_{\text{compute}})\) reveals how communication and synchronization overhead dominate as \(N\) increases. Even with perfect overlap and ideal algorithms, communication overhead grows with cluster size. For data parallelism, AllReduce time increases logarithmically with tree algorithms or linearly in the latency term with ring algorithms as GPU count grows. A 1,000-GPU cluster will never train 1,000\(\times\) faster than a single GPU; achieving 500\(\times\) speedup would be exceptional, and 100–200\(\times\) is more typical for communication-heavy workloads. Organizations that budget projects assuming linear scaling inevitably miss deadlines and overspend on compute.
Pitfall: Optimizing MFU without considering scaling efficiency.
A team achieves 50 percent MFU on a single node, scales to 1,024 GPUs, and expects 512 GPU-equivalents of useful work. Scaling efficiency at 1,024 GPUs can be near 50 percent for communication-heavy workloads, so actual useful throughput is roughly \(0.50 \times 0.50 = 0.25\) of peak, or 256 GPU-equivalents: half the expected value. MFU and scaling efficiency are independent multiplicative factors. Optimizing one without measuring the other produces capacity estimates that are off by 2\(\times\) or more. Capacity planning at fleet scale requires reporting both metrics together and tracking their product as “useful goodput.”
Fallacy: Hyperparameters tuned on small clusters transfer directly to large-scale training.
Engineers tune hyperparameters on 8-GPU workstations then deploy to 256-GPU clusters expecting identical behavior. At scale, convergence patterns change fundamentally. The most critical hyperparameter is learning rate: as section 1.2 explains, batch size increases proportionally with GPU count in data parallelism, requiring learning rate adjustments. The “linear scaling rule”10 (Goyal et al. 2017) suggests \(\eta_{\text{large}} = \eta_{\text{base}} \times (B_{\text{large}}/B_{\text{base}})\), but this relationship holds only within bounds. As models scale, they eventually encounter the critical batch size,11 where adding more data per step yields diminishing returns in convergence.
10 Linear Scaling Rule: Established by Goyal et al. (2017) at Facebook AI Research, who trained ResNet-50 on ImageNet in one hour across 256 GPUs with a batch size of 8,192 while matching small-batch accuracy. The rule – multiply learning rate by \(k\) when batch size increases by \(k\) – works only while the large-batch approximation remains valid. Above the workload’s measured critical-batch-size regime, gradient noise drops below the useful signal scale and additional parallelism yields diminishing convergence returns (McCandlish et al. 2018; Shallue et al. 2019).
Goyal, Priya, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.” arXiv Preprint arXiv:1706.02677 abs/1706.02677.
Shallue, Christopher J, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. 2019. “Measuring the Effects of Data Parallelism on Neural Network Training.” Journal of Machine Learning Research 20: 1–49.
11 Critical Batch Size: The gradient-noise-scale view introduced by McCandlish et al. (2018) treats the largest useful batch size as a measurable statistic that varies by domain, model, optimizer, and training phase. Scaling beyond that measured point improves hardware throughput but does not proportionally reduce the number of optimization steps needed to reach target quality, collapsing the scaling efficiency \((\eta_{\text{scaling}})\).
McCandlish, Sam, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. 2018. “An Empirical Model of Large-Batch Training.” arXiv Preprint arXiv:1812.06162, ahead of print. https://doi.org/10.48550/arXiv.1812.06162.
Beyond the critical batch size, this relationship breaks down in a model-, data-, and optimizer-dependent way. A team that takes a small-cluster learning rate, multiplies it mechanically with GPU count, and jumps directly to a much larger global batch may see lower final quality or slower sample efficiency even though hardware throughput improved. Warmup schedules, weight decay adjustment, layer-wise optimizers such as LARS/LAMB, and careful momentum tuning can recover accuracy in some regimes, but require systematic experimentation at target scale. Organizations that skip these scaling studies waste thousands of GPU-hours on suboptimal runs.
Pitfall: Adding GPUs to data-parallel jobs without modeling communication.
Engineers assume more GPUs always accelerate training. At scale, statistical efficiency limits overwhelm hardware gains. As section 1.2 establishes, data parallelism increases effective batch size proportionally with GPU count \((B_{\text{total}} = N \times B_{\text{local}})\), but gradient quality grows sublinearly beyond model-specific thresholds. A 100K-sample batch may provide only 2\(\times\) the gradient information of a 10K-sample batch, not 10\(\times\), because samples become redundant within the loss landscape. The critical batch size defines where marginal returns collapse: the examples in section 1.3.6 put ResNet-50 around 8K–16K and BERT-Large around 32K–65K, with exact thresholds depending on optimizer, schedule, and target quality. Beyond this threshold, doubling GPU count doubles cost but provides minimal convergence acceleration. In a representative cost model, a 1,024-GPU run that converges in 18 hours at $45,000 compute cost may be worse than a 512-GPU run that converges in 19 hours at $22,000, demonstrating how exceeding critical batch size wastes resources without meaningful time savings.
Fallacy: Memory capacity alone determines the parallelism strategy.
Engineers see that a 70B model exceeds 80 GB memory and immediately choose tensor parallelism or pipeline parallelism to split weights. In a deployed run, the effective strategy depends on the interaction between memory pressure, computation patterns, and communication topology. As section 1.8 explains, tensor parallelism splits each layer across devices with AllReduce synchronization per layer, achieving even memory distribution but placing communication on the critical path. Pipeline parallelism assigns complete layers to stages with point-to-point transfers between stages, reducing per-step communication but introducing pipeline bubble overhead that wastes 10–30 percent of cycles. For a 175B model on 64 A100 GPUs where tensor parallelism degree-8 enables training, pipeline parallelism with 8 stages achieves 23 percent higher throughput due to reduced all-to-all communication despite similar memory footprints. The decision requires profiling communication patterns and bubble overhead, not just checking if weights fit in memory.
Pitfall: Applying FSDP or ZeRO without measuring efficiency trade-offs.
Engineers adopt FSDP universally after reading that it “reduces memory and enables larger models”. In a deployed run, sharding introduces 10–25 percent communication overhead that only pays off when memory pressure justifies it. FSDP reduces memory footprint by sharding optimizer state, gradients, and optionally parameters across GPUs, but requires AllGather operations before each forward pass and ReduceScatter after backward pass. For a 7B model on A100-80 GB with batch size 4, standard DDP achieves 145 samples/second while FSDP achieves only 118 samples/second (19 percent slower) because the model fits comfortably without sharding and the added communication overhead provides no benefit. FSDP provides value when model plus optimizer state exceeds single-GPU memory, when enabling larger per-GPU batch sizes justifies the overhead, or when ZeRO-Offload to CPU memory extends capacity. A 65B model that cannot fit on 80 GB becomes trainable with FSDP ZeRO-3, accepting 15 percent throughput loss to enable training at all. Applying FSDP universally without measuring memory pressure wastes performance.
Fallacy: Parallelism overhead is roughly constant regardless of model size.
Engineers benchmark parallelism strategies on convenient small models then apply conclusions to large-scale training. At scale, the ratio between computation and communication time changes dramatically with model size, inverting strategic decisions. AllReduce communication time depends primarily on gradient tensor size and network bandwidth, growing roughly linearly with parameter count, while forward and backward pass computation time grows superlinearly due to larger matrix operations. For a 1B-parameter model where forward/backward pass takes 50 ms and AllReduce takes 25 ms, communication overhead consumes 33 percent of step time. For a 70B-parameter model where forward/backward takes 2400 ms and AllReduce takes 180 ms, communication overhead drops to 7 percent despite the gradient size being 70\(\times\) larger. Decisions made on small models (“pipeline parallelism’s 15 percent bubble overhead makes it always slower than data parallelism”) can invert at scale where data parallelism’s communication overhead reaches 25–40 percent. Reliable strategy selection requires either profiling at target scale or analytical models that account for how computation scales as \(\mathcal{O}(n^2)\) to \(\mathcal{O}(n^3)\) while communication scales as \(\mathcal{O}(n)\).
Pitfall: Treating scaling efficiency as a property of the hardware alone.
Vendor benchmarks publish “85 percent scaling efficiency at 1,024 GPUs” and procurement teams plan capacity as if that number is a cluster constant. Scaling efficiency is a joint property of four things: the workload’s communication-to-computation ratio, the parallelism strategy (data, pipeline, tensor, hybrid), the network topology (fat-tree vs. torus vs. dragonfly), and the software stack (NCCL version, kernel fusion, overlap quality). Reference numbers from large-language-model training benchmarks may overstate scaling for graph neural networks with irregular communication, and they may understate scaling for embedding-heavy recommendation models with mostly local computation. Capacity planning should use measured scaling on the target workload, not a vendor hero benchmark.
Fallacy: Gradient accumulation is free.
Engineers use gradient accumulation to simulate larger batch sizes, reducing synchronization frequency from every step to every \(K\) steps. The technique appears cost-free since it eliminates \((K-1)/K\) of synchronization events. In a real training loop, accumulation introduces update latency, memory-residency, and numerical precision risks. Standard implementations accumulate into one resident gradient buffer rather than allocating a separate full gradient tensor per microstep; for a 7B model, that FP16 gradient buffer is still 14 GB and must remain live throughout the accumulation window. If the implementation retains computation graphs, overlaps multiple outstanding microbatches, or increases local microbatch size, activation memory can grow further and exhaust HBM. Effective optimizer-step latency increases proportionally with accumulation steps, so accumulating 8 steps means optimizer updates occur 8\(\times\) less frequently, potentially slowing convergence despite higher throughput. Most critically, accumulated FP16 gradients risk overflow when summing hundreds of gradient tensors, particularly in early training when loss values are large. A team training a transformer model with 16-step gradient accumulation in FP16 experienced loss spikes and divergence at step 1200; switching to 4-step accumulation with more frequent synchronization resolved the instability despite higher communication costs. Gradient accumulation trades communication frequency for update latency, memory residency, and numerical stability.
Pitfall: Using fixed checkpoint intervals regardless of system characteristics.
Engineers checkpoint distributed training “every hour” or “every 1000 steps” based on intuition rather than analysis. Fault Tolerance derives the Young-Daly checkpoint law (principle 12); the training lesson here is that checkpoint cadence is not a constant. It depends on the mathematical relationship between checkpoint write cost and failure rate.
The formula sets the optimal checkpoint interval to \(\tau_{\text{opt}} = \sqrt{2 \cdot T_{\text{write}} \cdot \text{MTBF}_{\text{system}}}\), where \(T_{\text{write}}\) is checkpoint write time and \(\text{MTBF}_{\text{system}}\) is mean time between system failures. For a 1024-GPU cluster at the canonical 50,000-hour per-GPU MTBF, the cluster-level \(\text{MTBF}_{\text{system}}\) is 48.8 hours (\(\text{MTBF}_{\text{GPU}}/N\)). With a 5-minute checkpoint time, the optimal interval is approximately 171.2 minutes (~2.9 hours), with an unavoidable checkpoint-plus-rework tax of 5.8 percent. Checkpointing every 15 minutes “to be safe” raises the total tax to 33.6 percent, while checkpointing every 8 hours risks losing significant work on failure. For larger models where checkpoint time increases to 15 minutes due to model size and storage bandwidth, the optimal interval shifts again. The cost of guessing scales with the cluster: a 1024-GPU run loses approximately $13,638 per day to excessive checkpointing when it uses a 15-minute interval instead of the Young-Daly optimum.
The optimal checkpoint cadence balances write overhead against rework risk.
Self-Check: Question
A team achieves 50 percent Model Flops Utilization (MFU) on a single node, scales to 1,024 GPUs, and measures a scaling efficiency \((\eta_{\text{scaling}})\) of 50 percent on their communication-heavy workload. What is the cluster’s realized ‘useful goodput’ as a fraction of peak hardware compute?
- 25 percent of peak compute (delivering 256 GPU-equivalents of useful work)
- 50 percent of peak compute (delivering 512 GPU-equivalents of useful work)
- 100 percent of peak compute (delivering 1,024 GPU-equivalents of useful work)
- 12.5 percent of peak compute (delivering 128 GPU-equivalents of useful work)
Using the Young-Daly checkpoint formula \(\tau_{\text{opt}} = \sqrt{2 \cdot T_{\text{write}} \cdot \text{MTBF}_{\text{system}}}\), explain why arbitrarily choosing a fixed 15-minute checkpoint cadence on a 1,024-GPU cluster with a 48-hour system MTBF and 5-minute write time causes severe compute waste.
Why is gradient accumulation NOT a cost-free substitute for true physical scale-out even though it reduces gradient AllReduce communication frequency by a factor of \(K\)?
- It is mathematically incompatible with the Adam optimizer
- It requires keeping live gradient buffers resident in HBM, increases optimizer update latency by \(K\times\), and risks FP16 numerical overflow when summing hundreds of gradient tensors
- It requires dedicated InfiniBand links between CPU and GPU for each microstep
- It forces batch normalization layers to synchronize across all nodes on every forward pass
The optimal interval formula \(\tau_{\text{opt}} = \sqrt{2 \cdot T_{\text{write}} \cdot \text{MTBF}_{\text{system}}}\), which balances checkpoint write overhead against rework risk following hardware failure, is known as the ____.
True or False: If a 7B-parameter model fits comfortably in single-GPU memory with high compute utilization under standard DDP, universally enabling ZeRO-3 / FSDP full sharding will always increase training throughput by reducing per-GPU memory consumption.
Summary
This chapter introduced the “scaling wall”: the point where adding more GPUs eventually makes training slower rather than faster. Distributed training is not a simple hardware problem; it is a constraint satisfaction problem governed by the interaction between model size, batch size, and interconnect bandwidth.
The 3D Parallelism Cube (figure 18) is a useful framework for scaling large models. Data parallelism unrolls the outer loop of training to scale throughput; tensor parallelism vectorizes the inner loops of matrix multiplication to fit memory; and pipeline parallelism stages sequential layers to reduce communication frequency. Together, these strategies map models larger than any single memory bank onto a fleet of accelerators with finite bandwidth.
The choice of parallelism can be understood as a loop transformation applied by the cluster-level compiler. Matching logical communication patterns to physical hardware hierarchies moves the system from the “linear scaling regime” of small clusters to the communication-bound regime of very large accelerator fleets.
Throughout this chapter, the partitioning strategies were developed directly for dense large-model training and recommendation-style workloads. Table 10 also shows how the same constraint logic reaches the federated edge case, where the bottleneck shifts from accelerator memory and fabric bandwidth to unreliable devices, privacy, and intermittent connectivity.
Lighthouse 1.3: Distributed archetype spectrum
The operating point in the 3D Parallelism Cube shifts depending on the system’s primary bottleneck:
| Archetype | Primary Partitioning Strategy | The Logic |
|---|---|---|
| Archetype A (GPT-4/Llama-3) | Hybrid 3D Parallelism | Combine Tensor (width), Pipeline (depth), and Data (throughput) to fit 3.5 TB of weights. |
| Archetype B (DLRM at Scale) | Embedding Sharding | Partition massive 10 TB+ tables across a Parameter Server fleet; use sparse AllToAll updates. |
| Archetype C (Federated MobileNet) | Federated Learning | The same logic extends to coordinating on-device MobileNet updates across unreliable, privacy-constrained edge devices; Edge Intelligence develops this regime. |
The parallelism strategies explored throughout this chapter (data, tensor, pipeline, and expert) provide a toolkit for partitioning training workloads across a cluster. They are not mutually exclusive alternatives but complementary dimensions of a shared optimization space. Systems such as Megatron-LM achieve efficient scaling by combining multiple strategies, using tensor parallelism within nodes, pipeline parallelism across node groups, data parallelism for throughput, and expert parallelism for capacity scaling.
Key Takeaways: Parallelism relocates the tax
- Splitting work moves the bottleneck: Distributed training does not make the underlying computation smaller; it converts memory pressure into communication volume, synchronization delay, or idle pipeline time (principle 8). The winning strategy is the split that sends overhead to the least binding part of the fleet.
- Data parallelism ends at convergence: Replicas scale throughput cleanly only while larger global batches still improve optimization. Past the critical batch size, AllReduce cost and reduced gradient noise turn “more workers” into slower or less stable learning unless schedules, warmup, and accumulation change.
- Sharding buys memory with messages: ZeRO and FSDP make 100B+ parameter models feasible by partitioning optimizer state, gradients, and parameters (principle 3). The price is a stricter communication schedule of ReduceScatter and AllGather operations that must be overlapped or hidden.
- Tensor parallelism belongs near NVLink: Tensor parallelism splits matrix operations inside layers, so it needs the high-bandwidth intra-node fabric that A100 and H100 NVLink provide. Stretching that traffic across racks turns a memory-capacity solution into a communication bottleneck.
- Pipeline parallelism trades bytes for bubbles: Layer staging reduces communication frequency, but fill and drain slots leave accelerators idle (principle 10). Microbatching with \(m \gg p\) is the mechanism that turns model depth into throughput rather than pipeline slack.
- The 3D cube is a hardware map: Real frontier training combines data, tensor, pipeline, expert, and sharded parallelism because no single axis fits the model and the fleet (principle 9). Logical groups must map to HBM, NVLink, InfiniBand, and failure domains together.
Beneath the cube and its three axes lies a single trade. No way of splitting a model makes the underlying work smaller; each only moves it, turning a memory limit into a communication cost or a communication cost into idle time. Data, tensor, and pipeline parallelism spend communication to win the memory headroom no single accelerator has, and coordination is the tax paid to keep the split consistent. This is the displacement of overhead (principle 13), the law the rest of the volume turns on: the execution tax of scale cannot be removed, only relocated among compute, communication, and coordination. The cube is the map of where the tax can be sent; the engineering is choosing the destination that binds least.
What’s Next: From logic to traffic
Splitting the mathematics defines the logical traffic patterns of the Machine Learning Fleet. These patterns eventually reach the physical wires of the data center. Collective Communication opens the black box of the collective operations (AllReduce, AllToAll) that preserve consistency, moving from what must be sent to how it is routed through rings, trees, and rails.
Self-Check: Question
What fundamental architectural principle summarizes the trade-offs of all distributed training strategies in this chapter?
- Amdahl’s law can be eliminated entirely by combining tensor and pipeline parallelism
- Data parallelism scales indefinitely as long as Ring AllReduce is selected
- Model parameters can be replicated infinitely without consuming network bandwidth
- The displacement of overhead: distributed parallelism never eliminates total work, but only relocates the execution tax among memory footprint, communication volume, synchronization latency, and pipeline idle time
Contrast the primary partitioning strategy and binding hardware bottleneck for Archetype A (Dense Large Language Models like GPT-4/Llama-3) versus Archetype B (Large-Scale Recommendation Models / DLRM).
Explain why data, pipeline, and tensor parallelism are treated as orthogonal scaling dimensions rather than mutually exclusive strategies in the 3D Parallelism Cube, and provide an example configuration that combines them.
Self-Check Answers
Self-Check: Answer
What structural requirement most fundamentally distinguishes distributed training from a stateless distributed web service that handles independent HTTP queries across multiple replicas?
- Distributed training must run on strictly more nodes than web services because neural networks cannot execute on small clusters
- Distributed training requires all workers to maintain a consistent mathematical view of mutable shared parameters via collective gradient synchronization
- Distributed training is always compute-bound while web serving is strictly latency-bound, preventing the workloads from sharing cluster hardware
- Distributed training cannot tolerate hardware failures because synchronization protocols prevent checkpoint-based recovery
Answer: The correct answer is B. Stateless web service replicas process independent queries without needing consensus on shared internal state. In contrast, distributed training must produce a single, mathematically coherent set of updated weights after every iteration, making collective gradient synchronization and consistency barriers mandatory across all workers. The claim regarding cluster node counts is false because both small training jobs and massive serving deployments exist. The assertion that workloads are rigidly bound by distinct single constraints is a false dichotomy. Distributed training relies extensively on checkpointing to recover from frequent node failures.
Learning Objective: Distinguish distributed training execution semantics from stateless distributed service architectures
**Order the causal phases of a single synchronous data-parallel training iteration from earliest to latest:
- Update local model parameters using aggregated gradients via the optimizer
- Partition and assign a disjoint batch shard to each worker rank
- Synchronize local gradient tensors across all workers using AllReduce
- Compute forward and backward passes locally on the assigned batch shard**
Answer: (2) -> (4) -> (3) -> (1). First, (2) Partition and assign a disjoint batch shard ensures each worker receives unique training samples. Next, (4) Compute forward and backward passes evaluates the loss and computes local gradients. Then, (3) Synchronize local gradient tensors executes an AllReduce collective so every rank obtains identical averaged gradients. Finally, (1) Update local model parameters applies the optimizer update to keep replica weights synchronized.
Learning Objective: Order the causal execution phases of a synchronous data-parallel training iteration
Framing distributed parallelism strategies as compiler loop transformations across the training loop’s iterators (batches, layers, and operations), which distributed strategy corresponds to vectorization (SIMD) of inner matrix multiplications?
- Tensor parallelism, which splits inner matrix-multiplication operations across devices with NVLink acting as a cluster-scale register fabric
- Data parallelism, which unrolls the outer batch loop across devices so each rank processes independent data indices
- Pipeline parallelism, which stages sequential layer operations across devices analogous to CPU instruction pipelining
- Model replication, which duplicates the entire forward and backward execution redundantly across machines
Answer: The correct answer is A. Tensor parallelism splits the inner matrix multiplication operations within each layer across devices, effectively acting as a cluster-scale SIMD unit where NVLink provides the low-latency communication analogous to a vector register file. The loop transformation unrolling the outer batch loop describes data parallelism. Staging sequential layer blocks across time steps describes pipeline parallelism. Redundant execution describes replication rather than intra-operation parallelization.
Learning Objective: Classify distributed parallelism strategies according to compiler loop transformations
A 1,024-GPU Bulk Synchronous Parallel (BSP) training job reports a 180 ms average per-worker compute time, but the measured step time averages 340 ms with a p99 latency of 720 ms. Explain the mechanism behind this discrepancy and why it worsens as cluster size grows.
Answer: Bulk Synchronous Parallel (BSP) enforces a global synchronization barrier at every gradient aggregation step, forcing all 1,024 workers to wait for the slowest device (the straggler). Even if 1,023 GPUs finish compute in 180 ms, a single worker experiencing thermal throttling, NCCL network contention, host memory paging, or OS jitter delays the entire cluster. As the worker count \(N\) increases, the probability of sampling an extreme tail latency in each step approaches 1, magnifying barrier idle time and collapsing cluster-wide Model Flops Utilization (MFU).
Learning Objective: Explain how bulk synchronous barriers amplify per-worker tail latency across large clusters
True or False: In a synchronous distributed training cluster using collective communication, if control-flow divergence causes worker Rank 0 to execute an AllReduce collective while Rank 1 skips it, the cluster will automatically recover by substituting stale gradients from the previous step.
Answer: False. Collective communication primitives (such as NCCL AllReduce) require every participating rank in the communicator group to execute matching collective calls in the exact same sequence with identical tensor shapes. If one rank skips a collective due to conditional execution, participating ranks will block permanently waiting for the missing peer, causing an unrecoverable collective deadlock (hang).
Learning Objective: Analyze collective communication deadlocks caused by control-flow divergence across distributed workers
Self-Check: Answer
A 7B-parameter transformer in mixed precision requires 112 GB of static memory for standard replicated DDP (14 GB for FP16 weights, 14 GB for FP16 gradients, and 84 GB for FP32 Adam master weights, momentum, and variance). Under ZeRO Stage 3 (or FSDP Full Shard) across 64 GPUs, what is the resulting static training memory per GPU?
- \(112\text{ GB}\), because ZeRO-3 only reduces dynamic activation memory rather than model state
- \(28\text{ GB}\), because only optimizer states are sharded while parameters and gradients remain fully replicated
- \(1.75\text{ GB}\), because weights, gradients, and optimizer states are each partitioned equally across all 64 workers
- \(0.25\text{ GB}\), because INT4 quantization eliminates floating-point representation entirely
Answer: The correct answer is C. ZeRO-3 / FSDP partitions all three components of static training state—FP16 model weights (2 bytes/param), FP16 gradients (2 bytes/param), and FP32 Adam states (12 bytes/param)—across the \(N=64\) ranks (\(16\text{ bytes/param} / 64 = 0.25\text{ bytes/param}\) per device). For a 7B model (\(7 \times 10^9 \times 0.25\text{ bytes} = 1.75\text{ GB}\) per GPU), this leaves substantial HBM headroom for dynamic activations. Asserting that state remains 112 GB ignores parameter and optimizer sharding. Limiting sharding to optimizer state describes ZeRO-1 (\(28\text{ GB}\) weights + grads + sharded optim). Claiming INT4 compression confuses precision quantization with state sharding.
Learning Objective: Calculate per-GPU static memory reduction under ZeRO-3 parameter, gradient, and optimizer state sharding
The chapter’s GPT-2 scaling case study shows that 8 GPUs on a single NVLink-connected node using gradient accumulation achieves higher cost efficiency and throughput than naive scale-out to 32 GPUs across a 10 Gb/s commodity Ethernet network. Explain the quantitative mechanism driving this outcome.
Answer: Gradient accumulation performs multiple local forward and backward microsteps before synchronizing gradients, keeping communication inside the 8-GPU node’s high-speed NVLink fabric (>600 GB/s) where AllReduce latency is negligible (<10 ms). In contrast, naive scale-out to 32 GPUs over 10 Gb/s (1.25 GB/s) Ethernet forces large gradient tensors across a slow inter-node network, causing AllReduce time to exceed compute time (\(T_{\text{comm}} \gg T_{\text{compute}}\)). Scaling efficiency drops to ~30% on the commodity network while exceeding 95% on the single NVLink node.
Learning Objective: Compare gradient accumulation on high-speed intra-node fabrics against naive scale-out across bandwidth-constrained networks
**Order the memory sharding stages of data parallelism from least sharded (highest memory footprint) to deepest sharded (lowest memory footprint):
- ZeRO-2 (shards optimizer states and gradients using ReduceScatter)
- Standard DDP (full replication of parameters, gradients, and optimizer states)
- ZeRO-3 / FSDP (shards optimizer states, gradients, and model parameters with layer-wise AllGather/ReduceScatter)
- ZeRO-1 (shards optimizer states only, keeping parameters and gradients replicated)**
Answer: (2) -> (4) -> (1) -> (3). First, (2) Standard DDP replicates all 16 bytes/param on every GPU. Next, (4) ZeRO-1 shards optimizer states (~4\(\times\) static reduction). Then, (1) ZeRO-2 additionally shards gradients (~8\(\times\) static reduction). Finally, (3) ZeRO-3 / FSDP shards the parameters themselves, achieving linear \(1/N\) memory reduction.
Learning Objective: Classify the memory sharding stages of ZeRO and FSDP by depth of partitioned state
If a model’s parameters, gradients, and activations already fit comfortably within single-GPU memory with high compute utilization under standard DDP, what is the expected throughput impact of enabling ZeRO-3 / FSDP Full Sharding?
- Throughput will increase linearly by \(N\times\) because sharding reduces per-GPU gradient memory
- Throughput will decrease because FSDP introduces additional AllGather collectives before forward and backward passes and ReduceScatter on the critical path
- Throughput will remain exactly identical because NCCL automatically hides all collective communication overhead
- Throughput will double because parameter sharding eliminates the backward pass entirely
Answer: The correct answer is B. ZeRO-3 / FSDP is a memory-saving technique, not a throughput accelerator: it requires an AllGather before each layer’s forward pass, a second AllGather before each layer’s backward pass, and a ReduceScatter after backward computation. When memory capacity is not a binding constraint, these additional collective operations add latency to the critical path and reduce training throughput by 10–25% compared to standard DDP. Claiming linear speedup confuses memory partitioning with compute scaling. Asserting zero overhead ignores physical network latency and bandwidth limits. Claiming the backward pass is eliminated is mathematically false.
Learning Objective: Evaluate the communication throughput overhead of FSDP when device memory is not the binding constraint
Why did modern distributed training frameworks replace centralized parameter servers with decentralized AllReduce collective topologies for dense synchronous training?
- Parameter servers cannot execute stochastic gradient descent updates with mathematical validity
- Parameter servers are restricted to CPU hosts and cannot interface with GPU accelerators
- AllReduce eliminates all network traffic by updating weights locally without exchanging data
- Centralized parameter servers create an \(\mathcal{O}(N)\) network bottleneck at the server’s network interface, whereas AllReduce distributes communication symmetrically across all workers
Answer: The correct answer is D. In a parameter server architecture with \(N\) workers, the central server’s network interface must ingest gradients from and broadcast weights to all \(N\) workers simultaneously, creating an \(\mathcal{O}(N)\) bottleneck that collapses scaling past a few nodes. Symmetrical Ring and Tree AllReduce topologies distribute communication evenly across all workers, bounding per-GPU transfer volume to \(2(N-1)/N \times M_{\text{grad}}\) regardless of cluster size. The claim that parameter servers cannot implement valid SGD is false. The assertion that parameter servers cannot use GPUs is factually incorrect. The claim that AllReduce eliminates network communication is false since gradient synchronization requires explicit data exchange.
Learning Objective: Compare centralized and decentralized synchronization topologies in terms of bandwidth bottlenecks for dense gradient traffic
In standard PyTorch DistributedDataParallel, each worker uses a ____ to partition dataset indices deterministically and prevent sample overlap across ranks based on process rank and epoch seed.
Answer: DistributedSampler. The DistributedSampler ensures that each data-parallel worker processes a unique, non-overlapping subset of the dataset in each epoch, maintaining the statistical validity of the minibatch gradient estimate.
Learning Objective: Identify the component responsible for deterministic, non-overlapping dataset partitioning in data-parallel training
Self-Check: Answer
A team scales a data-parallel training run from 8 to 64 workers, increasing the effective global batch size from 512 to 4,096 samples. If the model operates comfortably below its critical batch size, which learning-rate adjustment rule is the standard default?
- Keep the learning rate unchanged, because large-batch variance reduction automatically stabilizes optimization
- Multiply the learning rate by 64 to match the total worker count rather than the batch expansion ratio
- Multiply the learning rate by \(\sqrt{8}\) from step zero without a warmup schedule
- Multiply the learning rate by \(8\times\) (matching the batch expansion ratio) and include a gradual linear warmup schedule
Answer: The correct answer is D. Below the critical batch size, the linear scaling rule (\(\eta_{\text{new}} = k \cdot \eta_{\text{base}}\) where \(k = 4096 / 512 = 8\)) maintains optimization dynamics by scaling step size proportionally with gradient variance reduction. A gradual warmup schedule is necessary during initial epochs to stabilize the optimizer while moving into well-conditioned loss regions. Keeping the learning rate unchanged under-utilizes the large batch, requiring unnecessarily many steps. Scaling by worker count (\(64\times\)) rather than the batch factor (\(8\times\)) causes catastrophic divergence. Applying square-root scaling without warmup is heuristic and sub-optimal below the critical batch size.
Learning Objective: Apply linear learning rate scaling with warmup for large-batch data-parallel training
Explain the distinction between hardware efficiency and statistical efficiency in distributed training, and describe a concrete scenario where a cluster configuration achieves high hardware efficiency but poor statistical efficiency.
Answer: Hardware efficiency measures how effectively cluster compute capacity (TFLOP/s) translates into step throughput (samples/sec), penalized by communication, kernel launches, and synchronization barriers. Statistical efficiency measures how effectively processed samples reduce the training loss (loss reduction per sample). In a scenario where global batch size is scaled far beyond the critical batch size (\(B \gg B^*\)) across 1,024 GPUs, hardware efficiency remains high (high MFU and fast step times), but statistical efficiency collapses because gradient updates become redundant; as a result, total training time and cost to reach target validation loss increase despite high throughput.
Learning Objective: Compare hardware throughput efficiency with statistical sample efficiency in distributed optimization
Why does Stale-Synchronous Parallel (SSP) training typically require a smaller effective learning rate than Bulk-Synchronous Parallel (BSP) at the same global batch size?
- Bounded parameter staleness introduces an optimization error term that destabilizes parameter updates unless damped by a smaller learning rate
- SSP workers compute exact full-batch gradients instead of stochastic minibatches
- SSP eliminates all inter-GPU network communication, which alters the loss landscape
- SSP is mathematically restricted to second-order quasi-Newton optimizers that require small step sizes
Answer: The correct answer is A. In SSP, workers compute gradients against parameter snapshots that may lag up to \(\tau_{\text{stale}}\) steps behind the most recent model state. This staleness introduces a gradient error term proportional to \(\tau_{\text{stale}} \cdot \eta \cdot \|\nabla \mathcal{L}\|\); damping the learning rate \(\eta\) is necessary to bound this staleness penalty and prevent divergence. The assertion that SSP computes full-batch gradients is false. The claim that SSP eliminates communication is incorrect because workers still synchronize parameters asynchronously. The claim that SSP is restricted to second-order optimizers is false.
Learning Objective: Explain the impact of bounded gradient staleness on optimizer stability and learning rate selection
The threshold batch size \(B^* \approx \frac{\text{tr}(\Sigma)}{\|\nabla \mathcal{L}(\theta)\|^2}\), representing the point where gradient variance equals true gradient magnitude and past which larger batches yield diminishing returns in sample efficiency, is known as the ____.
Answer: Critical Batch Size (or critical batch size). Below the critical batch size \(B^*\), doubling batch size halves required optimization steps; above \(B^*\), additional batch expansion provides diminishing returns in sample efficiency.
Learning Objective: Define critical batch size in terms of gradient covariance trace and gradient norm
True or False: Scaling data parallelism significantly beyond a model’s critical batch size (\(B \gg B^*\)) continues to reduce total GPU-hours to reach target loss because additional GPUs always increase processed samples per second.
Answer: False. Beyond the critical batch size \(B^*\), gradient updates become saturated with redundant information, causing sample efficiency (loss reduction per sample) to collapse. While hardware step throughput (samples/sec) may scale with added workers, the total number of samples and FLOPs required to achieve target loss grows disproportionately, increasing total GPU-hours and compute cost.
Learning Objective: Evaluate the compute cost consequences of scaling data parallelism beyond the critical batch size
Self-Check: Answer
In Megatron-LM style tensor parallelism for a transformer layer, how are the linear projections in the multi-head attention block arranged to minimize cross-device collective communication?
- Both QKV and Output projections are column-parallel, requiring an AllGather after each projection
- Both QKV and Output projections are row-parallel, requiring an AllReduce before each projection
- The QKV projection is column-parallel and the Output projection is row-parallel, requiring only a single AllReduce at the end of the attention block
- The QKV projection is partitioned across pipeline stages while the Output projection is replicated across data-parallel ranks
Answer: The correct answer is C. In Megatron-LM, the first linear layer (QKV projection) is column-parallel (\(W = [W_1 | W_2]\)), producing partitioned intermediate activations without communication. The second linear layer (Output projection) is row-parallel (\(W = [W_1 ; W_2]\)), taking those partitioned activations directly as inputs and generating partial sums. Only a single AllReduce at the end of the row-parallel projection is required to sum the partial outputs. Using column-parallel for both layers would require an intermediate AllGather. Using row-parallel for both layers would require an AllReduce before the second layer. Splitting QKV and Output across pipeline stages confuses intra-layer tensor parallelism with inter-layer pipelining.
Learning Objective: Analyze the column-then-row linear layer pairing in Megatron-LM tensor parallelism
In a synchronous 1F1B pipeline-parallel training schedule with \(p=8\) stages and \(m=32\) microbatches, calculate the theoretical pipeline bubble fraction and explain how increasing the microbatch count \(m\) affects bubble overhead and activation memory.
Answer: The bubble fraction is \(F_{\text{bubble}} = \frac{p-1}{m+p-1} = \frac{8-1}{32+8-1} = \frac{7}{39} \approx 17.9\%\). Increasing the microbatch count \(m\) relative to stage count \(p\) drives the bubble fraction down toward zero (\(F_{\text{bubble}} \to 0\)). Under 1F1B scheduling, peak activation memory remains bounded at \(\mathcal{O}(p)\) rather than scaling as \(\mathcal{O}(m \times p)\), making larger microbatch counts practical.
Learning Objective: Calculate pipeline bubble overhead and explain the role of microbatch count in pipeline utilization
**Order the four sequential phases of token routing and execution in Expert Parallelism (Mixture of Experts) across distributed workers:
- All-to-All Combine: Processed token embeddings are routed back to their original source devices for residual connections
- Computation: Sharded expert feed-forward networks process their assigned token batches
- Gating: A router network evaluates tokens and selects top-k destination experts
- All-to-All Dispatch: Tokens are shuffled across the network fabric to the devices hosting their selected experts**
Answer: (3) -> (4) -> (2) -> (1). First, (3) Gating evaluates token representations to select destination experts. Next, (4) All-to-All Dispatch routes tokens across the interconnect to the GPUs hosting the chosen experts. Then, (2) Computation executes the local expert feed-forward layers on the received tokens. Finally, (1) All-to-All Combine routes the processed representations back to their originating devices.
Learning Objective: Order the four phases of token routing and execution in expert parallelism
In a Mixture of Experts (MoE) distributed training system, what occurs when a hot expert receives more tokens than its allocated buffer capacity determined by the Capacity Factor \(C\)?
- Excess tokens are dropped and pass through the MoE layer unprocessed via the residual connection
- The expert GPU dynamically allocates host CPU memory over PCIe, halting computation on other ranks
- The All-to-All collective automatically pauses until all sibling experts process matching token counts
- The extra tokens are rerouted to a random cold expert regardless of gating network weights
Answer: The correct answer is A. To prevent memory overflow and load-imbalance stalls on devices hosting popular experts, MoE frameworks enforce a capacity limit \(C \cdot (B \times S)/E\). When an expert’s assigned token count exceeds this buffer, the excess tokens are dropped and bypass the feed-forward computation, flowing directly through the residual connection. Dynamically allocating host CPU memory would stall GPU execution. Collective communication does not pause dynamically for uneven token counts. Rerouting to random unselected experts would violate gating semantics.
Learning Objective: Explain the function of the capacity factor and token dropping in distributed MoE routing
Contrast the communication patterns and hardware interconnect requirements of dense LLM training (Megatron-style Tensor Parallelism) with large-scale Recommendation Systems (DLRM embedding sharding).
Answer: Dense LLM tensor parallelism performs frequent, dense AllReduce collectives inside every layer on the critical path, requiring high-bandwidth, low-latency NVLink (>900 GB/s) within GPU nodes. In contrast, DLRM recommendation models shard massive embedding tables (terabytes in size) across CPU/GPU parameter servers and communicate via sparse pull/push lookups, transferring only the embedding rows touched by a sparse batch; this workload is bound by aggregate memory capacity and random-access memory latency rather than dense bisection bandwidth.
Learning Objective: Compare dense model tensor parallelism against sparse embedding sharding communication patterns
True or False: Ring Attention scales context length beyond single-GPU memory capacity by circulating key/value blocks in an accelerator ring across \(N-1\) communication rounds while overlapping transfer with block-level attention computation.
Answer: True. Ring Attention partitions the query, key, and value sequences across \(N\) GPUs so each device holds only \(S/N\) tokens of key/value state. In each of the \(N-1\) communication rounds, each GPU computes attention on its local query block against the resident key/value block while concurrently receiving the next key/value block from its neighbor over NVLink, achieving linear context scaling.
Learning Objective: Evaluate the memory and communication mechanism of Ring Attention for long-context sequence parallelism
Self-Check: Answer
Why does standard FP8 mixed-precision training pair the E4M3 format for forward-pass activations and weights with the E5M2 format for backward-pass gradients?
- E4M3 has a wider numerical range than E5M2, which prevents gradient underflow during backpropagation
- E4M3 provides higher precision (3 mantissa bits, range \(\pm 448\)) suitable for bounded activations, while E5M2 provides a wider dynamic range (5 exponent bits, range \(\pm 57344\)) necessary for gradients spanning multiple orders of magnitude
- E4M3 is an integer format while E5M2 is a floating-point format supported exclusively on CPU hosts
- E5M2 requires half the memory bandwidth of E4M3, accelerating the backward pass AllReduce
Answer: The correct answer is B. Forward-pass activations and weights follow relatively well-behaved, bounded distributions where higher numerical precision (3 mantissa bits in E4M3) preserves representational accuracy. Backward-pass gradients can vary across many orders of magnitude and require the wider dynamic range of E5M2 (5 exponent bits, representing values up to \(\pm 57344\)) to prevent catastrophic overflow and underflow. The assertion that E4M3 has a wider range than E5M2 inverts their specifications. Both E4M3 and E5M2 are 8-bit floating-point formats occupying identical memory footprints (1 byte per element).
Learning Objective: Compare E4M3 and E5M2 FP8 formats based on numerical range, precision, and training tensor roles
Explain why per-tensor dynamic scaling is necessary in FP8 distributed training and describe how the scaling factor is computed during execution.
Answer: Because FP8 has a very narrow dynamic range (maximum representable value of 448 in E4M3), fixed scaling causes either gradient underflow (values rounding to zero) or overflow (values clipping to maximum). Per-tensor dynamic scaling tracks the running maximum absolute value of each tensor and dynamically computes a scale factor \(S = \text{FP8}_{\max} / \max(|X|)\), multiplying tensors by \(S\) before casting to FP8 and dividing by \(S\) after matrix operations to keep values within the representable range.
Learning Objective: Explain how dynamic scaling prevents underflow and overflow in narrow dynamic range FP8 tensors
True or False: Converting training tensors to FP8 eliminates the need for tensor and pipeline parallelism when training 100B+ parameter models because FP8 changes the geometric parallelization axes.
Answer: False. FP8 halves tensor payload sizes (reducing memory traffic and communication volume relative to FP16/BF16), but it does not change the parallelization axes. A 100B+ parameter model still exceeds the single-device memory capacity of modern accelerators even in FP8 and requires tensor, pipeline, or sharded data parallelism.
Learning Objective: Distinguish numerical precision reduction from model partitioning across distributed parallelism axes
Self-Check: Answer
In a 3D parallelism layout (\(N_{\text{total}} = d \times p \times t\)), what hardware-matching principle dictates mapping tensor parallelism intra-node while pipeline parallelism spans inter-node links?
- Pipeline parallelism requires more memory bandwidth than NVLink can provide, forcing it onto InfiniBand
- Tensor parallelism requires only point-to-point transfers, making it insensitive to link latency
- Hierarchy-aware parallelism (bandwidth matching): TP launches high-frequency per-layer AllReduce collectives demanding intra-node NVLink bandwidth (>900 GB/s), whereas PP transmits boundary activations tolerating inter-node InfiniBand
- Data parallelism cannot execute across nodes that use tensor parallelism internally
Answer: The correct answer is C. Hierarchy-aware parallelism maps each parallelism dimension to the interconnect tier matching its communication intensity. Tensor parallelism requires low-latency, high-frequency AllReduce operations inside every transformer layer, necessitating intra-node NVLink (>900 GB/s). Pipeline parallelism transmits only boundary activations between stages once per microbatch, tolerating slower inter-node InfiniBand links (50–100 GB/s). The claim that PP requires more bandwidth than NVLink is false. The assertion that TP is latency-insensitive contradicts its per-layer AllReduce requirement. Data parallelism routinely wraps model-parallel pipelines.
Learning Objective: Apply the hierarchy-aware bandwidth matching principle to place 3D parallelism dimensions onto hardware tiers
A 175B-parameter model is trained using 3D hybrid parallelism with \(t=8\) (tensor parallel) and \(p=16\) (pipeline parallel). Using 350 GB for FP16 weights and 1,400 GB for 8-byte Adam optimizer states, calculate the static memory footprint per GPU and explain why pipeline parallelism is necessary in addition to tensor parallelism.
Answer: Total static state is \(350\text{ GB} + 1400\text{ GB} = 1750\text{ GB}\). If using only tensor parallelism with \(t=8\), per-GPU static memory would be \(1750 / 8 = 218.75\text{ GB}\), causing out-of-memory on an 80 GB GPU. Adding 16-stage pipeline parallelism (\(p=16\)) reduces static memory to \(1750 / (8 \times 16) \approx 13.67\text{ GB}\) per GPU (budgeted ~14–15 GB), leaving ~65 GB of HBM for dynamic activation memory.
Learning Objective: Calculate per-GPU static memory requirements in a 3D hybrid parallelism configuration
**Order the three primary communication patterns in hybrid 3D parallelism from highest communication frequency (most frequent) to lowest communication frequency (least frequent):
- Data-parallel AllReduce across replica groups
- Tensor-parallel intra-node AllReduce within transformer blocks
- Pipeline-parallel point-to-point activation transfers across stage boundaries**
Answer: (2) -> (3) -> (1). First, (2) Tensor-parallel intra-node AllReduce occurs multiple times per transformer layer on every forward and backward pass (highest frequency). Next, (3) Pipeline-parallel point-to-point transfers occur once per microbatch at stage boundaries (intermediate frequency). Finally, (1) Data-parallel AllReduce occurs once per global optimizer step (lowest frequency).
Learning Objective: Rank the communication patterns of 3D parallelism by their temporal frequency across a training step
In a 16,384-GPU cluster configured with \(\text{TP}=8\), \(\text{PP}=16\), and \(\text{DP}=128\), how many GPUs participate in each individual data-parallel AllReduce communicator group?
- \(16{,}384\text{ GPUs}\), because all accelerators in the cluster must synchronize in a single global collective
- \(128\text{ GPUs}\), because each rank communicates only with its corresponding tensor-and-pipeline shard position across the 128 model replicas
- \(8\text{ GPUs}\), matching the tensor-parallel degree within each local node
- \(16\text{ GPUs}\), matching the pipeline stage depth across the cluster spine
Answer: The correct answer is B. In 3D parallelism, the cluster contains \(\text{DP}=128\) complete model replicas, each partitioned across \(\text{TP} \times \text{PP} = 8 \times 16 = 128\) distinct shard positions. For each shard position, the corresponding rank in every replica forms an independent data-parallel group of 128 GPUs that AllReduces its specific parameter shard. Participating in a single global collective of 16,384 GPUs would discard 3D partitioning. Confining DP groups to 8 or 16 GPUs confuses data-parallel replica count with tensor or pipeline dimensions.
Learning Objective: Analyze communicator group sizes and partitioning in a large-scale 3D parallelism cluster
True or False: A uniform fat-tree network topology providing equal bisection bandwidth between all node pairs is always more cost-effective than a rail-optimized topology when training large language models with hierarchy-aware 3D parallelism.
Answer: False. Hierarchy-aware 3D parallelism confines high-frequency tensor parallelism to intra-node NVLink and structures inter-node pipeline and data-parallel traffic along predictable rail dimensions. A rail-optimized topology provides dedicated high-bandwidth rails matching these communication groups at lower hardware and cabling cost, whereas a uniform any-to-any fat-tree over-provisions cross-cluster bisection bandwidth that 3D parallelism does not utilize.
Learning Objective: Evaluate how network topology design interacts with hierarchy-aware parallelism communication locality
Self-Check: Answer
Why does Proximal Policy Optimization (PPO) in multi-model RLHF create an infrastructure asymmetry challenge that does not exist in standard pretraining?
- It combines compute-intensive training models (Policy and Value models requiring backward passes and optimizer state) with memory-bandwidth-bound inference workloads (Reference and Reward models requiring autoregressive rollout and KV cache storage)
- All four models in PPO must be trained with identical optimizer hyperparameters on the same GPU
- PPO requires all models to be converted to INT4 precision to enable cross-node gradient synchronization
- PPO eliminates backward passes entirely, turning distributed training into a stateless inference service
Answer: The correct answer is A. PPO-style RLHF orchestrates four distinct models with asymmetric compute profiles: the Policy and Value models require full backward passes, gradient storage, and optimizer state (compute-intensive training), while the Reference and Reward models execute only forward passes and autoregressive token generation (memory-bandwidth-bound inference). The claim that all models share identical training updates is false because Reference and Reward models are frozen. The claim regarding INT4 conversion is incorrect. Asserting that PPO eliminates backward passes is false since policy optimization requires gradient updates.
Learning Objective: Analyze the infrastructure asymmetry between training and inference models in multi-model RLHF
Explain how Direct Preference Optimization (DPO) simplifies cluster infrastructure compared to PPO, detailing which models and computational phases are eliminated.
Answer: DPO eliminates the separate reward model and value/critic model entirely, reducing the fleet from four models to two (Policy in training mode and Reference in inference mode), which cuts static parameter memory by 16–48%. Furthermore, DPO eliminates online autoregressive sequence generation and dynamic KV cache storage, transforming the workflow into standard supervised-style batch training over static preference triples \((x, y_w, y_l)\) using pretraining 3D parallelism pipelines.
Learning Objective: Compare the memory and execution pipeline requirements of PPO and DPO alignment architectures
During the autoregressive generation phase of PPO alignment for a 70B-parameter model, what resource represents the primary memory consumer and system performance bottleneck?
- FP32 Adam momentum and variance buffers stored on CPU hosts
- Key-Value (KV) cache memory in GPU HBM, bounded by memory bandwidth during token-by-token generation
- AllReduce gradient communication buffers across InfiniBand switches
- Host filesystem NVMe write bandwidth for checkpoint dumping
Answer: The correct answer is B. During autoregressive sequence rollout, the model generates tokens one by one, where every step requires loading model weights from HBM to compute a single token (memory-bandwidth bound). The accumulating Key-Value (KV) cache for hundreds of concurrent generation sequences consumes tens to hundreds of gigabytes of accelerator HBM (e.g. ~86 GB for Llama-2-70B GQA at batch 256). Adam optimizer buffers and AllReduce gradient buffers are active during the subsequent backward training phase, not the autoregressive generation phase. Filesystem checkpointing occurs periodically, not per generation token.
Learning Objective: Identify the binding resource bottleneck during the generation phase of online RLHF
The scheduling approach in RLHF distributed systems where the accelerator fleet alternates between an autoregressive rollout phase dominated by KV cache memory and a backward training phase dominated by matrix GEMM compute is known as ____.
Answer: temporal multiplexing. Temporal multiplexing reconfigures hardware resources dynamically between inference-shaped sequence generation and training-shaped gradient backpropagation.
Learning Objective: Define temporal multiplexing as the scheduling strategy for alternating execution phases in RLHF
True or False: Because Direct Preference Optimization (DPO) eliminates reward and value models and avoids online sequence generation, it is strictly superior to PPO for all model alignment and exploration tasks.
Answer: False. DPO is restricted to optimizing against a static offline dataset of preference pairs, preventing the policy from generating novel outputs and exploring beyond the provided data. For tasks requiring iterative self-improvement and exploration (such as multi-step mathematical reasoning), PPO’s online rollout and reward feedback loop can discover superior trajectories, justifying its higher infrastructure overhead.
Learning Objective: Evaluate the functional trade-offs between static DPO optimization and dynamic PPO exploration
Self-Check: Answer
According to the chapter’s parallelism decision framework, which condition indicates that standard DistributedDataParallel (DDP) is the optimal default strategy over FSDP, TP, or PP?
- The model’s single layer weight matrices exceed the HBM capacity of an individual GPU
- The cluster is connected by slow 10 Gb/s commodity Ethernet across 1,024 nodes
- The entire model state (parameters, gradients, and optimizer state) fits comfortably within single-device memory with sufficient headroom for activations
- The model architecture contains sparse Mixture-of-Experts routing layers
Answer: The correct answer is C. When full model state (weights, gradients, and optimizer state) fits within single-GPU memory with adequate room for batch activations, standard DDP provides maximum throughput by avoiding the per-layer AllGather/ReduceScatter collective overhead of FSDP and the pipeline bubbles of PP. When single layers exceed GPU memory, tensor parallelism is required. Scaling across slow 10 Gb/s Ethernet causes communication walls that require gradient accumulation or high-speed fabrics. Sparse MoE layers require expert parallelism with All-to-All routing.
Learning Objective: Identify the binding constraint criteria that validate using standard Data Parallelism
An engineering team observes that an 8-GPU training job using pure pipeline parallelism achieves only 35 percent Model Flops Utilization (MFU), while interconnect links are largely idle. Diagnosing the issue with the bubble formula \(F_{\text{bubble}} = \frac{p-1}{m+p-1}\), explain the root cause and propose two concrete adjustments to recover throughput.
Answer: The low utilization is caused by pipeline bubble overhead, where GPUs idle during pipeline fill and drain phases because too few microbatches (\(m\)) are scheduled relative to the stage depth (\(p\)). To recover throughput, the team can: (1) increase the microbatch count \(m\) (using 1F1B scheduling to keep peak activation memory bounded), and (2) reduce the pipeline stage count \(p\) by switching intra-node GPUs to tensor parallelism.
Learning Objective: Analyze pipeline bubble underutilization and propose architectural remedies
**Order the following parallelism strategies by their structural granularity from coarsest (whole model / batch level) to finest (intra-operator level):
- Tensor Parallelism (Megatron column/row splits)
- Standard Data Parallelism (batch dimension splitting)
- Pipeline Parallelism (inter-layer stage splitting)**
Answer: (2) -> (3) -> (1). First, (2) Standard Data Parallelism operates at the coarsest granularity, replicating the entire model and partitioning only the outer batch loop. Next, (3) Pipeline Parallelism partitions at the inter-layer boundary (assigning sequential layer groups to stages). Finally, (1) Tensor Parallelism operates at the finest intra-operator granularity, partitioning individual weight matrices within each layer.
Learning Objective: Rank distributed parallelism strategies by their structural decomposition granularity
Self-Check: Answer
In distributed training framework runtimes (such as PyTorch DDP), what mechanism combines multiple small layer gradient tensors into contiguous memory buffers before launching collective communication?
- Activation checkpointing
- ZeRO Stage 3 parameter gathering
- Temporal multiplexing
- Gradient bucketing
Answer: The correct answer is D. Gradient bucketing groups individual layer gradient tensors into larger contiguous memory buffers (typically 25–50 MB buckets) during the backward pass. Launching an AllReduce on a single aggregated bucket amortizes collective invocation latency and saturates network link bandwidth far more effectively than issuing hundreds of tiny per-tensor transfers. Activation checkpointing recomputes activations to save memory. ZeRO-3 parameter gathering reconstructs sharded weights on demand. Temporal multiplexing alternates execution modes in multi-model RLHF.
Learning Objective: Explain the role of gradient bucketing in amortizing collective communication latency
Contrast the collective communication primitive invocation frequency of standard DDP against full-shard FSDP with resharding for a model with \(N_L\) layers during a single training step.
Answer: Standard DDP executes a single AllReduce collective (or a few bucketed AllReduces) at the end of the backward pass. In contrast, full-shard FSDP with resharding executes approximately \(3N_L\) collective operations per step: an AllGather before each layer’s forward pass, a second AllGather before each layer’s backward pass, and a ReduceScatter after each layer’s backward computation.
Learning Objective: Compare collective communication primitive invocations between standard DDP and full-shard FSDP
Explain why framework optimizations such as CUDA graphs and asynchronous communication streams cannot eliminate the physical scaling penalty of distributed training as the cluster size \(N\) becomes very large.
Answer: While framework optimizations eliminate CPU launch overheads and overlap non-dependent transfers, they cannot change the underlying communication-computation ratio \(\rho = T_{\text{comm}}(N) / (T_{\text{compute}}/N)\). As \(N \to \infty\), per-worker computation time shrinks toward zero while collective communication volume and physical interconnect latency remain non-zero, making network transfer the dominant term in step time and capping speedup according to Amdahl’s Law.
Learning Objective: Evaluate the physical limits of framework-level communication optimizations
Self-Check: Answer
A team achieves 50 percent Model Flops Utilization (MFU) on a single node, scales to 1,024 GPUs, and measures a scaling efficiency \((\eta_{\text{scaling}})\) of 50 percent on their communication-heavy workload. What is the cluster’s realized ‘useful goodput’ as a fraction of peak hardware compute?
- 25 percent of peak compute (delivering 256 GPU-equivalents of useful work)
- 50 percent of peak compute (delivering 512 GPU-equivalents of useful work)
- 100 percent of peak compute (delivering 1,024 GPU-equivalents of useful work)
- 12.5 percent of peak compute (delivering 128 GPU-equivalents of useful work)
Answer: The correct answer is A. Realized useful goodput is the multiplicative product of hardware compute efficiency (MFU) and cluster scaling efficiency: \(\text{Goodput} = \text{MFU} \times \eta_{\text{scaling}} = 0.50 \times 0.50 = 0.25\) (25% of peak, or \(0.25 \times 1024 = 256\) GPU-equivalents). Optimizing MFU in isolation ignores communication overhead and results in a \(2\times\) overestimation of delivered cluster capacity. Claiming 50% ignores scaling efficiency loss. Claiming 100% assumes perfect linear scaling and zero hardware overhead. Claiming 12.5% introduces an extra ungrounded halving.
Learning Objective: Calculate cluster goodput by composing hardware MFU with scaling efficiency
Using the Young-Daly checkpoint formula \(\tau_{\text{opt}} = \sqrt{2 \cdot T_{\text{write}} \cdot \text{MTBF}_{\text{system}}}\), explain why arbitrarily choosing a fixed 15-minute checkpoint cadence on a 1,024-GPU cluster with a 48-hour system MTBF and 5-minute write time causes severe compute waste.
Answer: With \(T_{\text{write}} = 5\text{ min} = 0.0833\text{ hr}\) and \(\text{MTBF}_{\text{system}} = 48\text{ hr}\), the optimal interval is \(\tau_{\text{opt}} = \sqrt{2 \times 0.0833 \times 48} \approx 2.83\text{ hours}\) (~170 minutes), incurring about 6% total checkpoint-plus-rework overhead. Checkpointing every 15 minutes wastes \(5 / 15 = 33.3\%\) of all cluster compute solely on write overhead, losing thousands of dollars per day compared to the Young-Daly optimum.
Learning Objective: Apply the Young-Daly checkpoint formula to calculate optimal checkpoint intervals and evaluate over-checkpointing losses
Why is gradient accumulation NOT a cost-free substitute for true physical scale-out even though it reduces gradient AllReduce communication frequency by a factor of \(K\)?
- It is mathematically incompatible with the Adam optimizer
- It requires keeping live gradient buffers resident in HBM, increases optimizer update latency by \(K\times\), and risks FP16 numerical overflow when summing hundreds of gradient tensors
- It requires dedicated InfiniBand links between CPU and GPU for each microstep
- It forces batch normalization layers to synchronize across all nodes on every forward pass
Answer: The correct answer is B. Gradient accumulation requires keeping a live gradient buffer resident in HBM throughout the accumulation window, delays parameter updates by \(K\) forward/backward steps (slowing convergence dynamics), and risks numerical overflow when repeatedly accumulating unscaled FP16 values in early training. Gradient accumulation is fully compatible with Adam. It does not require special CPU-GPU InfiniBand links. It does not force cross-node batch normalization synchronization.
Learning Objective: Analyze the memory, latency, and numerical stability trade-offs of gradient accumulation
The optimal interval formula \(\tau_{\text{opt}} = \sqrt{2 \cdot T_{\text{write}} \cdot \text{MTBF}_{\text{system}}}\), which balances checkpoint write overhead against rework risk following hardware failure, is known as the ____.
Answer: Young-Daly Checkpoint Law (or Young-Daly formula / Young-Daly law). The Young-Daly law determines the mathematically optimal checkpointing cadence to minimize total wasted compute in large-scale distributed clusters.
Learning Objective: Define the principle governing optimal checkpointing intervals in distributed clusters
True or False: If a 7B-parameter model fits comfortably in single-GPU memory with high compute utilization under standard DDP, universally enabling ZeRO-3 / FSDP full sharding will always increase training throughput by reducing per-GPU memory consumption.
Answer: False. When model state and activations already fit in GPU memory, the memory reduction from FSDP provides no execution benefit. Instead, FSDP introduces 10–25% communication overhead by placing AllGather and ReduceScatter collectives on the critical path of every layer, causing overall training throughput to decrease (e.g. dropping from 145 to 118 samples/second).
Learning Objective: Evaluate the performance pitfall of applying FSDP when memory capacity is not binding
Self-Check: Answer
What fundamental architectural principle summarizes the trade-offs of all distributed training strategies in this chapter?
- Amdahl’s law can be eliminated entirely by combining tensor and pipeline parallelism
- Data parallelism scales indefinitely as long as Ring AllReduce is selected
- Model parameters can be replicated infinitely without consuming network bandwidth
- The displacement of overhead: distributed parallelism never eliminates total work, but only relocates the execution tax among memory footprint, communication volume, synchronization latency, and pipeline idle time
Answer: The correct answer is D. No parallelism strategy reduces the total mathematical work of training; each strategy only relocates overhead to different dimensions of the system (converting memory capacity limits into network communication volume, synchronization delays, or pipeline bubbles). The winning strategy is the one that shifts overhead to the least binding resource tier in the cluster. Amdahl’s law remains a hard mathematical bound. Data parallelism is bounded by critical batch size and network bisection bandwidth. Parameter replication requires gradient synchronization across workers.
Learning Objective: Synthesize the principle of displacement of overhead across distributed training strategies
Contrast the primary partitioning strategy and binding hardware bottleneck for Archetype A (Dense Large Language Models like GPT-4/Llama-3) versus Archetype B (Large-Scale Recommendation Models / DLRM).
Answer: Archetype A (dense LLMs) uses Hybrid 3D Parallelism (Tensor, Pipeline, and Data Parallelism) and is bound by GPU compute throughput and high-bandwidth interconnects (NVLink and InfiniBand). Archetype B (DLRM) uses Embedding Sharding across Parameter Server architectures and is bound by aggregate memory capacity (terabytes of DRAM/HBM) and random-access memory latency for sparse lookups rather than dense compute.
Learning Objective: Compare distributed training archetypes by their primary partitioning strategy and binding hardware bottleneck
Explain why data, pipeline, and tensor parallelism are treated as orthogonal scaling dimensions rather than mutually exclusive strategies in the 3D Parallelism Cube, and provide an example configuration that combines them.
Answer: Data, pipeline, and tensor parallelism partition different dimensions of the workload (batch dimension, model depth across layers, and intra-layer tensor operations, respectively). Because they address independent bottlenecks (throughput, depth-induced memory, and operator-level memory), they compose multiplicatively (\(N_{\text{total}} = d \times p \times t\)). For example, a frontier training run may use \(\text{TP}=8\) within NVLink nodes, \(\text{PP}=16\) across nodes over InfiniBand, and \(\text{DP}=128\) across the cluster to balance memory, latency, and global batch throughput.
Learning Objective: Evaluate the orthogonal composability of the 3D parallelism cube axes