Rules of Thumb

The C\(^3\) Taxonomy

In a distributed training cluster, “it is slow” is even less informative than on a single machine. A job can stall because accelerators are underutilized (Compute), gradient synchronization saturates interconnects (Communication), or checkpointing and recovery delay progress (Coordination). This appendix extends single-node diagnostics to fleet-scale infrastructure, providing the C\(^3\) scorecard used throughout the book to isolate the binding bottleneck across distributed systems.

How to Use This Appendix

This appendix is designed as a reference. Start with the diagnostic summary table, form a hypothesis about which C\(^3\) axis dominates, and then pick the tool that can confirm (or falsify) that hypothesis.

When training throughput is low, check model FLOPs utilization (MFU), communication fraction, and goodput ratio, then map each to its compute, communication, or coordination axis. When scaling efficiency drops below expectations, use the fleet law decomposition to identify which term grew. When cost is exploding, use the C\(^3\) scorecard to ensure that effort targets the dominant term, not a nonbottleneck.

The C\(^3\) Taxonomy is the diagnostic framework for fleet-scale ML systems engineering. Where the single-machine foundations (as introduced in The D·A·M Taxonomy) diagnose bottlenecks within a single node—data starvation, algorithmic overhead, or hardware saturation—the C\(^3\) taxonomy diagnoses bottlenecks across the distributed fleet. Most fleet-scale performance problems can be diagnosed by identifying the dominant axis: Compute (are the accelerators doing useful math?), Communication (is the network moving data fast enough?), or Coordination (is the system spending too much time on synchronization, failure recovery, and scheduling?). Many production bottlenecks sit at intersections between these axes.

From D·A·M to C\(^3\)

The C\(^3\) taxonomy does not replace D·A·M—it extends it. When a workload moves from one machine to a fleet, each D·A·M axis acquires new failure modes that the single-machine framework cannot capture. Table 1 shows how the transition works.

Table 1: Mapping D·A·M to C\(^3\): How single-machine diagnostic concerns generalize to fleet-scale challenges.

D·A·M Axis	Single-Machine Concern	C\(^3\) Extension	What Changes at Fleet Scale
Data	Host disk \(\to\) GPU memory pipeline	Communication	Data movement crosses network fabrics; AllReduce, AllGather, and pipeline transfers dominate
Algorithm	Operator choice, autograd graph	Compute/Coordination	Workload must be partitioned (\(N\)-way sharding); communication-to-compute ratio determines scaling
Machine	SM occupancy, HBM bandwidth	Coordination	Single-device faults become cluster-wide stragglers; failure recovery, scheduling, and power density constrain \(N\)

The most important row in table 1 is the last one. On a single machine, the overhead term (\(L_{\text{lat}}\)) in the iron law is typically small—kernel launch latency, Python dispatch, synchronization barriers. At fleet scale, coordination becomes an axis in its own right: checkpoint writes, failure detection and recovery, pipeline bubble overhead, scheduler preemptions, and maintenance windows collectively consume a significant fraction of wall time. Coordination is the axis that this book exists to address.

Diagnostic Summary

With the mapping in place, diagnosis becomes a matter of matching symptoms to the axis that constrains the fleet. Table 2 provides the main reference table for fleet-scale diagnosis. Each C\(^3\) axis maps to a physical constraint, observable symptoms, measurable metrics, and engineering levers.

Table 2: C\(^3\) Diagnostic Summary: Each axis maps to a distinct physical constraint and a high-leverage optimization strategy. Start diagnosis here: identify which constraint binds, then follow the optimization pointer to the relevant chapter.

C\(^3\) Axis	Physical Constraint	Symptoms	Key Metric	High-Leverage Optimization
Compute (\(C_1\))	Arithmetic throughput (\(R_{\text{peak}} \times \eta_{\text{hw}}\))	Low MFU, GPU utilization below 80%, poor per-GPU performance	MFU (Model FLOPs Utilization)	Kernel optimization, mixed precision, operator fusion (Performance Engineering)
Communication (\(C_2\))	Network bandwidth (\(\text{BW}_{\text{net}}\))	High AllReduce time, low scaling efficiency, communication > 30% of step	Scaling efficiency (\(\eta_{\text{scaling}}\)), communication fraction (\(T_{\text{comm}}(N)/T_{\text{step}}(N)\))	Gradient compression, overlap compute/communication, topology optimization (Collective Communication)
Coordination (\(C_3\))	Synchronization overhead and failure recovery	Low goodput ratio, frequent restarts, large pipeline bubbles, scheduler churn	Goodput ratio (\(T_{\text{useful}}/T_{\text{wall}}\))	Async checkpointing, elastic training, faster failure detection (Fault Tolerance)

The Fleet Law

The same classification can be written as a time budget for each distributed step. The fleet law, introduced in The C^3 Taxonomy: Foundations of Scale, decomposes every distributed training step into the distributed-step time budget:

\[ T_{\text{step}}(N) = \frac{T_{\text{compute}}}{N} + T_{\text{comm}}(N) + T_{\text{sync}}(N) - T_{\text{overlap}} \]

This equation is the fleet-scale counterpart of the iron law. Where the iron law decomposes single-machine execution into data movement, compute, and overhead, the fleet law decomposes distributed execution into local arithmetic, network data transfer, and synchronization logic. The diagnostic strategy is identical: measure each term, identify which dominates, and direct engineering effort at the dominant term.

Component decomposition

Each fleet law term maps to specific measurable activities:

• \(T_{\text{compute}}/N\): Forward pass, backward pass, optimizer step—all local arithmetic after distribution across \(N\) devices. Governed by MFU and per-GPU kernel efficiency. Improvements come from better kernels, mixed precision, and operator fusion.
• \(T_{\text{comm}}(N)\): AllReduce of gradients, AllGather of parameters (in FSDP/ZeRO), activation transfers in tensor/pipeline parallelism. Governed by network bandwidth and collective algorithm choice. Improvements come from gradient compression, hierarchical collectives, and compute-communication overlap.
• \(T_{\text{sync}}(N)\): Synchronization barriers, checkpoint writes, failure detection and recovery, pipeline bubble idle time, scheduler preemptions, and maintenance windows. Governed by cluster reliability and orchestration software. Improvements come from asynchronous checkpointing, elastic training, and faster failure detection.
• \(T_{\text{overlap}}\): Communication or coordination time hidden behind useful arithmetic. Governed by scheduling and implementation overlap.

The fleet’s efficiency follows directly:

\[ f_{\text{compute}} = \frac{T_{\text{compute}}/N}{T_{\text{step}}(N)} \]

When \(f_{\text{compute}}\) drops below 0.5, the fleet spends more time on communication and coordination than on useful arithmetic. This compute-time fraction is not total fleet efficiency; useful fleet efficiency also depends on MFU, scaling efficiency, and goodput. The C\(^3\) taxonomy identifies which noncompute term is responsible.

Intersection Landscape

Like D·A·M, the C\(^3\) axes interact at their boundaries. Production bottlenecks often sit at an intersection where two axes compound.

Compute \(\cap\) communication

This intersection governs whether the system can hide communication behind computation. The communication-computation ratio (\(\rho = T_{\text{comm}}(N)/(T_{\text{compute}}/N)\)) is the key metric (The communication-computation ratio). When \(\rho < 1\), computation takes longer than communication and the network transfer can be overlapped—the system is compute-bound and healthy. When \(\rho > 1\), GPUs finish their local work before the network delivers the next round of data, and the system is communication-bound.

Engineering at this intersection focuses on overlap strategies: launching AllReduce during the backward pass, using CUDA streams to pipeline local computation with network transfers, and increasing the computation per synchronization point (larger microbatches, gradient accumulation). Distributed Training and Collective Communication cover these techniques in depth.

Communication \(\cap\) coordination

This intersection captures the synchronization cost embedded in communication. Every AllReduce is both a data transfer (communication) and a synchronization barrier (coordination)—all participants must reach the barrier before any can proceed. The cost of stragglers manifests here: if one GPU is 10 percent slower, every other GPU waits, converting a communication operation into a coordination bottleneck.

Engineering at this intersection focuses on reducing barrier sensitivity: asynchronous gradient methods that decouple communication from synchronization, hierarchical AllReduce that limits the blast radius of stragglers, and straggler detection with proactive mitigation. Fault Tolerance addresses straggler management.

Compute \(\cap\) coordination

This intersection captures the idle compute caused by coordination overhead. Pipeline bubbles are the canonical example: during warmup and cooldown phases of pipeline parallelism, some stages are idle while others compute. Checkpoint writes that block the training loop convert coordination overhead into wasted compute capacity. Failure recovery that requires rolling back and recomputing work transforms a coordination event into a computation penalty.

Engineering at this intersection focuses on minimizing idle time: increasing microbatches to shrink the pipeline bubble fraction, using asynchronous checkpointing to overlap writes with compute, and reducing the blast radius of failures so that recomputation is bounded. Distributed Training covers pipeline scheduling; Fault Tolerance covers recovery strategies.

In the middle of a production incident, fast heuristics narrow the search space before a profiler is needed. These thresholds provide that first line of defense.

The C\(^3\) traffic light

Table 3 provides threshold-based triage for each C\(^3\) axis.

Table 3: C\(^3\) Traffic Light: Quick triage thresholds for fleet-scale diagnosis. Green means the axis is healthy; yellow means it deserves investigation; red means it is the likely bottleneck. These thresholds assume well-optimized large-model training on current-generation hardware.

C\(^3\) Axis	Green (Healthy)	Yellow (Investigate)	Red (Bottleneck)
Compute	MFU \(>\) 50%	MFU 30%–50%	MFU \(<\) 30%
Communication	Comm fraction \(<\) 20%	Comm fraction 20–40%	Comm fraction \(>\) 40%
Coordination	Goodput ratio \(>\) 90%	Goodput ratio 75–90%	Goodput ratio \(<\) 75%

The bottleneck diagnostic table

Once the bottleneck axis is identified, table 4 shows which optimizations help and which ones are wasted.

Table 4: What Works vs. What Is Wasted at Fleet Scale: Optimizing a nondominant C\(^3\) axis yields limited improvement and often worsens cost efficiency. A communication-bound fleet will gain little from faster GPUs until AllReduce and other communication bottlenecks are addressed.

If the fleet is…	Dominant Term	Optimization That Works	Optimization That is Wasted
Compute-bound	\(T_{\text{compute}}/N\)	Better kernels, mixed precision, operator fusion, next-gen accelerators	More network bandwidth (GPUs are not waiting on the network)
Communication-bound	\(T_{\text{comm}}(N)\)	Gradient compression, compute-comm overlap, hierarchical collectives, InfiniBand upgrade	Faster GPUs (they will just idle faster while waiting for the network)
Coordination-bound	\(T_{\text{sync}}(N)\)	Async checkpointing, elastic training, faster failure detection, fewer pipeline stages	Neither faster GPUs nor faster network (the time is lost to overhead, not to data movement or arithmetic)

C\(^3\) Case Studies

Theoretical constraints manifest as confusing symptoms in production. These scenarios illustrate how to apply the C\(^3\) taxonomy to fleet-scale performance problems. Each case isolates one dominant axis before showing which optimization levers follow from that diagnosis.

Case 1: The underutilized fleet (Compute)

Symptom

An engineering team provisions 4,096 H100 GPUs for a large language model training run. The training loop runs without errors, but the PyTorch Profiler shows MFU of only 15 percent. The network profiler shows communication accounts for less than 10 percent of step time. The cluster is running but barely working.

Diagnosis

The Compute axis is the bottleneck. With 15 percent MFU, 85 percent of the fleet’s peak capacity is not credited to useful model FLOPs. This is not a communication or coordination problem—the network is fast enough and the system is stable. The system is not feeding work to the GPUs efficiently.

The fix

This is a per-GPU efficiency problem that happens to be multiplied across 4,096 accelerators. Target the compute axis:

• Mixed precision: Ensure BF16/FP8 Tensor Cores are engaged. A common culprit is FP32 fallback in normalization layers or loss computation.
• Operator fusion: Use torch.compile or similar just-in-time (JIT) compilation to fuse element-wise operations and reduce kernel launch overhead.
• Batch size tuning: If per-GPU batch size is too small, the matrix multiplications have insufficient arithmetic intensity to saturate the Tensor Cores.

Raising MFU from 15 percent to 50 percent on the same hardware delivers 50 percent/15 percent = 3.3× more useful work—the equivalent of tripling the fleet without buying a single GPU.

Case 2: The communication wall (Communication)

Symptom

A 512-GPU data-parallel training run achieves 45 percent illustrative compute-phase efficiency during compute kernels on each GPU—good per-device kernel efficiency. Wall-clock MFU over the full step (including AllReduce) is only 20.2 percent because AllReduce consumes 55 percent of every training step. Scaling from 64 GPUs to 512 GPUs yields only 4× speedup instead of the expected 8×, or 50 percent scaling efficiency.

Diagnosis

The Communication axis dominates. Each GPU has healthy compute-phase efficiency, but more than half the step time is spent synchronizing gradients across the network. The system is Communication-Bound: adding more GPUs will make it worse, not better, because AllReduce time grows with participant count while per-GPU computation stays constant.

The fix

Target the communication axis without touching the per-GPU computation:

• Compute-communication overlap: Launch AllReduce during the backward pass rather than waiting until it completes. Modern frameworks (FSDP, DeepSpeed) support this natively.
• Gradient compression: Apply TopK sparsification or quantization to reduce the bytes crossing the network by 10–100\(\times\).
• Hierarchical collectives: Use intra-node NVLink for the first reduction stage, then inter-node InfiniBand only for cross-node aggregation, reducing cross-node traffic by 8\(\times\).

If communication were eliminated entirely, throughput would increase by 2.2× (Amdahl’s Law applied to the 45 percent compute fraction). Realistically, reducing communication from 55 percent to 20 percent of step time would recover most of the lost scaling.

Case 3: The coordination tax (Coordination)

Symptom

A 10,000-GPU training run shows 40 percent MFU per device and communication accounts for only 15 percent of step time—both healthy. The job’s Goodput Ratio (useful training steps per second divided by the ideal or allocated step rate), however, is only 60 percent. The remaining 40 percent of wall time is consumed by checkpoint writes, failure recovery restarts, pipeline bubble idle time, scheduler preemptions (17 percent of wall time), and maintenance windows.

Diagnosis

The Coordination axis dominates. Per-GPU computation and inter-node communication are both efficient, but 40 percent of wall time is consumed by nonproductive overhead: 10 percent failure recovery (at 10,000 GPUs, GPU failures occur about every 5 hours), 5 percent pipeline bubbles, 3 percent checkpoint writes, and 5 percent maintenance windows. Neither faster GPUs nor faster networks will help—coordination, not computation or communication, consumes the time.

The fix

Target the coordination axis:

• Asynchronous checkpointing: Overlap checkpoint writes with the next training step, reducing visible checkpoint overhead from 3 percent to near zero.
• Elastic training: When a node fails, shrink the job and continue rather than halting all 10,000 GPUs for recovery. This converts the 10 percent failure recovery cost into a smaller throughput reduction.
• Pipeline schedule optimization: Switch from GPipe to an interleaved 1F1B schedule to reduce bubble fraction, or increase microbatch count per pipeline flush.
• Faster failure detection: Reduce heartbeat timeout from 30 seconds to 5 seconds with hardware-level health monitoring, cutting the idle time between failure occurrence and recovery initiation.

Production Troubleshooting

Table 5 provides a diagnostic matrix for common fleet-scale failure modes.

Table 5: C\(^3\) Troubleshooting Matrix: Root cause identification and remediation for common fleet-scale bottlenecks. Each row connects a user-visible symptom to the C\(^3\) axis most likely responsible, reducing the search space before reaching for a profiler.

Symptom	C\(^3\) Axis	Diagnostic Question	Measurement	Action
Low MFU despite fast network	Compute	Are Tensor Cores engaged? Is batch size sufficient for arithmetic intensity?	Per-GPU kernel trace (Nsight/PyTorch Profiler)	Enable mixed precision, increase per-GPU batch size
Throughput plateaus when adding GPUs	Communication	Does AllReduce time grow faster than computation shrinks?	NCCL trace, \(\rho\) ratio	Gradient compression, hierarchical collectives, overlap
Frequent job restarts	Coordination	What is the cluster MTBF? Is detection fast enough?	Failure logs, MTBF calculation	Elastic training, faster detection, smaller blast radius
High GPU-hours but slow progress	Coordination	What fraction of GPU-hours produce useful training steps?	Goodput ratio (\(T_{\text{useful}}/T_{\text{wall}}\))	Async checkpointing, reduce pipeline stages, eliminate scheduler churn
Scaling efficiency drops with cluster size	Comm/Coord	Is the bottleneck network bandwidth or synchronization barriers?	Separate \(T_{\text{comm}}(N)\) from \(T_{\text{sync}}(N)\)	If comm: compress or overlap. If coord: async methods
Stragglers slow entire job	Comm \(\cap\) Coord	Is one node consistently last to reach the AllReduce barrier?	Per-node step time histogram	Straggler detection + replacement, bounded staleness, backup workers

Tooling Map

Engineers must measure abstract C\(^3\) axes with concrete profiling tools. Table 6 maps each axis to the utilities that confirm or falsify a hypothesis.

Table 6: C\(^3\) Tooling Map: Profiling utilities for diagnosing fleet-scale bottlenecks. Start with the primary tool for quick triage; use secondary tools for deep-dive analysis. Compute tools operate per-GPU; Communication tools operate at the network layer; Coordination tools operate at the cluster/job level.

C\(^3\) Axis	Key Metric	Primary Tool	Secondary Tool
Compute	MFU, kernel utilization	PyTorch Profiler (TensorBoard plugin)	Nsight Compute (per-kernel roofline analysis)
Communication	AllReduce time, \(\rho\) ratio	NCCL debug logs (NCCL_DEBUG=INFO)	Nsight Systems (timeline), ibstat/perfquery (IB)
Coordination	Goodput ratio, restart count	Cluster scheduler logs (Slurm, K8s event logs)	Custom goodput dashboards (for example, Google ML Goodput)

C\(^3\) Scorecard

The C\(^3\) Scorecard grades fleet efficiency against known thresholds, extending the single-machine scorecard (The D·A·M Taxonomy) to the distributed environment. Table 7 defines the three metrics that characterize fleet health.

Table 7: The C\(^3\) Efficiency Rubric: Use these three numbers to characterize fleet health. The MFU denominator uses aggregate fleet peak, written as \(N R_{\text{peak,device}}\) when \(R_{\text{peak}}\) is a per-device value. A fleet that passes all three thresholds has exhausted its easy optimizations; further gains require architectural changes, hardware upgrades, or larger problem sizes to improve the scaling regime.

C\(^3\) Axis	Metric	Definition	Failing Grade	Passing Grade
Compute	MFU	\(\frac{O_{\text{step}}}{N R_{\text{peak,device}} \times T_{\text{step}}}\)	\(<\) 30%	\(>\) 50%
Communication	Scaling Efficiency (\(\eta_{\text{scaling}}\))	\(\frac{T_1}{N \times T_N}\)	\(<\) 35%	\(>\) 70%
Coordination	Goodput Ratio	\(\frac{T_{\text{useful}}}{T_{\text{wall}}}\) or \(\frac{\text{useful steps/sec}}{\text{ideal or allocated steps/sec}}\)	\(<\) 75%	\(>\) 90%

C³ Infrastructure Diagnostic Decision Tree & Fleet D·A·M Scorecard

To bridge hardware-level telemetry and architectural optimization, cluster operations teams require a systematic procedure for mapping low-level hardware signals to the binding C\(^3\) bottleneck wall.

Telemetry signal mapping to C\(^3\) walls

Modern GPU clusters generate dense telemetry across silicon, PCIe buses, thermal sensors, and network switches:

• MFU Drops: Sudden or sustained drops in model FLOPs utilization below expected baselines.
• PCIe AER (Advanced Error Reporting) Retries: Non-fatal bus errors causing packet retries, link downgrades (e.g., PCIe Gen 5 \(\to\) Gen 3), or host-to-device direct memory access (DMA) throttling.
• Thermal Throttling: GPU SM clock downclocking (e.g., 1.9 GHz \(\to\) 1.1 GHz) triggered by junction thermal limits or cooling fan degradation.
• InfiniBand Credit Stalls & PFC Pause Frames: Priority flow control (PFC) pause frames, congestion notification packets (CNP), or switch buffer credit starvation.

Diagnostic decision tree

The hardware-first diagnostic tree below guides triage from initial telemetry signals to the underlying C\(^3\) wall and remediation action.

                    [Telemetry Signal/Performance]
                         [Anomaly Detected]
                                  |

                 /----------------+----------------\
                 v                                 v
       [System Telemetry]                  [Network Telemetry]
  [MFU drops, Thermal, AER]          [PFC Pauses, Credit Stalls]
                 |                                 |

          /------+------\                   /------+------\
          v             v                   v             v
     [Thermal]       [PCIe]            [IB Credit]      [High]
       [Temp]         [AER]              [Stalls]       [Tail]
      [> 85°C]      [Retries]             [> 5%]      [Latency]
          |             |                   |             |

          v             v                   v             v
     [COMPUTE]      [COMPUTE]            [COMM.]       [COORD.]
       [WALL]         [WALL]              [WALL]         [WALL]

      (Thermal       (PCIe Bus           (Network      (Straggler
     Throttling)    Degradation)         Congestion)     Barrier)

Structured diagnostic telemetry table

Table 8 maps hardware-level telemetry counters to root causes, C\(^3\) bottleneck walls, and concrete operational remediations.

Table 8: C\(^3\) Telemetry Diagnostic Mapping: Direct mapping from silicon-level telemetry signals to distributed bottleneck walls and remediation procedures.

Telemetry Signal	Hardware/OS Counter	Root Cause	C\(^3\) Bottleneck Wall	Remediation Action
MFU Drop with High GPU Temp	gpu_temperature > 85°C, clocks_throttle_reasons.sw_thermal	Thermal throttling reduces GPU SM clock speed	Compute Wall	Check coolant flow, replace thermal interface material (TIM), adjust fan curves
MFU Drop with Low Host-GPU BW	pcie_aer_correctable_errors, pcie_link_width < x16	PCIe link degradation/retries due to signal noise	Compute Wall	Reseat GPU/PCIe riser card, clean PCIe edge connectors, replace cable
AllReduce Stalls with Pause Frames	PFC_PAUSE_XOFF_FRAMES, infiniband_credit_stalls	Network fabric congestion, IB buffer exhaustion	Communication Wall	Enable Adaptive Routing (AR), rebalance ECMP hashing, tune NCCL buffer size
NCCL Barrier Timeout/Hang	NCCL_WARN=INFO log Unhandled CUDA error, heartbeat_miss	Single-node straggler or silent kernel hang	Coordination Wall	Drain degraded node, isolate via node health check script, trigger fast restart
High Step Overhead with Idle GPU	nvlink_throughput == 0, fs_write_latency > 1s	Blocking synchronous checkpoint write to storage	Coordination Wall	Switch to asynchronous double-buffered checkpointing or host RAM staging

Fleet D·A·M scorecard

The Fleet D·A·M Scorecard in table 9 generalizes single-node Data, Algorithm, and Machine (D·A·M) diagnostics to fleet-scale C\(^3\) bottlenecks.

Table 9: Fleet D·A·M Scorecard: Quantifies how single-node hardware telemetry maps into fleet-level C\(^3\) execution constraints.

D·A·M Axis	Fleet Telemetry Signal	C\(^3\) Bottleneck Wall	Target Efficiency Benchmark
Data (\(D\))	Network fabric RX/TX saturation, storage write I/O wait times	Communication/Coordination	I/O wait time \(< 2\%\) of step time; NCCL ring bandwidth \(> 85\%\) of fabric peak
Algorithm (\(A\))	Model FLOPs utilization (MFU), pipeline bubble idle fraction	Compute/Coordination	MFU \(> 60\%\); Pipeline bubble fraction \(< \frac{p-1}{2K}\)
Machine (\(M\))	PCIe AER errors, thermal throttling counters, GPU power limit throttling	Compute	Thermal throttle events \(= 0\); PCIe link width held at max (Gen 5 x16)

Scaling Laws Through the C\(^3\) Lens

Scaling laws are usually written in algorithmic FLOPs, but a training fleet delivers only the useful FLOPs that survive utilization, communication, and coordination losses. Scaling-law targets translate directly into C\(^3\) terms: the hidden perfect-systems assumption abstracts away losses, while effective FLOP/s defines the usable throughput connecting model-quality forecasts to real cluster capacity.

Why scaling laws assume perfect C\(^3\)

Scaling laws—Kaplan, Chinchilla, and their successors—predict model quality as a function of algorithmic training compute. They usually abstract away systems efficiency: wall-clock time, accelerator peak FLOP/s, compute-phase efficiency, communication overhead, and scheduler losses enter later when teams provision hardware to deliver the target compute budget. In C\(^3\) terms, scaling-law FLOPs must be converted into raw fleet capacity after compute-phase, distributed-step, and operational-goodput losses.

The gap between scaling-law predictions and observed training outcomes is, in large part, a C\(^3\) gap. A team that budgets \(10^{24}\) FLOPs for training will actually deliver far fewer effective FLOPs to the model, because each FLOP must survive three multiplicative losses: per-GPU compute-phase efficiency, inter-node scaling efficiency (\(\eta_{\text{scaling}}\)), and operational goodput.

The effective FLOP/s concept

A fleet’s Effective FLOP/s is the usable throughput after compounding three disjoint C\(^3\) efficiency factors:

\[\text{Effective} = \text{Peak} \times \underbrace{\eta_{\text{compute-phase}}}_{\text{Compute}} \times \underbrace{\eta_{\text{scaling}}}_{\text{Distributed step}} \times \underbrace{\eta_{\text{operational}}}_{\text{Operational goodput}}\]

The compute-phase factor measures useful model arithmetic against peak throughput only while local computation is active. Scaling efficiency captures communication and other distributed-step losses, including pipeline bubbles. Operational goodput captures time outside those measured steps, including checkpointing, failure recovery, scheduler preemptions, and maintenance. These disjoint measurement windows prevent the same loss from being counted twice.

A concrete estimate makes the multiplicative loss visible at fleet scale.

Systems Perspective 1.1: The C³ tax on a 100,000-GPU cluster

Consider a 100,000-GPU H100 cluster with 98,900 PFLOP/s of peak aggregate throughput. After the three disjoint C\(^3\) efficiency factors:

\[\text{Effective} = 98,900 PFLOP/s \times 0.50 \times 0.35 \times 0.65 \approx 11249.9 PFLOP/s\]

The fleet delivers 11.4 percent of its peak capacity as useful training work. The C\(^3\) Tax—the ratio of peak to effective—is 8.8×: achieving a given effective compute budget requires 8.8× the raw hardware. The compute phase retains a 50 percent efficiency factor. The distributed step retains a 35 percent scaling factor, using the 8,192-GPU reference as an illustrative proxy that includes pipeline losses. Operations retain a 65 percent goodput factor after checkpoints, failures, scheduler preemptions, and maintenance.

These losses reflect the physics of fleet-scale computation rather than a failure of engineering. The C\(^3\) taxonomy quantifies where the losses occur so that optimization effort targets the dominant term.

Summary

The C\(^3\) taxonomy provides a systematic framework for diagnosing fleet-scale bottlenecks. Each axis maps to a distinct physical constraint: arithmetic throughput and MFU bound compute; network bandwidth and collective algorithm efficiency bound communication; and synchronization overhead, failure recovery, and operational losses bound coordination. The fleet law quantifies these constraints, enabling systematic diagnosis. Use the C\(^3\) traffic light for quick triage, the bottleneck diagnostic table to choose the right lever, and the C\(^3\) scorecard to grade fleet health.

Key Takeaways: Where to look first at fleet scale

• Every fleet-scale bottleneck has a dominant C\(^3\) axis: Compute, Communication, or Coordination, with many real bottlenecks spanning intersections. Identify the dominant axis before optimizing.
• Measure the C\(^3\) scorecard: Use MFU \(>\) 50 percent, scaling efficiency \(>\) 70 percent, and goodput ratio \(>\) 90 percent before investing in optimizations.
• The C\(^3\) tax is multiplicative: Peak FLOP/s \(\times\) compute-phase efficiency \(\times\) scaling efficiency \(\times\) operational goodput = effective FLOP/s. At 100,000 GPUs, expect only ~11.4 percent of peak.
• Coordination is the new axis: On a single machine, overhead is negligible. At fleet scale, checkpoints, failures, pipeline bubbles, and scheduling consume 40 percent or more of wall time.
• Optimizing the wrong C\(^3\) axis yields limited improvement: Faster GPUs cannot fix a communication-bound fleet by themselves; faster networks cannot fix coordination overhead.