One model, six domains, twelve walls: a complete systems analysis in 60 seconds.
capstone
advanced
Compose 6+ solvers across all six taxonomy domains to produce a holistic training analysis across memory capacity, compute, communication, reliability, and sustainability walls.
The Question
What does a complete systems analysis look like? No single solver captures the full picture. Training a 71B-parameter model (Llama-3.1-70B) on 512 NVIDIA H100 GPUs involves compute walls, memory walls, communication overhead, checkpoint I/O, energy costs, and carbon emissions, all simultaneously. This tutorial traces all six taxonomy domains and exercises 12 of the 22 systems walls through a single workload.
Compose six solver families across all taxonomy domains into a holistic analysis
Identify which of the 22 systems walls bind for a real training workload
Quantify the hidden costs: checkpoint overhead, carbon, water, and TCO
Produce a summary table mapping domain -> solver -> binding wall
TipSolver Quick Reference
This capstone uses solvers from all six domains. For readers who arrived via an accelerated learning path, the table below summarizes what each solver does:
Solver
Domain
What It Computes
SingleNodeModel
Node
Roofline bottleneck, latency, throughput
DataModel
Data
Whether the data pipeline can sustain GPU demand
ScalingModel
Algorithm
Compute-optimal training budget (Chinchilla)
DistributedModel
Fleet
Communication overhead and scaling efficiency
ReliabilityModel
Fleet
Cluster MTBF and optimal checkpoint intervals
EconomicsModel
Ops
CapEx, OpEx, and total cost of ownership (TCO)
SustainabilityModel
Ops
Energy, carbon footprint, and water usage
SensitivitySolver
Analysis
Partial derivatives identifying the binding constraint
SynthesisSolver
Analysis
Minimum hardware specs from a latency target
TipBackground: The Six Taxonomy Domains
The MLSys wall taxonomy organizes 22 systems walls into six domains:
Domain
Walls
What It Covers
Node
1–3
Compute, memory capacity, memory bandwidth
Data
8–10
Storage throughput, data pipeline stalls
Algorithm
11–13
Scaling laws, compute-optimal training
Fleet
14–16
Communication, synchronization, reliability
Ops
17–20
TCO, energy, carbon, water, safety
Analysis
21–22
Sensitivity, inverse synthesis
No single solver spans all six. The insight emerges from composition.
1. Setup: Build the Fleet
We construct a 512-GPU training cluster: 64 DGX H100 nodes, 8 GPUs per node, NVLink intra-node (900 GB/s), InfiniBand NDR inter-node (400 Gbps), powered by Quebec (Hydro)’s hydroelectric grid (20 g CO2/kWh).
=== Domain: Node (Walls 1-3) ===
Feasible: False
Required memory: 131.5 GiB
Available memory: 80 GiB
Bottleneck: Memory
Per GPU latency: 3,089.6 ms
Throughput: 1.3 samples/s
At 70.6B parameters, Llama-3.1-70B weights alone require 141.2 GB in FP16, exceeding a single NVIDIA H100’s 80 GiB (85.9 GB) HBM capacity (feasible = False). Because the unsharded model spills across PCIe (64 GB / s), SingleNodeModel reports Memory-bound at 3089.6 ms per step (1.3 samples/s).
Wall 2 (Memory Capacity) and Wall 3 (Memory Bandwidth) immediately rule out unsharded single-GPU execution and force intra-node tensor parallelism (TP=8 in Section 5). The next question is whether the storage and preprocessing pipeline can keep up with 512 GPUs demanding training tokens.
3. Data (Walls 8–10): Whether the Pipeline Can Keep Up
The storage and preprocessing pipeline must deliver tokenized batches at least as fast as the cluster consumes them. If it cannot, the GPUs stall.
from mlsysim.solvers import DataModel# Estimate data demand per step: 4 samples/GPU * 512 GPUs * 2048 tokens * 2 bytes ≈ 8 MB/stepdata_demand = Q_("8 MB/s")data_solver = DataModel()data_result = data_solver.solve( workload_data_rate=data_demand, hardware=h100)banner("Domain: Data (Walls 8-10)")info(Data_demand=data_result.demand_bw, Data_supply=data_result.supply_bw, Utilization=f"{data_result.utilization:.2%}", Stalled=data_result.is_stalled, Bottleneck=data_result.bottleneck)
=== Domain: Data (Walls 8-10) ===
Data demand: 8.00e-03 GB/s
Data supply: 7 GB/s
Utilization: 0.11%
Stalled: False
Bottleneck: Storage
At 8 MB / s demand against 7.0 GB / s storage bandwidth, storage utilization is only 0.11% (is_stalled = False). For text-based training, the data pipeline is rarely the bottleneck because tokenized text is compact; for image or video training, this wall can dominate.
The remaining question is whether the 30-day compute budget on 512 GPUs is allocated optimally according to scaling laws.
The test of whether this training budget is compute-optimal is the Chinchilla scaling law, which prescribes \(D = 20P\) (tokens = \(20\times\) parameters) for optimal allocation.
from mlsysim.solvers import ScalingModel# MFU (Model FLOP Utilization): the fraction of peak hardware FLOP/s that goes# to useful model computation (excluding communication, idle time, overhead).# MFU = 0.4 means 40% of theoretical peak.gpu_flops = h100.compute.peak_flops.to("flop/s").magnitudetotal_flops = fleet.total_accelerators * gpu_flops *30*86400*0.4compute_budget = Q_(total_flops, "flop")scaling_solver = ScalingModel()scaling_result = scaling_solver.solve( compute_budget=compute_budget, target_model_size=model.parameters)banner("Domain: Algorithm (Walls 11-13)")info(Compute_budget=compute_budget.to('EFLOP'), Optimal_tokens=f"{scaling_result.optimal_tokens.magnitude:.2e}", Tokens_per_parameter=f"{scaling_result.tokens_per_parameter:.1f}", Chinchilla_ratio=f"{'OVER'if scaling_result.tokens_per_parameter >20else'UNDER'}-trained")
With 512 GPUs at 989 TFLOP/s peak FP16 and 40% MFU over 30 days, the compute budget reaches 525002 EFLOP, funding 1.24T tokens (17.6 tokens per parameter, slightly below the Chinchilla 20 tokens/parameter ratio). If the tokens-per-parameter ratio is significantly above or below 20, the training budget is not optimally allocated: over-training wastes compute, while under-training wastes model capacity.
5. Fleet (Walls 14–16): Communication and Reliability
Sharding Llama-3.1-70B with TP=8 within each DGX H100 node (PP=1, DP=64 across the 64 nodes) divides the weights across 8 GPUs per replica. The distributed solver models both the replica’s 2,048-token forward-backward pass and the AllReduce communication overhead, while the reliability solver computes cluster MTBF and optimal checkpoint intervals.
Fleet MTBF: 83.71 h
Failure probability: 99.98%
Expected failures: 8.6
Optimal ckpt interval: 141.7 min
With TP=8 across each node, the 2,048-token forward-backward pass is Compute-bound (7019.9 ms replica compute latency), and overlapping DP AllReduce (695.6 ms unoverlapped) with backward compute plus intra-node TP AllReduce (670.3 ms) yields a 7794.6 ms step time and 90.06% scaling efficiency.
At 512 GPUs, however, the cluster MTBF shrinks to 83.7 h, with 8.6 expected failures over the 30-day run (99.98% failure probability) and an optimal Young-Daly checkpoint interval of 141.7 min. Checkpoint overhead and rollback recovery become a non-trivial fraction of wall-clock time—the hidden cost that single-node analysis misses entirely.
6. Ops (Walls 17–20): TCO, Energy, Carbon, Water
The economics solver combines CapEx, OpEx, and sustainability into a single financial model.
IT Energy: 149.7 MWh
Total Energy PUE: 158.6 MWh
Carbon footprint: 3173 kg CO2
Water usage: 0 liters
PUE: 1.06
Region: Quebec (Hydro)
For the 30-day run, EconomicsModel reports $422,325 in total TCO ($350,685 amortized CapEx, $19,038 energy OpEx, and $52,603 maintenance OpEx). Quebec (Hydro)’s hydroelectric grid (20 g CO2/kWh, PUE 1.06) keeps total emissions to 3,173 kg CO2 (3.2 tonnes across 158.6 MWh of facility energy). The same run in Poland (820 g CO2/kWh, PUE 1.58) produces 193,910 kg CO2 (193.9 tonnes, 61× higher), making infrastructure geography a first-class engineering variable.
7. Analysis (Walls 21–22): Sensitivity and Synthesis
Finally, run SensitivitySolver on a single NVIDIA H100 and SynthesisSolver for a 200 ms step latency target.
from mlsysim.solvers import SensitivitySolver, SynthesisSolver# Sensitivity: check the single-GPU binding constraint for unsharded Llama-3.1-70Bsens_solver = SensitivitySolver()sens_result = sens_solver.solve( model=model, hardware=h100, precision="fp16")banner("Domain: Analysis (Walls 21-22)")info(Binding_constraint=sens_result.binding_constraint)sens_rows = [[param, f"{val:+.4f}"] for param, val in sens_result.sensitivities.items()]table(["Parameter", "Sensitivity"], sens_rows)
On an unsharded single NVIDIA H100, SensitivitySolver identifies memory_capacity as the binding constraint because the 141.2 GB FP16 weight footprint exceeds 80 GiB (85.9 GB), pinning all small local perturbations at PCIe offload bandwidth until model parallelism (TP=8) brings the per-GPU footprint inside HBM.
8. Summary Table: The Complete Picture
We have now traced a single workload through all six domains. Each solver answered one question in isolation. The systems engineer’s job, however, is synthesis: seeing the complete picture at once. The table below is that picture, and its most important property is that no single row captures the full story.
No single solver captures the full picture; the systems view emerges from composition. This end-to-end trace exercises 12 of 22 walls through a single model. At the single-GPU level, memory capacity (141.2 GB vs. 80 GiB) forces tensor parallelism (TP=8), where each 8-GPU replica is compute-bound (7020 ms per step at 90% scaling efficiency). The hidden costs only appear at fleet scale: a 83.7 h cluster MTBF forces checkpoints every 142 min, and infrastructure geography (Quebec (Hydro) vs. Poland (Coal)) changes grid carbon intensity by 41× and PUE-adjusted carbon emissions by 61× (as Tutorial 7 demonstrated). A complete systems analysis is not one solver run but the composition of all six domains.