The 22 Laws of ML Systems — Plain English

Every equation in mlsysim answers one question. Here is what each one says, why it matters, and how to explain it to anyone in 30 seconds.

Domain 1: Node — What Can One Chip Do?

Wall 1: The Compute Wall

Equation: Time = Operations / (Peak_FLOPS × Efficiency)

Plain English: How long does it take to do all the math? Take the total number of multiply-adds your model needs, divide by how fast your chip can do math (adjusted for real-world inefficiency), and that’s your compute time.

30-second version: “If your model needs 16 billion multiplications and your GPU can do 989 trillion per second at 50% efficiency, the math takes 0.03 milliseconds.”

Why it matters: This is the speed limit for any workload. No software optimization can make your model run faster than the hardware can crunch numbers.

Wall 2: The Memory Wall

Equation: Time = Weight_Bytes / Memory_Bandwidth

Plain English: How long does it take to load the model from memory? Your model’s weights sit in HBM (the GPU’s memory). To use them, the GPU must read them through a pipe with a fixed width (bandwidth). Bigger model = longer loading time.

30-second version: “A 16 GB model on a GPU with 3.35 TB/s bandwidth takes 4.8 ms just to load the weights — before any computation even starts.”

Why it matters: For most LLM inference at batch size 1, this is THE bottleneck. The GPU finishes the math long before it finishes reading the weights. This is why memory bandwidth matters more than FLOPS for serving chatbots.

Wall 3: The Software Wall

Equation: MFU = Achieved_FLOPS / Peak_FLOPS

Plain English: How much of the chip’s power are you actually using? MFU (Model FLOPs Utilization) measures the gap between what the hardware could do and what your software actually achieves. Low MFU means wasted silicon.

30-second version: “If your GPU can do 989 TFLOPS but your training only achieves 450 TFLOPS, your MFU is 45%. The other 55% is lost to kernel launches, memory stalls, and framework overhead.”

Why it matters: A 50% MFU means you’re paying for twice the hardware you’re using. Improving MFU is often cheaper than buying more GPUs.

Wall 4: The Serving Wall

Equation: TTFT = Prefill_FLOPs / Peak; ITL = Weights / Bandwidth

Plain English: LLM serving has two phases with completely different bottlenecks. Prefill (processing your prompt) is compute-bound — it’s doing tons of math. Decode (generating each token) is memory-bound — it’s loading weights for every single token. These two phases need different optimizations.

30-second version: “Prefill is like reading a book fast (CPU-intensive). Decode is like looking up one word at a time in a dictionary (memory-intensive). That’s why your first token takes 50ms but each subsequent token takes 5ms.”

Wall 5: The Batching Wall

Equation: KV_cache = 2 × Layers × Heads × Dim × SeqLen × Batch × BytesPerElem

Plain English: Each active request in an LLM server needs its own “memory” of the conversation (the KV cache). This memory grows with sequence length and eats into the GPU’s finite HBM. More concurrent requests = more KV cache = less room for new requests.

30-second version: “Each Llama-3-8B request at 4K context needs ~500 MB of KV cache. An 80 GB GPU can only serve ~100 concurrent requests before running out of memory.”

Wall 6: The Streaming Wall

Equation: Time_per_layer = max(Weight_Injection_Time, Compute_Time)

Plain English: Wafer-scale chips (like Cerebras) flip the normal GPU bottleneck. Instead of weights sitting in memory and being read by compute, weights are streamed from external memory nodes while activations live on-chip in SRAM. The bottleneck shifts from memory bandwidth to the injection interconnect speed.

Wall 7: The Tail Latency Wall

Equation: Erlang C queueing model (M/M/c)

Plain English: When your servers are busy, new requests have to wait in line. The waiting time grows non-linearly — at 80% utilization, the P99 latency might be 2x the average. At 95% utilization, it can be 10x. This is why you can’t run servers at 100% — the tail latency destroys your SLA.

30-second version: “It’s like a highway. At 70% capacity, traffic flows. At 95% capacity, everyone’s stuck. Servers work the same way.”

Domain 2: Data — Can You Feed the Beast?

Wall 8: The Ingestion Wall

Equation: Utilization = Data_Demand / Storage_Supply

Plain English: Your GPU is hungry. Can your storage feed it fast enough? If the GPU consumes data faster than the disk can supply it, the GPU sits idle waiting for food. This ratio tells you if your data pipeline is the bottleneck.

Wall 9: The Transformation Wall

Equation: Time = Batch × Sequence / CPU_Throughput

Plain English: Before data reaches the GPU, CPUs must decode images, tokenize text, and apply augmentations. If these CPU operations can’t keep up with GPU consumption, the GPU starves. This is why training pipelines often need 10-100x more CPU cores than you’d expect.

Wall 10: The Locality Wall

Equation: Effective_BW = Link_BW × Bisection_Fraction / Oversubscription

Plain English: Not all network bandwidth is created equal. A fat-tree network gives full bisection bandwidth (any node can talk to any other at full speed). A ring or torus does not — distant nodes communicate through intermediate hops, reducing effective bandwidth. The topology determines how much of the raw link speed you actually get for collective communication.

Domain 3: Algorithm — How Much Compute Do You Need?

Wall 11: The Complexity Wall (Chinchilla Scaling)

Equation: Compute = 6 × Parameters × Tokens; Optimal_Params = √(Budget/120)

Plain English: There’s a sweet spot between model size and training data. The Chinchilla scaling law says: for a fixed compute budget, the optimal model size is proportional to the square root of your budget, and you should train on 20 tokens per parameter. Train a too-big model on too-little data = waste. Train a too-small model on too-much data = also waste.

30-second version: “With $10M of compute, you should train a ~90B parameter model on ~1.8T tokens. Going bigger without more data just wastes money.”

Wall 12: The Reasoning Wall [Emerging]

Equation: Time = Steps × Time_per_Step

Plain English: Chain-of-thought reasoning makes models think longer before answering. Each reasoning step costs compute. More steps = better answers but higher cost. This is inference-time scaling — spending more compute at serving time instead of training time.

Wall 13: The Fidelity Wall (Compression)

Equation: Compression = 32/bits (quantization); Compression = 1/(1-sparsity) (pruning)

Plain English: You can shrink a model by using fewer bits (quantization: FP32→INT8 = 4x smaller) or by removing weights (pruning: 50% sparse = 2x smaller). But shrinking costs accuracy. The question is always: how much accuracy can you afford to lose?

Key insight: Storage shrinks for all compression, but inference speedup depends on the method. Unstructured pruning saves storage but gives zero speedup on GPU GEMM. Only structured pruning and N:M sparsity accelerate actual hardware execution.

Domain 4: Fleet — What Happens at Scale?

Wall 14: The Communication Wall (AllReduce)

Equation: Time = 2(N-1)/N × Message/Bandwidth + 2(N-1) × Latency

Plain English: Distributed training means every GPU must share its gradients with every other GPU after each training step. This “AllReduce” operation moves data through the network. The time depends on how much data (gradient size) and how fast the network (bandwidth). As you add more GPUs, the communication cost grows, eventually dominating over the compute savings.

30-second version: “1 GB of gradients across 8 GPUs at 50 GB/s NVLink takes 35ms. That’s time your GPUs are waiting instead of computing.”

Wall 15: The Fragility Wall (Reliability)

Equation: Cluster_MTBF = Component_MTBF / N_components

Plain English: If one GPU fails every 50,000 hours, a 1000-GPU cluster has a failure every 50 hours. At 10,000 GPUs, you get a failure every 5 hours. This is why large training runs need checkpointing — without it, a single GPU failure wastes days of compute.

30-second version: “At 1000 GPUs, something breaks every 2 days. At 100,000 GPUs, something breaks every 30 minutes.”

Wall 16: The Multi-tenant Wall (Queueing)

Equation: Wait_Time = Utilization / [2 × Service_Rate × (1 - Utilization)]

Plain English: When multiple teams share a cluster, jobs wait in line. The wait time grows non-linearly with utilization — at 80% cluster utilization, the average wait is 2x the job duration. This is why shared research clusters feel slow even when they’re “only” 80% busy.

Domain 5: Operations — Is It Worth It?

Wall 17: The Capital Wall (TCO)

Equation: TCO = CapEx + OpEx_energy + OpEx_maintenance

Plain English: The total cost of running ML infrastructure is hardware purchase (CapEx, amortized over 3-5 years) plus electricity plus maintenance. A 1024-GPU H100 cluster costs ~$30M in hardware. Electricity is surprisingly small (~10% of total). The dominant cost lever is utilization — idle GPUs cost the same as busy ones.

Wall 18: The Sustainability Wall

Equation: Carbon = Energy × PUE × Carbon_Intensity

Plain English: Every GPU-hour converts electricity into carbon emissions. How much depends on three things: how much energy (determined by power and time), how efficiently the datacenter uses it (PUE — a 1.4 PUE wastes 40% on cooling), and how dirty the grid is (Quebec hydro = 20 gCO₂/kWh vs Poland coal = 820 gCO₂/kWh).

30-second version: “Training the same model in Quebec produces 41x less carbon than training it in Poland. Geography is the biggest lever for sustainable AI.”

Wall 19: The Checkpoint Wall

Equation: MFU_penalty = Write_Time / Checkpoint_Interval

Plain English: Long training runs must periodically save their state to disk in case of failure. This I/O burst interrupts training and reduces effective utilization. The optimal checkpoint frequency balances the cost of saving too often (I/O overhead) against the cost of saving too rarely (wasted compute when failures hit).

Wall 20: The Safety Wall (Privacy)

Equation: Slowdown ∝ 1/ε (DP-SGD)

Plain English: Training with differential privacy (adding noise to protect individual data points) makes training slower. Stronger privacy guarantees (smaller ε) mean more noise, which means more training steps to converge. Privacy is not free — it’s a compute tax.

Domain 6: Meta-Analysis — Cross-Cutting Tools

Wall 21: Sensitivity Analysis

Equation: ∂Time/∂parameter (partial derivatives)

Plain English: Which constraint is actually binding? Sensitivity analysis perturbs each parameter (bandwidth, FLOPS, network speed) by a small amount and measures how much the total time changes. The parameter with the largest derivative is your bottleneck — that’s where to invest next.

Wall 22: Synthesis (Inverse Roofline)

Equation: Required_BW = Weights / Target_Latency

Plain English: Given a latency target (e.g., “50ms per token”), what hardware specs do you need? This flips the Roofline model backwards — instead of predicting performance from hardware, it derives the minimum hardware specs from a performance requirement.

30-second version: “If you need 50ms decode latency for a 140 GB model, you need at least 2.8 TB/s of memory bandwidth. That’s more than one A100 — you need tensor parallelism across two H100s.”

The Iron Law (The Master Equation)

Equation: Time = FLOPs / (N × Peak × MFU × η_scaling × Goodput)

Plain English: This one equation governs all of ML systems. Training time equals the total compute divided by how much useful compute you can actually deliver. Each term in the denominator represents a different loss:

Term What it means What reduces it
N Number of devices Budget
Peak Raw hardware speed GPU generation
MFU Software efficiency Kernel optimization, FlashAttention
η_scaling Communication overhead Network bandwidth, gradient compression
Goodput Time lost to failures Checkpointing, fault tolerance

Every wall in the taxonomy maps to one of these five terms. That’s the whole framework.

Back to top