Numerical sensitivity and inverse Roofline synthesis reveal where hardware budgets create value.
analysis
advanced
Use partial derivatives of latency to identify the binding constraint for any model-hardware pair. Then invert the Roofline to derive minimum hardware specs from an SLA.
The Question
A team has budget for one hardware upgrade. Should it buy more FLOP/s or more memory bandwidth? Intuition suggests that more compute is always better, yet for single-stream Llama-3.1-70B (70.6B parameters) inference on an NVIDIA A100, a 10% bandwidth upgrade changes latency by -8.90% (an elasticity of \(\frac{\Delta T / T}{\Delta \text{BW} / \text{BW}} = \-0\.89\)), whereas a 10% compute upgrade changes latency by +0.00% (memory transfer takes 84× longer than compute). This tutorial computes those sensitivities analytically and then inverts the Roofline to derive minimum hardware specifications from an SLA.
Compute numerical sensitivities of latency with respect to each hardware parameter
Identify the binding constraint for any model-hardware pair
Quantify the asymmetry between bandwidth and FLOP/s sensitivity
Derive minimum hardware specs from a latency SLA using inverse Roofline synthesis
TipBackground: Sensitivity Analysis
In optimization, the binding constraint is the resource that actually limits performance, the one holding with equality at the solution. SensitivitySolver perturbs each hardware parameter (peak_flops, memory_bandwidth, memory_capacity) by a fixed percentage (\(\Delta x / x = 10%\)) and measures the normalized latency response \(\Delta T / T = (T_{\text{perturbed}} - T_{\text{base}}) / T_{\text{base}}\) (with corresponding elasticity \(\frac{\Delta T / T}{\Delta x / x}\)). The parameter with the largest absolute response is the binding constraint and the lever most worth investing in.
The analysis targets Llama-3.1-70B (70.6B parameters) inference on a single NVIDIA A100 (80 GiB / 85.9 GB HBM2e). At FP16 (141.2 GB of weights), the model exceeds a single 80 GiB GPU and binds on memory_capacity; quantizing to INT8 (70.6 GB of weights) fits within HBM and exposes the steady-state hardware sensitivities.
from mlsysim.solvers import ServingModelfrom mlsysim.solvers import SensitivitySolver, SynthesisSolverfrom mlsysim.core.units import Q_from mlsysim.show import table, infomodel = mlsysim.Models.Language.Llama3_70Bhardware = mlsysim.Hardware.Cloud.A100# Compute normalized latency response to a 10% upgrade in each hardware parametersolver = SensitivitySolver()res = solver.solve(model=model, hardware=hardware, precision="int8")info("Configuration", Model=model.name, Hardware=hardware.name, Baseline_latency=res.baseline_latency.to('ms'), Perturbation=f"{res.perturbation_pct}%")rows = [[param, f"{sensitivity:+.4f}"] for param, sensitivity in res.sensitivities.items()]table(["Parameter", "Sensitivity"], rows)
Each reported sensitivity is the fractional latency change \(\Delta T / T\) for a 10% parameter increase. A sensitivity of -0.0890 on memory_bandwidth (elasticity \(\frac{\Delta T / T}{\Delta \text{BW} / \text{BW}} = \-0\.89\)) means a 10% bandwidth increase yields an 8.9% latency reduction (from 38.90 ms to 35.44 ms). A sensitivity of +0.0000 on peak_flops means additional compute produces zero latency reduction.
3. The Binding Constraint
info("Binding Constraint", Constraint=res.binding_constraint, Interpretation=f"{res.binding_constraint} is the hardware knob most worth turning for {model.name} on {hardware.name}")
── Binding Constraint ──────────────────────
Constraint: memory_bandwidth
Interpretation: memory_bandwidth is the hardware knob most worth turning for Llama-3.1-70B on NVIDIA A100
For Llama-3.1-70B (70.6B parameters) at batch size 1, every decode step streams 70.6 GB of INT8 weights (plus activations) from HBM. The arithmetic intensity is 1.82 FLOP/byte, far below the NVIDIA A100 ridge point of 153.0 FLOP/byte. Memory transfer takes 38.09 ms while compute takes only 0.45 ms, confirming quantitatively that the workload is deeply memory-bound.
4. The Bandwidth vs. Compute Asymmetry
The asymmetry between memory bandwidth and compute throughput appears directly in the sensitivity comparison:
sens_bw =abs(res.sensitivities.get("memory_bandwidth", 0))sens_flops =abs(res.sensitivities.get("peak_flops", 0))if sens_flops >0: ratio = sens_bw / sens_flops info("Sensitivity Asymmetry", Bandwidth_sensitivity=f"{sens_bw:.4f}", FLOPS_sensitivity=f"{sens_flops:.4f}", Ratio=f"{ratio:.1f}x", Verdict=f"A dollar spent on bandwidth improvement is ~{ratio:.0f}x more impactful than the same dollar spent on more FLOP/s")else: info("Sensitivity Asymmetry", Bandwidth_sensitivity=f"{sens_bw:.4f}", FLOPS_sensitivity=f"{sens_flops:.4f}", Verdict="FLOPS has zero sensitivity --- purely memory-bound")
── Sensitivity Asymmetry ───────────────────
Bandwidth sensitivity: 0.0890
FLOPS sensitivity: 0.0000
Verdict: FLOPS has zero sensitivity --- purely memory-bound
ImportantKey Insight
Sensitivity analysis quantifies why bandwidth dominates single-stream LLM inference. Under the Roofline model \(T = \max(T_{\text{compute}}, T_{\text{memory}}) + T_{\text{overhead}}\), memory transfer for Llama-3.1-70B on NVIDIA A100 takes 38.09 ms versus 0.45 ms for compute (an 84× gap). Consequently, a 10% bandwidth increase reduces latency by 8.90% (\(\Delta T / T = \-0\.0890\), elasticity \(\-0\.89\)), whereas a 10% compute increase yields 0.00% improvement (\(\Delta T / T = \+0\.0000\)). The binding constraint, not the headline spec, determines where a hardware budget creates value.
WarningFallacy: Investing in the Highest-Spec Number Maximizes Performance
GPU vendors advertise peak FLOP/s prominently (312 TFLOP/s on NVIDIA A100, 989 TFLOP/s on NVIDIA H100) because the number is large. For memory-bound workloads where \(T_{\text{memory}}\) exceeds \(T_{\text{compute}}\) by 84×, a 10% bandwidth increase cuts latency by 8.9% while a 10% compute increase yields 0.0%. The datasheet headline and the binding constraint are often different parameters, and sensitivity analysis shows which one actually matters.
5. Inverse Roofline: From SLA to Hardware
Sensitivity analysis identifies which parameter is worth improving. The natural follow-up is how much improvement a given performance target requires.
The SynthesisSolver inverts the Roofline model. Instead of “given hardware, what is the latency?”, it asks: “given a latency SLA, what hardware do I need?”
Suppose a deployment requires an FP16 inter-token latency (ITL) of 50 ms or less:
── Inverse Roofline: Required Hardware ─────
Target SLA: 50 ms ITL
Min memory BW: 2.82 TB/s
Min compute: 5.65 TFLOPs/s
Min memory: 141.2 GB
Meeting a 50 ms FP16 SLA for Llama-3.1-70B requires 2.82 TB/s of memory bandwidth (1.38× what NVIDIA A100’s 2.04 TB/s provides), at least 5.65 TFLOP/s of effective compute, and 141.2 GB of HBM capacity. This immediately explains why single-GPU FP16 exceeded the 80 GiB (85.9 GB) NVIDIA A100 and narrows the single-device search to high-capacity H100/H200/B200-class accelerators or multi-GPU tensor parallelism.
6. Generational Comparison: Whether the Binding Constraint Shifts
Sensitivity analysis also reveals whether hardware upgrades shift the binding constraint. The comparison below spans three GPU generations for INT8 single-GPU inference:
Across all three generations (NVIDIA A100 at 38.90 ms, NVIDIA H100 at 23.99 ms, and NVIDIA H200 at 16.99 ms), memory_bandwidth remains the binding constraint (-0.0890, -0.0878, and -0.0866). In FP16 (141.2 GB required), by contrast, NVIDIA A100 and NVIDIA H100 (85.9 GB capacity) bind on memory_capacity, NVIDIA H200 (151.4 GB capacity) exhibits a memory_capacity sensitivity of +0.0000 because a 10% capacity increase crosses the 141.2 GB fit threshold, and NVIDIA B200 (193.3 GB capacity) shifts the binding constraint from memory_capacity to memory_bandwidth (-0.0873 at 20.22 ms).
Next Steps
GPU vs. Wafer-Scale: See how a fundamentally different architecture changes which wall binds
Full-Stack Audit: Compose all solvers into a complete systems analysis
The Memory Wall: Revisit the foundational tutorial on memory-bound vs. compute-bound