Why 3.2× more FLOPS gives only 1.6× speedup, and how to know in advance.
node
beginner
Compare A100 and H100 GPUs to discover that for memory-bound workloads, bandwidth, not compute, determines performance. The most important fallacy in ML systems.
The Question
NVIDIA’s H100 has 3.2× more FLOP/s than the A100. Does upgrading deliver a 3.2× speedup?
It does not. For the workloads that matter most in production (LLM inference, recommendation models, any memory-bound task), the speedup is closer to 1.6× (tracking the 1.6× HBM bandwidth ratio). This tutorial shows why, and how to predict the actual speedup before spending a dollar on hardware.
NotePrerequisites
Complete Tutorial 0: Hello, Roofline. This tutorial assumes familiarity with memory-bound vs. compute-bound regimes and the ridge point concept.
NoteLearning Objectives
Calculate the actual speedup between two GPUs for a given workload
Explain why the binding constraint determines which spec matters
Predict whether a hardware upgrade will help a specific model
Apply the roofline model to hardware procurement decisions
TipBackground: The Two Specs That Matter
GPU vendors advertise peak FLOP/s prominently, yet every GPU also has a memory bandwidth spec (in TB/s) that is equally important. Which spec determines actual performance depends entirely on which regime the workload is in:
Regime
Binding Constraint
Speedup Scales With
Memory-bound
HBM bandwidth (TB/s)
Bandwidth ratio between GPUs
Compute-bound
Peak arithmetic (FLOP/s)
FLOP/s ratio between GPUs
The key numbers for this tutorial:
Spec
A100
H100
Ratio
Peak FP16
312 TFLOP/s
989 TFLOP/s
3.2×
HBM Bandwidth
2.04 TB/s
3.35 TB/s
1.6×
For a memory-bound workload, the speedup ceiling is 1.6×, regardless of the 3.2× compute improvement.
1. Setup
import mlsysimfrom mlsysim import Engine
2. Side-by-Side Hardware Comparison
Load both GPUs from the Silicon Zoo and confirm the specs:
The FLOP/s ratio is 3.2× (989 vs. 312 TFLOP/s), whereas the bandwidth ratio is only 1.6× (3.35 vs. 2.04 TB/s). The theoretical ridge point also shifts from 153.0 to 295.2 FLOP/byte: the H100 has a higher ridge, meaning more workloads fall into the memory-bound regime on the H100 than on the A100.
3. The Fallacy: LLM Inference Speedup
The “3.2× speedup” claim can be tested with a workload that dominates production today, Llama-3 8B inference at batch size 1:
A100 H100 Speedup
─────────────────────────────────────────
Bottleneck Memory Memory
Latency 9.00 ms 5.60 ms 1.6x
Throughput 111.1 1/s 178.5 1/s
Both GPUs report memory-bound. The actual speedup is approximately 1.6× (9.00 ms vs. 5.60 ms), close to the 1.6× bandwidth ratio rather than the 3.2× FLOP/s ratio. The remaining 2.0× of the H100’s compute advantage goes unused for this workload.
Sanity check: Llama-3 8B at FP16 has 8.03B parameters × 2 bytes = 16.06 GB of weights. On the A100 (2.04 TB/s), pure weight-streaming latency is 16.06 GB ÷ 2.04 TB/s = 7.9 ms (9.0 ms in Engine.solve after activation traffic and kernel launch tax). On the H100 (3.35 TB/s), pure weight-streaming latency is 16.06 GB ÷ 3.35 TB/s = 4.8 ms (5.6 ms in Engine.solve). The weight-streaming speedup is 7.9 / 4.8 = 1.64×, matching the bandwidth ratio, as expected for a memory-bound workload.
4. When 3.2× Matters: The Batch Size Crossover
The FLOP/s advantage takes effect only once the workload crosses into the compute-bound regime. Sweeping batch size on both GPUs locates the crossover:
rows = []for batch in [1, 4, 16, 32, 64, 128, 256]: pa = Engine.solve(model=model, hardware=a100, batch_size=batch, precision="fp16") ph = Engine.solve(model=model, hardware=h100, batch_size=batch, precision="fp16") la = pa.latency.to("ms").magnitude lh = ph.latency.to("ms").magnitude sp = la / lh if lh >0else0 ai = pa.arithmetic_intensity.to("flop/byte") rows.append([batch, f"{ai:~.1f}", pa.bottleneck, ph.bottleneck, f"{sp:.1f}x"])table(["Batch Size", "Arithmetic Intensity", "A100 Bottleneck", "H100 Bottleneck", "Actual Speedup"], rows)
Batch Size Arithmetic Intensity A100 Bottleneck H100 Bottleneck Actual Speedup
──────────────────────────────────────────────────────────────────────────────────
1 0.9 flop / B Memory Memory 1.6x
4 2.9 flop / B Memory Memory 1.6x
16 6.2 flop / B Memory Memory 1.6x
32 7.6 flop / B Memory Memory 1.6x
64 8.6 flop / B Memory Memory 1.6x
128 9.3 flop / B Memory Memory 1.6x
256 9.6 flop / B Memory Memory 1.6x
At batch size 256, where both GPUs are compute-bound, the speedup reaches 1.6×, approaching the 3.2× compute ratio. The roofline plot below shows where Llama-3 8B at batch size 1 sits on the H100 relative to its 295.2 FLOP/byte ridge point:
from mlsysim.viz.plots import plot_roofline# Plot the H100 roofline and see where Llama-3 8B (batch 1) fallsfig, ax = plot_roofline(h100, workloads=[model])
ImportantKey Insight
The binding constraint determines which hardware spec matters. For a memory-bound workload, speedup scales with the bandwidth ratio (1.6×). For a compute-bound workload, speedup scales with the FLOP/s ratio (up to 3.2×). The transition happens at different batch sizes on each GPU because the H100’s higher ridge point (295.2 vs. 153.0 FLOP/byte) means it stays memory-bound longer. In a procurement decision, the first question is not “how many FLOP/s?” but “which regime will the production workload operate in?”
5. The Procurement Table: Three Generations
Extending the analysis across three GPU generations shows the trend:
GPU TFLOP/s BW (TB/s) Ridge Latency Bottleneck
─────────────────────────────────────────────────────
V100 125.0 0.9000 138.9 19.97 ms Memory
A100 312.0 2.04 153.0 9.00 ms Memory
H100 989.0 3.35 295.2 5.60 ms Memory
Across three generations (from 125 TFLOP/s and 0.9 TB/s on the V100 to 989 TFLOP/s and 3.35 TB/s on the H100), compute has grown 7.9× while bandwidth has grown 3.7×. The theoretical ridge point rises from 138.9 to 153.0 to 295.2 FLOP/byte, which means more workloads are memory-bound on newer hardware. This is the memory wall, and it is getting steeper with each generation.
