Skip to main content

The Differential Explainer

Automated ‘Why?’ analysis for hardware upgrades.

analysis
intermediate
Learn how to use the DifferentialExplainer to automatically compare two configurations and generate a written explanation of the performance delta.

The Question

An analytical comparison between an NVIDIA A100 and an NVIDIA H100 on Llama-3.1-8B at batch size 1 reports:

  • A100 Latency: 9.00 ms
  • H100 Latency: 5.60 ms

The speedup is 1.61×, yet the hardware specification sheet states that the H100 has 3.2× more FLOP/s (and 1.64× more HBM bandwidth). How do we automatically explain this discrepancy to a user or a stakeholder without manually digging through the formulas?

NoteLearning Objectives
  • Compare two system evaluations automatically.
  • Generate a human-readable explanation of why a speedup did (or did not) match hardware specs.
  • Identify “Regime Shifts” where an upgrade fundamentally changes the bottleneck.

1. Setup

Import the necessary modules. The tutorial uses the standard Engine to get the baseline and proposed profiles, and the DifferentialExplainer to compare them.

import mlsysim
from mlsysim.engine.engine import Engine
from mlsysim.engine.explainers import DifferentialExplainer

2. A Memory-Bound Upgrade (The Disappointment)

The classic scenario is a hardware upgrade for LLM inference at a low batch size. For Llama-3.1-8B at batch size 1, both GPUs sit in the memory-bound regime, so the 1.61× speedup tracks the 1.64× HBM bandwidth ratio rather than the 3.2× compute ratio:

model = mlsysim.Models.Language.Llama3_8B

# Get the two profiles
prof_a100 = Engine.solve(model=model, hardware=mlsysim.Hardware.Cloud.A100, batch_size=1)
prof_h100 = Engine.solve(model=model, hardware=mlsysim.Hardware.Cloud.H100, batch_size=1)

# Ask the explainer what happened
explanation = DifferentialExplainer.compare_performance(
    baseline=prof_a100,
    proposal=prof_h100
)

print(explanation)
📊 Differential Analysis: Proposal vs. Baseline
• Speedup: 1.61x
• Baseline Regime: Memory
• Proposal Regime: Memory

Analysis: The workload remained Memory Bound. The speedup is constrained strictly by the ratio of HBM bandwidth between the two configurations. Any additional compute capacity (FLOP/s) in the proposal was left unutilized.

3. Compute-Bound Scaling and Regime Shifts

For a high-intensity workload such as ResNet-50 at batch size 256, both GPUs operate in the compute-bound regime, giving a 2.94× speedup that approaches the 3.2× peak compute ratio:

cnn = mlsysim.Models.Vision.ResNet50

# At batch size 256, both GPUs are compute-bound
prof_a100_batch = Engine.solve(model=cnn, hardware=mlsysim.Hardware.Cloud.A100, batch_size=256)
prof_h100_batch = Engine.solve(model=cnn, hardware=mlsysim.Hardware.Cloud.H100, batch_size=256)

explanation_batch = DifferentialExplainer.compare_performance(
    baseline=prof_a100_batch,
    proposal=prof_h100_batch
)

print(explanation_batch)
📊 Differential Analysis: Proposal vs. Baseline
• Speedup: 2.94x
• Baseline Regime: Compute
• Proposal Regime: Compute

Analysis: The workload remained Compute Bound. The speedup is constrained strictly by the ratio of peak arithmetic throughput (FLOP/s) between the two configurations. Additional memory bandwidth was not the limiting factor.

When an upgrade crosses the ridge point—such as ResNet-50 at batch size 1, where the NVIDIA A100 is Compute-bound (0.57 ms) and the NVIDIA H100’s 3.2× higher compute ceiling shifts the bottleneck to Memory (0.53 ms, a 1.08× speedup)—DifferentialExplainer flags the Regime Shift automatically:

prof_a100_b1 = Engine.solve(model=cnn, hardware=mlsysim.Hardware.Cloud.A100, batch_size=1)
prof_h100_b1 = Engine.solve(model=cnn, hardware=mlsysim.Hardware.Cloud.H100, batch_size=1)

print(DifferentialExplainer.compare_performance(baseline=prof_a100_b1, proposal=prof_h100_b1))
📊 Differential Analysis: Proposal vs. Baseline
• Speedup: 1.08x
• Baseline Regime: Compute
• Proposal Regime: Memory

Analysis: Regime Shift. The proposed hardware provided enough compute capacity to lift the compute wall, exposing a memory bandwidth bottleneck. The speedup is a blended result of higher compute and the new memory ceiling.

Next Steps

Back to top