Calibration and validation scope
MLSysIM estimates are analytical outputs, not hardware measurements. The table below is generated from engine.empirical; it distinguishes calibrated checks, broad sanity ranges, and formula accounting. Passing these checks does not establish an accuracy percentage for unseen workloads.
| Anchor | Evidence category | Analytical output | Reference | Difference |
|---|---|---|---|---|
resnet50_a100_train_bs256 |
Calibrated regression | 3234.44 1/s | 3200 1/s | +1.08% |
resnet50_h100_train_bs256 |
Calibrated regression | 5070.12 1/s | 5000 1/s | +1.40% |
llama3_8b_h100_bs1_itl |
Family sanity range | 5.19416 ms | 3–10 ms | within range |
gpt3_175b_training_flops |
Formula accounting | 3.15e+23 flop | 3.14e+23 flop | +0.32% |
Executed configurations
- resnet50_a100_train_bs256: ResNet-50 training on A100, batch_size=256, efficiency=0.26. The 10% band is intentionally wider than MLPerf run-to-run variation because MLSYSIM uses a first-order roofline plus a calibrated efficiency term.
- resnet50_h100_train_bs256: ResNet-50 training on H100, batch_size=256, efficiency=0.13. The calibrated efficiency is lower than A100 because ResNet-50 does not saturate H100 peak tensor throughput in this first-order model.
- llama3_8b_h100_bs1_itl: Llama-3-8B serving on H100, seq_len=2048, batch_size=1, efficiency=0.5. Decode is memory-bandwidth dominated; this range is a sanity envelope, not a production SLA for a specific scheduler or prompt distribution.
- gpt3_175b_training_flops: 175B parameters, 300B training tokens, 6PD transformer training rule. The tight band is appropriate here because the comparison is formula-level FLOP accounting, not a hardware throughput measurement.
What the comparisons establish
The ResNet efficiencies are calibration inputs, not independently predicted utilization. The H100 reference is a per-accelerator result from a distributed training run, not a matched single-device experiment. The Llama interval is a family sanity range; it does not identify one engine, precision, and request shape. The GPT-3 comparison checks FLOP accounting, not wall-clock performance. Source identities and limitations are recorded in the provenance catalog.
For an external accuracy claim, pin the source revision, hardware topology, model, engine, precision, batch/concurrency, input/output lengths, and metric. Retain the raw timing artifact and declare efficiency assumptions before the comparison. Evaluate held-out configurations separately from calibration. At batch one, decode ITL is (end_to_end_latency - TTFT) / (output_tokens - 1); end-to-end latency divided by output tokens includes prefill and is a different metric. A TTFT prediction must not be checked against an ITL reference.
Modeling conventions
efficiencyderates compute FLOP/s. It does not derate memory bandwidth.- Serving decode uses the larger of compute and weight/KV memory time, plus the configured layer tax. Continuous batching uses the same step accounting.
- Prefill is a compute-based estimate, with attention work and dispatch taxes; very short prompts can be memory- or overhead-limited in practice.
- A weight-only bandwidth floor remains useful, but is not a complete latency prediction or a serving capacity guarantee.
- Runtime workspaces, scheduling interference, and inter-device communication require separate modeling or measurement. Feasible means the modeled state fits; it does not reserve all memory used by a production engine.
Reproduce
From an editable checkout’s mlsysim/ directory:
python scripts/calibration_report.py --check
python -m pytest tests/test_empirical.pyRun the script without --check to regenerate this file. Numerical outputs and reference differences are computed by the script, not entered by hand.