Skip to main content

For Engineers & Researchers

Back-of-envelope estimates before provisioning hardware.

MLSys·im produces quick, type-safe analytical estimates for capacity planning, hardware selection, cost modeling, and sustainability analysis, in seconds and from specifications alone. Equations are documented and tied to the relevant literature; hardware registry specs come from datasheets where available.

Why Use Analytical Models

Before running expensive benchmarks or provisioning cloud instances, an engineer needs directional answers:

  • Will this model fit in GPU memory? Check before renting the GPU
  • Why does training fail when inference fits? Separate weights, gradients, optimizer state, and activations
  • What is the expected TTFT for my LLM? Estimate before building the serving stack
  • How many replicas do I need for a P99 target? Size serving capacity before benchmarking
  • What is the MoE routing tax? Sweep expert imbalance before choosing EP topology
  • How many H100s do I actually need? Model scaling efficiency before buying the cluster
  • What will this cost per year? TCO analysis before signing the contract
  • How often will my training job crash? Reliability modeling before committing to a 30-day run
  • What is the carbon footprint of this deployment? Quantify before the sustainability review

MLSys·im answers these in microseconds using first-order equations. It does not replace profiling, but it shows where to start profiling.

TipTheory Behind the Tools

Each solver implements equations from the Math Foundations page. For the full conceptual framework, see the companion slide decks linked in the Solver-to-Slides Map below.

Quick Start: Roofline Analysis

The Engine implements the Roofline Performance Model (Williams et al. 2009) to classify workloads as compute-bound or memory-bound.

import mlsysim
from mlsysim import Engine

# Single-node: Is ResNet-50 memory-bound on A100?
profile = Engine.solve(
    model=mlsysim.Models.Vision.ResNet50,
    hardware=mlsysim.Hardware.Cloud.A100,
    batch_size=1, precision="fp16"
)
print(f"{profile.bottleneck}, {profile.latency.to('ms'):~.2f}")
print(f"MFU: {profile.mfu:.1%}, Arithmetic Intensity: {profile.arithmetic_intensity:~.2f}")
Compute, 0.57 ms
MFU: 4.6%, Arithmetic Intensity: 145.84 flop / B

The returned PerformanceProfile reports latency, throughput, bottleneck classification, Model FLOPs Utilization (MFU), arithmetic intensity, energy, and a feasibility flag, which together support a first-pass hardware assessment.

LLM Serving Analysis

The ServingModel models the two-phase LLM inference lifecycle: compute-bound pre-fill and memory-bound decoding.

import mlsysim
from mlsysim.solvers import ServingModel

serving = ServingModel()
result = serving.solve(
    model=mlsysim.Models.Language.Llama3_70B,
    hardware=mlsysim.Hardware.Cloud.H100,
    seq_len=4096, batch_size=1
)
print(f"TTFT:       {result.ttft.to('ms'):~.1f}")
print(f"ITL:        {result.itl.to('ms'):~.2f}")
print(f"KV-cache:   {result.kv_cache_size.to('GB'):~.1f}")
print(f"Feasible:   {result.feasible}")
print(f"Mem util:   {result.memory_utilization:.0%}")
TTFT:       1258.5 ms
ITL:        43.35 ms
KV-cache:   1.3 GB
Feasible:   False
Mem util:   166%

The feasibility check shows immediately whether the model plus its KV-cache fit in device memory, before an out-of-memory error appears in production.

Hardware Sweep Pattern

Compare devices programmatically instead of reading datasheets:

import mlsysim
from mlsysim import Engine

model = mlsysim.Models.Vision.ResNet50

for hw in [mlsysim.Hardware.Cloud.H100,
           mlsysim.Hardware.Cloud.A100,
           mlsysim.Hardware.Cloud.T4,
           mlsysim.Hardware.Edge.JetsonOrinNX]:
    p = Engine.solve(model=model, hardware=hw, batch_size=32, precision="fp16")
    print(f"{hw.name:22s}  {p.bottleneck:10s}  {p.latency.to('ms'):>8.2f~}  {p.throughput.to('1/s').magnitude:>8.0f} img/s")
NVIDIA H100             Compute         1.04 ms     30750 img/s
NVIDIA A100             Compute         2.20 ms     14565 img/s
NVIDIA T4               Compute         8.60 ms      3719 img/s
NVIDIA Jetson Orin NX   Compute        21.69 ms      1475 img/s

Distributed Training Analysis

The DistributedModel models 3D parallelism (data, tensor, pipeline) with communication overhead from ring all-reduce and pipeline bubbles.

import mlsysim
from mlsysim.solvers import DistributedModel

dist = DistributedModel()
result = dist.solve(
    model=mlsysim.Models.Language.Llama3_70B,
    fleet=mlsysim.Systems.Clusters.Research_256,
    batch_size=512, precision="fp16",
    tp_size=8, pp_size=4, microbatch_count=16
)
print(f"Scaling efficiency:  {result.scaling_efficiency:.1%}")
print(f"DP all-reduce:       {result.dp_communication_latency.to('ms'):~.1f}")
print(f"TP overhead:         {result.tp_communication_latency.to('ms'):~.1f}")
print(f"Pipeline bubble:     {result.bubble_fraction:.1%}")
print(f"Step latency:        {result.step_latency_total.to('ms'):~.1f}")
Scaling efficiency:  64.3%
DP all-reduce:       618.5 ms
TP overhead:         668.7 ms
Pipeline bubble:     15.8%
Step latency:        5456.6 ms

Tune tp_size, pp_size, and microbatch_count to find the parallelism configuration that maximizes scaling efficiency for a given cluster topology.

Composing Solvers for Real Questions

The core solvers are designed to chain. Three common engineering workflows follow.

Serving Llama-70B on H100s within budget

import mlsysim
from mlsysim.solvers import ServingModel, EconomicsModel

# Step 1: Does it fit and what's the latency?
serving = ServingModel()
result = serving.solve(
    model=mlsysim.Models.Language.Llama3_70B,
    hardware=mlsysim.Hardware.Cloud.H100,
    seq_len=4096, batch_size=1
)

# Step 2: What does that fleet cost?
econ = EconomicsModel()
cost = econ.solve(
    fleet=mlsysim.Systems.Clusters.Research_256,
    duration_days=365,
    kwh_price=0.08
)
print(f"Annual TCO: ${cost.tco_usd:,.0f}")
print(f"  CapEx:    ${cost.capex_usd:,.0f}")
print(f"  OpEx:     ${cost.total_opex_usd:,.0f}")
Annual TCO: $2,593,987
  CapEx:    $2,133,333
  OpEx:     $460,653

Choosing a training region to minimize carbon

import mlsysim
from mlsysim.solvers import SustainabilityModel

sustain = SustainabilityModel()
for grid in [mlsysim.Infrastructure.Grids.Quebec, mlsysim.Infrastructure.Grids.US_Avg,
             mlsysim.Infrastructure.Grids.Poland]:
    r = sustain.solve(
        fleet=mlsysim.Systems.Clusters.Research_256,
        duration_days=30,
        datacenter=grid
    )
    carbon_tons = r.carbon_footprint.to("metric_ton").magnitude
    print(f"{grid.name:16s}  {carbon_tons:8.1f} tCO2e  "
          f"PUE={r.pue:.2f}  Water={r.water_usage_liters:,.0f} L")
Quebec (Hydro)         2.7 tCO2e  PUE=1.06  Water=0 L
US Average            62.0 tCO2e  PUE=1.12  Water=260,112 L
Poland (Coal)        167.2 tCO2e  PUE=1.58  Water=366,944 L

For the theory behind PUE, carbon intensity, and the energy hierarchy, see the Sustainable AI slide deck.

Reliability of a 30-day training run on 256 GPUs

import mlsysim
from mlsysim.solvers import ReliabilityModel

rel = ReliabilityModel()
result = rel.solve(
    fleet=mlsysim.Systems.Clusters.Research_256,
    job_duration_hours=720,       # 30 days
    checkpoint_time_s=120.0       # 2 minutes per checkpoint
)
print(f"Fleet MTBF:              {result.fleet_mtbf.to('hour'):~.1f}")
print(f"P(failure before done):  {result.failure_probability:.1%}")
print(f"Optimal ckpt interval:   {result.optimal_checkpoint_interval.to('minute'):~.1f}")
print(f"Expected failures:       {result.expected_failures:.1f}")
Fleet MTBF:              146.5 h
P(failure before done):  99.3%
Optimal ckpt interval:   187.5 min
Expected failures:       4.9

This solver implements the Young-Daly checkpoint model, which sets the checkpoint interval for long training jobs.

Writing Custom Solvers

Follow the built-in solver pattern to create a new analysis:

from mlsysim.hardware.types import HardwareNode

class PowerEfficiencyModel:
    def solve(self, hardware: HardwareNode) -> dict:
        flops_per_watt = hardware.compute.peak_flops / hardware.tdp
        return {
            "device": hardware.name,
            "flops_per_watt": flops_per_watt.to("TFLOPs/s/kW"),
        }

See Writing a Custom Solver for the full guide.

Type Safety

All quantities are pint.Quantity objects. Unit conversions are explicit, and dimensional errors are caught at runtime:

import mlsysim
hw = mlsysim.Hardware.Cloud.A100
print(hw.compute.peak_flops.to("TFLOPs/s"))
print(hw.memory.bandwidth.to("TB/s"))
try:
    hw.memory.bandwidth.to("second")
except Exception as err:
    print(f"{type(err).__name__}: bytes per second can never become seconds")
312.0 TFLOPs/s
2.039 TB/s
DimensionalityError: bytes per second can never become seconds

Pint catches physical-dimension mismatches (bytes per second can never become seconds). Since 2026-06, compute work also has its own pint dimension, so a bare .to("flop/s") on a bandwidth raises DimensionalityError directly, as does adding a FLOP rate to a memory bandwidth:

for label, attempt in [
    ("FLOP rate + bandwidth", lambda: hw.compute.peak_flops + hw.memory.bandwidth),
    ("bandwidth -> TFLOPs/s", lambda: hw.memory.bandwidth.to("TFLOPs/s")),
]:
    try:
        attempt()
        print(f"{label}: accepted (unexpected)")
    except Exception as err:
        print(f"{label}: {type(err).__name__}")
FLOP rate + bandwidth: DimensionalityError
bandwidth -> TFLOPs/s: DimensionalityError

Bytes, counts, parameters, and dollars remain dimensionless aliases, so MLSys·im additionally enforces unit-family checks at every schema boundary (require_unit_family in mlsysim.core.types) to keep those distinct. A field declared as memory bandwidth rejects a count rate, and vice versa.

Solver-to-Slides Map

Each MLSys·im solver maps to specific chapters and slide decks from the Machine Learning Systems textbook. Use these for the full theoretical grounding behind each solver.

MLSys·im Solver What It Models Slide Deck
Engine / SingleNodeModel Roofline analysis, compute vs. memory bottleneck Hardware Acceleration (Vol I, Ch 11)
ServingModel TTFT, ITL, KV-cache memory Model Serving (Vol I, Ch 13) and Inference at Scale (Vol II, Ch 10)
DistributedModel 3D parallelism, all-reduce, pipeline bubbles Distributed Training (Vol II, Ch 6) and Collective Communication (Vol II, Ch 5)
EconomicsModel CapEx, OpEx, total cost of ownership Compute Infrastructure (Vol II, Ch 2)
SustainabilityModel Energy, carbon footprint, water usage Sustainable AI (Vol II, Ch 15)
ReliabilityModel MTBF, Young-Daly checkpointing Fault Tolerance (Vol II, Ch 7)

Volume I: Introduction to Machine Learning Systems

Browse Vol I Decks | Download All (PDF)

Volume II: Scaling Machine Learning Systems

Browse Vol II Decks | Download All (PDF)

Next Steps

Back to top