Skip to main content

Getting Started

Install MLSys·im and run a first analysis in under 5 minutes.

NotePrerequisites

MLSys·im assumes basic Python familiarity (variables, functions, pip install). No prior ML or hardware knowledge is required. Key concepts like roofline analysis, memory-bound vs. compute-bound, and FLOP/s are explained in context throughout the tutorials. For a full reference of terms, see the Glossary.

Installation

MLSys·im requires Python 3.10+ and installs with pip:

pip install mlsysim

For development or to follow along with tutorials locally:

git clone https://github.com/harvard-edge/cs249r_book
cd cs249r_book/mlsysim
pip install -e ".[dev]"

Verify the installation:

python -c "import mlsysim; print(mlsysim.__version__)"
TipLocal install recommended

Tutorials are pure Python and run in any Python 3.10+ environment. The local workflow is the supported path for the current release and matches the commands used throughout the documentation.

A First Analysis

For a map of Engine, specialized solvers, and scenarios, see Choosing a Python API. The example below uses the generic single-device Roofline model.

Once MLSys·im is installed, a complete roofline analysis takes five lines. The roofline model is the foundation of ML systems performance reasoning. It determines whether a workload is limited by compute (arithmetic units) or memory (data movement). For a visual walkthrough, see the Hardware Acceleration slide deck (Vol I, Ch 11).

import mlsysim
from mlsysim import Engine

# 1. Load a model and hardware from the vetted Zoo
model    = mlsysim.Models.Vision.ResNet50
hardware = mlsysim.Hardware.Cloud.A100

# 2. Solve -- the Engine applies the roofline model
profile = Engine.solve(model=model, hardware=hardware, batch_size=1, precision="fp16", efficiency=1.0)

# 3. Read the results
print(f"Bottleneck: {profile.bottleneck}")
print(f"Latency:    {profile.latency.to('ms'):~.2f}")
print(f"Throughput: {profile.throughput.to('1/s'):~.0f}")
Bottleneck: Memory
Latency:    0.54 ms
Throughput: 1843 / s

The output above is produced by running the code when this page is built, so it always matches the installed version of MLSys·im.

NoteWorking with units

MLSys·im uses the Pint library for physical units. All quantities carry attached units (ms, GB, TFLOP/s, etc.). Use .to('ms') to convert between units. Use .magnitude to extract the raw number for calculations or plotting.

Understanding the Output

Engine.solve() returns a PerformanceProfile, a structured result containing everything the roofline model can determine about the workload.

Core fields

Field What it means
bottleneck 'Memory' or 'Compute': which resource limits performance
latency Time to process one batch: the roofline bound plus fixed overhead
throughput Samples per second = batch_size / latency
latency_compute Time if only compute were the constraint
latency_memory Time if only memory bandwidth were the constraint
latency_overhead Fixed dispatch and kernel-launch time added on top of the roofline bound
arithmetic_intensity Operations per byte, the x-axis of the roofline plot

Extended fields

Field What it means
energy Estimated energy consumption (Joules)
memory_footprint Total memory required for the workload
mfu Model FLOPs Utilization, the fraction of peak compute achieved
feasible Whether the workload fits in device memory
TipThe key insight

If latency_memory > latency_compute, the workload is memory-bound, and faster arithmetic units will not help. The remedies are to increase batch size, use a more compute-dense operation (e.g., fused attention), or reduce data movement. When the workload is compute-bound, parallelism and quantization pay off.

This is the same insight taught in the Neural Network Computation slides (Vol I, Ch 5) and the Performance Engineering slides (Vol II, Ch 9).

Exploring the Zoo

MLSys·im ships with vetted registries of hardware, models, infrastructure, and systems, with source metadata where available. Use tab-completion to explore.

Hardware

Five tiers spanning the full deployment spectrum:

# Cloud accelerators
mlsysim.Hardware.Cloud.A100
mlsysim.Hardware.Cloud.H100
mlsysim.Hardware.Cloud.H200

# Workstation / desktop GPUs
mlsysim.Hardware.Workstation.DGX_Spark

# Mobile processors
mlsysim.Hardware.Mobile.iPhone15Pro
mlsysim.Hardware.Mobile.Snapdragon8Gen3

# Edge devices
mlsysim.Hardware.Edge.JetsonOrinNX

# Tiny / microcontroller targets
mlsysim.Hardware.Tiny.ESP32_S3
mlsysim.Hardware.Tiny.HimaxWE1

For the theory behind this hardware spectrum, see the Compute Infrastructure slides (Vol II, Ch 2).

Models

Organized by application domain:

# Language models
mlsysim.Models.Language.GPT2
mlsysim.Models.Language.Llama3_8B
mlsysim.Models.Language.Llama3_70B

# Vision models
mlsysim.Models.Vision.ResNet50
mlsysim.Models.Vision.MobileNetV2
mlsysim.Models.Vision.AlexNet

# Tiny / edge models
mlsysim.Models.Tiny.DS_CNN
mlsysim.Models.Tiny.WakeVision

Infrastructure

Regional grids and datacenter configurations for sustainability analysis:

# Regional power grids -- carbon intensity varies by energy source
for grid in (
    mlsysim.Infrastructure.Grids.Quebec,
    mlsysim.Infrastructure.Grids.US_Avg,
    mlsysim.Infrastructure.Grids.Poland,
):
    print(f"{grid.name:28s} ({grid.primary_source}): {grid.carbon_intensity_g_kwh:.0f} gCO2/kWh")
Quebec (Hydro)               (hydro): 20 gCO2/kWh
US Average                   (mixed): 429 gCO2/kWh
Poland (Coal)                (coal): 820 gCO2/kWh

The Sustainable AI slides (Vol II, Ch 15) explain why datacenter location is a first-class engineering decision.

Systems

Cluster definitions for distributed analysis:

# Network fabrics
mlsysim.Systems.Fabrics.InfiniBand_NDR
mlsysim.Systems.Fabrics.Ethernet_100G

# Pre-configured clusters
mlsysim.Systems.Clusters.Frontier_8K
mlsysim.Systems.Clusters.Research_256

For the full topology and cluster modeling, see the Distributed Training slides (Vol II, Ch 6) and Network Fabrics slides (Vol II, Ch 3).

Support registries

Beyond the six primary zoos, MLSys·im exposes cited literature anchors, deployment envelopes, MLOps thresholds, and solver calibration knobs:

# Deployment paradigms (latency/RAM envelopes — not chip specs)
mlsysim.Platforms.Cloud.latency_range_ms
mlsysim.Platforms.Mobile.ram

# Cited appendix scalars (MFU bands, Chinchilla, scaling η)
mlsysim.Literature.Chinchilla.TokensPerParam
mlsysim.Literature.Training.MfuHigh

# MLOps drift thresholds
mlsysim.Ops.Monitoring.PsiWarnThreshold

# Reusable scenario anchors
mlsysim.ReferenceStats.ClinicalImaging.RetinalPhotoSize
mlsysim.ReferenceStats.StorageTrainingCorpus.TrainingTokens

# Composed physical systems
mlsysim.Systems.Nodes.DGX_H100
mlsysim.Systems.Racks.DGX_H100_4Node
mlsysim.Systems.Clusters.Frontier_8K

# Solver/engine defaults (not hardware specs)
mlsysim.engine.calibration.REFERENCE_MFU_SUSTAINED
ImportantCanonical paths in Python

Always use nested registry paths in Python code (Hardware.Cloud.H100, Models.Vision.ResNet50). The CLI still accepts short names like Llama3_8B and H100. See Provenance & registry paths.

Complete registry listings are available in the Zoo reference pages.

Adjusting the Efficiency Parameter

The efficiency parameter (\(\eta\)) is the parameter that most affects MLSys·im results. It represents the fraction of theoretical peak hardware performance that is actually achieved in practice. Unoptimized single-request inference often runs near 5% of peak, whereas well-tuned training and batched serving workloads reach 30%–50%.

# Default: well-optimized training (η = MfuHigh)
profile_default = Engine.solve(
    model=model, hardware=hardware,
    batch_size=32, precision="fp16", efficiency=mlsysim.Literature.Training.MfuHigh
)

# Conservative: typical inference workload (η = 0.35)
profile_inference = Engine.solve(
    model=model, hardware=hardware,
    batch_size=32, precision="fp16", efficiency=0.35
)

print(f"Training estimate:  {profile_default.latency.to('ms'):~.2f}")
print(f"Inference estimate: {profile_inference.latency.to('ms'):~.2f}")
Training estimate:  2.20 ms
Inference estimate: 2.92 ms

Typical efficiency ranges:

Scenario \(\eta\) range Notes
Well-optimized training (fp16) 0.30–0.50 Megatron-LM, DeepSpeed
Inference (fp16) 0.25–0.40 vLLM, TensorRT-LLM
Inference (int8) 0.20–0.40 Quantized serving

See the Accuracy & Validation page for guidance on choosing η for different scenarios. The gap between theoretical peak and achieved throughput is covered in detail in the Performance Engineering slides (Vol II, Ch 9).

Defining Custom Models

Analyses are not limited to Zoo models. Define any model by specifying its parameters and FLOPs:

from mlsysim import ureg
from mlsysim.models.types import TransformerWorkload

my_model = TransformerWorkload(
    name="My-Custom-LLM",
    architecture="Transformer",
    parameters=13e9 * ureg.param,
    layers=40,
    hidden_dim=5120,
    heads=40,
    kv_heads=8,
    inference_flops=2 * 13e9 * ureg.flop  # Rule of thumb: 2 FLOPs per parameter
)

profile = Engine.solve(model=my_model, hardware=hardware, batch_size=1)
print(f"Bottleneck: {profile.bottleneck}")
print(f"Latency:    {profile.latency.to('ms'):~.2f}")
print(f"Feasible:   {profile.feasible}")
Bottleneck: Memory
Latency:    14.44 ms
Feasible:   True

The Model Compression slides (Vol I, Ch 10) explain why parameter count and precision together determine both the memory footprint and the arithmetic intensity of a workload.

Companion Slide Decks

MLSys·im is the hands-on companion to the Machine Learning Systems textbook. The concepts modeled with MLSys·im are taught in the Beamer lecture decks for Volumes I and II, which carry speaker notes and in-class exercises.

Concept in MLSys·im Slide Deck Key Topics
Engine.solve() and the roofline model Hardware Acceleration (Vol I, Ch 11) Roofline model, arithmetic intensity, systolic arrays, memory wall
FLOPs, MACs, and compute cost Neural Network Computation (Vol I, Ch 5) Forward/backward pass cost, training memory breakdown
Training memory and mixed precision Model Training (Vol I, Ch 8) Iron Law of Training, gradient checkpointing, mixed precision
Quantization and compression Model Compression (Vol I, Ch 10) Pruning, quantization, knowledge distillation
Hardware Zoo tiers Compute Infrastructure (Vol II, Ch 2) Accelerator spectrum, HBM architecture, TCO
DistributedModel Distributed Training (Vol II, Ch 6) 3D parallelism, scaling efficiency, communication overhead
ServingModel and LLM inference Model Serving (Vol I, Ch 13) TTFT, ITL, KV-cache, batching strategies
SustainabilityModel Sustainable AI (Vol II, Ch 15) Energy wall, carbon geography, PUE
Efficiency parameter (η) Performance Engineering (Vol II, Ch 9) Operator fusion, FlashAttention, precision engineering
Benchmarking and validation Benchmarking (Vol I, Ch 12) MLPerf, measurement methodology, latency percentiles

Volume I: Introduction to Machine Learning Systems

Browse Volume I on the slide website

Volume II: Scaling Machine Learning Systems

Browse Volume II on the slide website

Next Steps

TipRecommended path

Follow the structured learning path on the Tutorials page, starting with the Hello, Roofline Tutorial. Each tutorial pairs with a companion slide deck for visual explanations and active learning exercises.

For a complete reference of which solver to use for different questions, see the Solver Guide.

Back to top