🚧 DEVELOPMENT PREVIEW - Built from dev@2c8f1297 • 2026-11-12 08:41 EST • Stable version →
MLSys·im: analytical models of ML training and inference. The books: Introduction to Machine Learning Systems · Scaling Machine Learning Systems (preview) · Agentic Machine Learning Systems (in preparation) · Physical AI (in preparation). All are open access. Companion resources: TinyTorch (build) · Hardware Kits (deploy) · MLSys·im (model) · Labs (explore) · Lecture Slides (teach) · Instructor Hub (teach) · StaffML (practice) · Newsletter
Analytical models of ML systems
Performance, cost, and carbon, estimated from first principles before any hardware is provisioned.
Open source · v0.3.1 · Companion to the Machine Learning Systems textbook
MLSys·im is an analytical modeling framework for the stage before hardware is benchmarked or provisioned. It makes the first-order constraints explicit (memory, compute, network, data, reliability, cost, and carbon), which narrows the design space and shows which experiments are worth running.
pip install mlsysim
372 physics formulas
278 solvers and optimizers
103 sourced hardware specs
44 reference workloads
Each of the four questions below maps to a solver. The analytical answer comes first; the sensitive parameters are then validated on the real stack once hardware is available.
Memory fit
Estimate weights, activations, optimizer state, KV cache, and communication buffers before a job fails at runtime.
The binding constraint
Separate compute, memory bandwidth, network communication, data input, reliability, cost, and carbon constraints.
Serving capacity
Size serving replicas for a QPS and p99 target, including batching and queueing assumptions.
Design-space sweeps
Sweep batch size, precision, parallelism, efficiency, geography, and hardware without needing cluster access.
from mlsysim import Engine, Models, Hardware
profile = Engine.solve(
model=Models.Vision.ResNet50,
hardware=Hardware.Cloud.A100,
batch_size=1,
precision="fp16",
efficiency=1.0,
)
print(f"Bottleneck: {profile.bottleneck}")
print(f"Latency: {profile.latency.to('ms'):~.2f}")
print(f"Throughput: {profile.throughput.to('1/s'):~.0f}")Bottleneck: Memory
Latency: 0.54 ms
Throughput: 1843 / sThe engine applies the roofline model [1]: a step takes at least as long as its compute time or its memory-transfer time, whichever is larger.
\[T = \max\!\left(\frac{\text{FLOPs}}{\text{Peak} \times \eta},\ \frac{\text{Bytes}}{\text{BW}}\right) + T_{\text{overhead}}\]
At batch 1 on the theoretical ceiling (\(\eta = 1.0\)), ResNet-50’s arithmetic intensity is 146 FLOP/byte, just under the A100’s ridge point of 153 FLOP/byte. The memory term (0.028 ms) edges out the compute term (0.026 ms), though both are small next to the fixed dispatch overhead of 0.515 ms. The 0.54 ms total is mostly overhead, which is why batching helps so much here. Start from the Model Zoo and Silicon Zoo, or define a custom workload and hardware target.
Every solver takes typed registry objects and returns an analytical estimate with units attached.
Roofline analysis
Compute versus memory bottlenecks, single-node latency and throughput.
LLM serving
Time to first token, inter-token latency, and KV-cache pressure. Llama 3 8B on an H100 with a 2,048-token prompt: 71 ms to first token, 5.2 ms per token after that.
Memory and capacity
Training memory breakdowns, serving replicas, and MoE routing imbalance. The same Llama 3 8B needs 15 GiB of weights plus 0.25 GiB of KV cache, out of 80 GiB on the H100.
3D parallelism
Data, tensor, and pipeline parallel scaling, ring all-reduce, and pipeline bubbles.
Total cost of ownership
Capital and operating cost, electricity, and per-query economics over any horizon.
Sustainability
Energy, carbon footprint, and water use across datacenter regions.
Reliability
Fleet MTBF, failure probability, and the Young-Daly checkpoint interval.
Physical AI
Stopping distances, information age, actuator limits, and fieldbus budgets for robots.
The outputs are analytical estimates, not measurements. Use them to narrow the design space, expose assumptions, and choose the next experiment.
efficiency parameter. The defaults are literature-informed starting points; replace them with measured MFU or sustained throughput once those measurements exist.See the accuracy and validation guide for how the solvers compare with published measurements.
Beginner
Memory-bound versus compute-bound in a few lines of Python. Sweep batch size and find the crossover.
Beginner
Why most LLM inference is limited by memory bandwidth rather than peak FLOP/s.
Intermediate
Prefill is compute-bound and decode is memory-bound. Model both and diagnose KV-cache pressure.
Intermediate
Training memory, serving replicas, and sparse expert routing with explicit assumptions.
Advanced
Ring all-reduce, pipeline bubbles, and scaling efficiency on GPU clusters.
Advanced
Perturb hardware parameters and find which constraint is actually worth improving.
Build intuition for why ML systems behave as they do. Run roofline analysis, see the memory wall, and compute carbon footprints, all without GPU hardware.
Assign problem sets with deterministic, reproducible outputs, and pair them with the lecture slide decks.
Pre-deployment estimates for distributed training, LLM serving, and multi-region sustainability before any hardware is provisioned.
MLSys·im pairs with the Machine Learning Systems textbook. The lecture slides for Volume I and Volume II cover the concepts; the MLSys·im tutorial slides walk from a single-node roofline to fleet-scale carbon analysis. For course planning, see the teaching guide.
Coursework, research, or infrastructure analysis that uses MLSys·im should cite it as shown below. The textbook has its own citation for work that draws on the book or the course materials.
A four-volume open textbook on machine learning systems.
Beamer lecture decks with speaker notes for each volume and TinyML.
Interactive Marimo notebooks that measure the book's claims.
Hands-on embedded ML deployment on real devices.
Course maps, syllabi, and adoption resources for teaching.
A machine learning framework built from scratch, one module at a time.
The analytical modeling engine behind the book's quantitative figures.