Scaling Machine Learning Systems

Syllabus · Semester plan

Textbook Volume II: Scaling Machine Learning Systems
Prerequisites The introductory course or equivalent systems background
Format A 16-week semester or a 10-week quarter, with two 75-minute lectures per week
Materials Slides · Labs · no cluster needed

Description

Students study how distributed training, communication, fault recovery, and serving work across many machines. The course ends with a design project that students defend in a written report.

Learning outcomes

By the end of the course, students can:

  • Choose a parallelism strategy for a training workload and justify it with measurements.
  • Explain how a training job recovers its progress after failures.
  • Size a serving system for a stated demand, cost, and energy budget.
  • Defend a fleet design in a written report.

Reading schedule

In the semester plan, students read one Volume II chapter each week; the quarter plan reads related chapters together. Each chapter has a matching lab, and the instructor decides which labs to assign.

Week Topic Reading Slides
Part I: The Fleet
1 The Scale Imperative Introduction to Scale Ch 1
2 Compute Infrastructure Compute Infrastructure Ch 2
3 Network Fabrics Network Fabrics Ch 3
4 Distributed Storage Data Storage Ch 4
Part II: Distributed Algorithms
5 Collective Communication Collective Communication Ch 5
6 Distributed Training Distributed Training Ch 6
7 Fault Tolerance Fault Tolerance Ch 7
8 Fleet Orchestration Fleet Orchestration Ch 8
Part III: Deployment at Scale
9 Performance Engineering Performance Engineering Ch 9
10 Inference at Scale Inference at Scale Ch 10
11 Edge Intelligence Edge Intelligence Ch 11
12 Operations at Scale Ops at Scale Ch 12
Part IV: The Responsible Fleet
13 Security and Privacy Security & Privacy Ch 13
14 Robust AI Robust AI Ch 14
15 Sustainable and Responsible AI Sustainable AI and Responsible AI Ch 15 · Ch 16
16 Capstone: Fleet Synthesis Conclusion Ch 17

Every chapter is kept. Related chapters are read together, and distributed training, performance engineering, and inference keep a week of their own.

Week Topic Reading Slides
1 The Scale Imperative Introduction to Scale Ch 1
2 Compute Infrastructure + Network Fabrics Compute Infrastructure · Network Fabrics Ch 2 · Ch 3
3 Distributed Storage + Collective Communication Data Storage · Collective Communication Ch 4 · Ch 5
4 Distributed Training Distributed Training Ch 6
5 Fault Tolerance + Fleet Orchestration Fault Tolerance · Fleet Orchestration Ch 7 · Ch 8
6 Performance Engineering Performance Engineering Ch 9
7 Inference at Scale Inference at Scale Ch 10
8 Edge Intelligence + Operations at Scale Edge Intelligence · Ops at Scale Ch 11 · Ch 12
9 Security and Privacy + Robust AI Security & Privacy · Robust AI Ch 13 · Ch 14
10 Sustainable and Responsible AI + Capstone: Fleet Synthesis Sustainable AI and Responsible AI · Conclusion Ch 15 · Ch 16 · Ch 17

Assessment

A suggested split; each offering sets its own.

Component Weight
Lab Decision Logs (Labs 01–16) 40%
Design challenges (one per part) 30%
Capstone: a fleet design with evidence, a report, and a presentation 30%

Suggested papers

Optional readings that pair with the chapters, for a seminar or a graduate section.

Week Topic Paper
1 Scale Imperative Kaplan et al., “Scaling Laws for Neural Language Models” (2020)
5 Collective Comms Narayanan et al., “Efficient Large-Scale Language Model Training on GPU Clusters” (2021)
6 Distributed Training Shoeybi et al., “Megatron-LM: Training Multi-Billion Parameter Language Models” (2020)
7 Fault Tolerance Jeon et al., “Analysis of Large-Scale Multi-Tenant GPU Clusters” (ATC 2019)
10 Inference at Scale Kwon et al., “Efficient Memory Management for LLM Serving with PagedAttention” (SOSP 2023)
12 Ops at Scale Zhao et al., “Characterizing GPU Cluster Failures at Scale” (ATC 2024)
15 Sustainability Patterson et al., “Carbon Emissions and Large Neural Network Training” (2021)

Adapting this syllabus

Back to top