Physical AI Systems: Course Syllabus
ETH Zurich · Spring 2027 · Draft
Course: Physical AI Systems: Machine Learning Systems That Sense and Act
Course numbers: Harvard CS/EE 288 · ETH Zurich 151-0851 (to confirm)
Instructor: Prof. Vijay Janapa Reddi
Teaching assistant: Dr. Andrea Mattia Garavagno
Textbook: Physical AI: Machine Learning Systems That Sense and Act (MLSysBook Volume IV)
Status: Draft. Dates, rooms, and policies marked TBD are set before the term.
1. Course Description
A robot changes the physical world through motor commands. An incorrect command can cause a collision, so the system must constrain motion as well as predict actions. This course examines how to build, deploy, evaluate, and govern such a system. Teams run a learned policy (SmolVLA or ACT) on the Qualcomm Linux processor of an Arduino UNO Q. An independent STM32 safety governor checks whether to permit each command to a Seeed SO-101 arm. Over fourteen weeks, students derive the arm’s kinematics, train and evaluate its policy, and use their measurements to defend a bounded release decision.
2. Course at a Glance
| Format | Two meetings a week: a 90-minute systems seminar and a 3-hour supervised lab. Six labs, then a three-week capstone. No written exam. |
| Length | 14 teaching weeks |
| Prerequisites | Programming in Python, basic linear algebra and probability, introductory machine learning, and first-year Newtonian physics. C, operating systems, or control experience helps but is not required; staff provide the baseline firmware and support its extension. |
| Teams | Two or three students per bench station, with rotating roles |
| Hardware | One station per team: Arduino UNO Q, SO-101 arm, overhead camera, and separate motor power. See the Station Reference Card. |
| Workload | Seminar and lab each week, plus reading, analysis, and booked bench time. Credits: TBD |
| Schedule | Seminar early in the week (Monday or Tuesday), lab later in the week (Thursday or Friday). Times and rooms: TBD |
3. Learning Outcomes
By the end of the course, students should be able to:
- Draw a complete path from sensing to actuation and identify which component proposes, which permits, and which executes each physical action.
- Calibrate sensors, measure actuator limits, safe states, and end-to-end latency, and set an operating envelope from those measurements.
- Compare a simulation with the real hardware and explain where and why they disagree.
- Collect and audit robot demonstrations, train and deploy a learned policy on edge hardware, and compare it with a nonlearned baseline.
- Distinguish offline model metrics from physical task outcomes, including recovery, abstention, and incomplete motion.
- Build and attack a real-time safety governor, and defend a bounded release or no-release decision with evidence from their experiments.
The twelve competencies on the Competency Card define the skills that staff assess at the bench.
Before week 1. Read the Volume IV reader guide and prerequisites and Chapter 1. The book’s appendices provide optional review material on machine learning, systems, and control. Students are not expected to master the appendices before the course begins.
4. Weekly Structure
Each week connects a chapter reading to a seminar discussion and a supervised experiment. The reading introduces the principles, the seminar examines their implications, and the lab tests them on the station.
Seminar (early in the week, about 90 minutes). Students complete the assigned reading before the seminar. The instructor poses a physical AI question from that reading, and a rotating team presents a raw trace or failed prediction from its station. The class evaluates what the trace supports and considers alternative explanations. The seminar ends by defining the experiment each team will run in the lab.
Lab (later in the week, about 3 hours). The teaching assistant reviews the station’s safe state, motion limits, and the measurement required that week. Teams then complete the assigned lab part. Each part ends with a checkpoint, an artifact that staff inspect before the team proceeds.
Preparation and records. Before each lab, students record their predictions. After the experiment, teams add the raw trace, measurements, differences from the prediction, and the next planned experiment to their shared notebook. Staff check the notebook at the bench. Its records support the lab deliverables; separate weekly lab reports are not required.
5. Schedule
Students complete each week’s reading before the seminar. Readings follow the book in order, so some chapters prepare students for labs scheduled a week or two later. The lab template defines the structure of each lab.
| Week | Reading | Lab part | Due |
|---|---|---|---|
| 1 | Ch 1: The Causal Boundary | Getting Started | |
| 2 | Ch 2: The Physical Body | 1A Measure the body | |
| 3 | Ch 3: The Cognitive Brain | 1B Build the digital twin | Lab 1 |
| 4 | Ch 4: The Nervous System | 2A IK on the real arm | Lab 2 |
| 5 | Ch 5: Physical Data | 3A VLAs in simulation | |
| 6 | Ch 6: Policy Training; read ahead: calibration in Ch 8 | 3B The VLA on real hardware | Lab 3 |
| 7 | Ch 7: Closed-Loop Evaluation | 4A Real demonstrations | |
| 8 | Ch 8: Sensor Perception, Ch 9: Spatial Memory | 4B Fine-tuned & untethered | Lab 4, capstone proposal |
| 9 | Ch 10: Grounded Intent, Ch 11: Trajectory Planning | 5A Stress the loop | Lab 5 |
| 10 | Ch 12: Safety Enforcement, Ch 13: Silicon Placement | 6A Build the governor | Capstone task frozen |
| 11 | Ch 14: Supervisory Intervention, Ch 15: Adversarial Verification | 6B Break the governor | Lab 6 |
| 12 | Ch 16: Safe Release | C1 Capstone task build | |
| 13 | Ch 17: Responsible Physical AI | C2 Peer fault exchange | |
| 14 | Ch 18: The Epistemic Frontier | C3 Final trials and defense | Capstone |
6. The Capstone
Each team defends a release decision for the system it builds. After establishing a working autonomous loop, the team proposes a task at the end of week 8 and fixes its scope at the end of week 10. During weeks 12–14, teams implement the task, test the system against a peer team’s approved faults, and conduct final trials. Each student then defends the team’s Physical Release Dossier in an individual oral exam. A team that demonstrates with rigorous evidence that its system should not be released can earn full marks. The capstone brief specifies the deliverables.
7. Assessment, Milestones, and Grading
Grades are earned through witnessed bench demonstrations, lab deliverables, and the capstone defense. Labs 1–2 and Labs 3–4 are graded together because each pair builds one working system.
| Graded work | Due | Weight | Witnessed Evidence | Competencies |
|---|---|---|---|---|
| Labs 1–2: Station & envelope | End of Weeks 3 and 4 | 15% | Getting Started and Labs 1–2. Safe-state check, the body’s measured limits, a digital twin driven by the team’s derived kinematics, and the simulation-to-hardware gap measured with the team’s inverse kinematics on the real arm. | A1; A3 timing foundation |
| Labs 3–4: Learned policy on the edge | End of Weeks 6 and 8 | 25% | Labs 3–4. Camera calibration, training/validation curves, a sim-trained VLA deployed with a measured sim2real gap, a real LeRobot dataset with all four action taps, and a fine-tuned policy running untethered on the UNO Q. | A2, A3 (complete), B1, B2, B3, C1 |
| Lab 5: Stress-tested loop | End of Week 9 | 10% | Lab 5. Chunk-horizon benchmark, live recovery from a displaced target, and a language check. | C2, C3 |
| Lab 6: Safety governor | End of Week 11 | 25% | Lab 6. The team’s governor clamps or vetoes proposals outside the declared checked limits; staff-approved fault trials measure its response against declared limits, including the watchdog and stale-command rejection on resume. | D1, D2 |
| Capstone: Release & defense | End of Week 14 | 25% | Capstone. Proposal and frozen task, then live held-out trials under disturbance, baseline comparison, oral defense, and the Physical Release Dossier. | D3 (full card) |
Three principles govern grading:
- Evidence. A favorable model metric does not compensate for a bypassed safety path or unexplained motion. Staff may require a repair and a repeated trial before a team moves on.
- Negative results. A negative result supported by measurements and a sound explanation earns more credit than an unmeasured success.
- Individual accountability. Teams submit one system, but each student explains their own measurements and a randomly selected trace at the final defense.
The notebook, seminar participation, and the week-13 rehearsal receive feedback but are not graded separately.
8. Course Policies
Collaboration. Limited bench hardware requires students to complete lab work in teams. Students may discuss ideas across teams, but each student must be able to explain their team’s work at the final defense.
Use of AI tools. Students may use AI tools without penalty. The notebook or deliverable must identify the tools used and describe how they were used. Each student must explain the work without those tools at the final defense.
Late work and repairs. Each lab depends on earlier results, so delays can prevent a team from completing the next experiment. The following rules govern extensions, repairs, and late milestones:
- Equipment failures. A team receives an automatic extension for failures outside its control, such as a failed servo or broken station.
- Missed checkpoints. A team may complete a missed checkpoint during booked bench time before the next lab. If the team falls further behind, staff provide the reference artifact so that it can continue.
- Repairs. Staff may require a correction and a repeated trial before approving a result. A repair finished within a week earns full credit.
- Milestones. Late milestones otherwise lose 10% per day unless the team arranges an extension before the deadline.
Accommodations. Students with approved accommodations should contact the instructor in the first week. Lab assessments require physical demonstrations at the bench. Approved lab accommodations provide extra time for checkpoints and milestones.
Office hours. TBD