Checkpoint Roulette

Fault tolerance and checkpointing at scale.

Hold Space to train, release to save a checkpoint, and reach 100% before the minute ends.

progress 0% hold Space to train · release to save · 60 seconds

How to play

Press Enter or tap the game to start. Hold Space or the Hold to train button to advance. Release to write a checkpoint, which pauses training for 1.1 seconds. A green marker shows the last completed checkpoint. If a node fails while you are training or writing, progress returns to that marker. Reach 100% within 60 seconds. Press R or tap the canvas to retry.

The Systems Concept

At large scale, hardware failures are inevitable. Frequent checkpoints limit lost work, but writing them consumes I/O bandwidth and pauses training. A checkpoint only protects work after its write completes. The game compresses this tradeoff into a one-minute scheduling decision.

---

Part of MLSysBook Playground. Found a bug? Report an issue.