Reliability Foundations

Individual accelerators fail rarely, but scaling to thousands of nodes multiplies component failure rates until interruption becomes a continuous operating condition. At cluster scale, mean time between failures (MTBF) drops from years to hours, turning failure recovery into a core execution path. This appendix collects the quantitative reliability calculations behind the book’s fault-tolerance chapters, deriving failure probabilities, Young-Daly optimal checkpoint intervals, and availability stacking models. In C\(^3\) terms, it develops the coordination axis, where checkpointing, recovery, and availability determine how much of the fleet’s compute and communication becomes useful work.

How to Use This Appendix

This appendix serves as a reference for moving from intuition (“failures happen more often at scale”) to quantitative engineering decisions (“how often should one checkpoint?” or “how many spare nodes are needed?”). It translates component-level reliability metrics into cluster-wide checkpoint cadences and availability bounds.

The worked examples in this appendix use a 10,000-GPU training fleet, a larger variation on the 1,024-GPU Archetype A reference cluster, unless noted otherwise. The questions below map common operational problems to the sections that develop their analytical models:

• “How often will something fail?” Start with section 1.1 and the MTBF cascade in section 1.1.3.
• “Where does the \(1/\RequiredComponentCount\) come from?” The probability tools in section 1.1.1 derive it from independent exponential lifetimes.
• “How often to checkpoint?” Use the Young-Daly model in section 1.2.1 and the worked example in section 1.2.3.
• “How much time is lost to recovery?” See the recovery anatomy in section 1.3.1 and the runtime goodput loss in section 1.3.2.
• “When to use redundancy vs. checkpointing?” Compare strategies in section 1.4.1 and availability stacking in section 1.4.2.
• “What if checkpoints are not cheap relative to MTBF?” Use the higher-order interval in section 1.5.
• “How can one flipped bit derail training?” See the silent data corruption model in section 1.6.

Failure Probability at Scale

Consider a training run on a 10,000-GPU cluster running for 30 days. The probability that at least one modeled node component fails during that time is effectively 100 percent. That answer determines whether the system needs fault tolerance as a core design requirement or as an optional feature.

Individual hardware components are reliable. A data center-grade GPU operates for tens of thousands of hours before failing. The physics of large-scale systems, however, works against reliability: every additional component is another opportunity for failure, and the aggregate failure rate scales linearly with component count. Component-level failure rates aggregate systematically into cluster-wide failure predictions.

Probability tools for failure analysis

A 10,000-GPU training job requires every accelerator and network card to stay healthy at the same time. At a 99.99 percent per-device daily survival rate, the job still has less than a 40 percent chance of finishing a single day uninterrupted. Four results from elementary probability carry that failure arithmetic throughout this book. Each assumes that components fail independently; correlated failures, such as a shared power feed or a common software bug, violate that assumption and must be modeled separately.

The first is the product rule for independent events. If a job needs all \(\RequiredComponentCount\) of its components and each is available with probability \(\ComponentAvailability\), the job’s components are all available with probability

\[ \JobAvailability= \ComponentAvailability^{\RequiredComponentCount} \approx e^{-\RequiredComponentCount(1-\ComponentAvailability)} \quad \text{for } \ComponentAvailability \text{ close to } 1 \tag{1}\]

Equation 1 is the reliability collapse behind the reliability gap in The reliability gap. A per-component availability that looks excellent in isolation decays exponentially in \(\RequiredComponentCount\).

The second is its complement, the probability that at least one of \(\RequiredComponentCount\) independent events occurs when each has probability \(\IndependentEventProbability\):

\[ \Pr(\text{at least one}) = 1-(1-\IndependentEventProbability)^{\RequiredComponentCount} \approx \RequiredComponentCount \IndependentEventProbability\quad \text{when } \RequiredComponentCount \IndependentEventProbability\ll 1 \tag{2}\]

The linear approximation in equation 2 holds only while \(\RequiredComponentCount \IndependentEventProbability\) is small; as \(\RequiredComponentCount\) grows, the probability approaches 1. This arithmetic is why one slow worker paces a large synchronous job. A delay that rarely strikes any one worker strikes some worker of a large synchronous job on almost every step.

The third is the exponential lifetime model. A component with constant failure rate \(\ComponentFailureRate\) survives to time \(\TimeInstant\) with probability \(e^{-\ComponentFailureRate \TimeInstant}\) and has mean time to failure \(1/\ComponentFailureRate\). The model is memoryless. A component that has run for a year is no more likely to fail in the next hour than a new one, which is why it describes the useful-life phase and not burn-in or wear-out. The earliest of \(\RequiredComponentCount\) independent exponential failures is itself exponential with rate \(\RequiredComponentCount\ComponentFailureRate\), so the time to the first failure among \(\RequiredComponentCount\) identical components has mean \(1/(\RequiredComponentCount\ComponentFailureRate)\). That single fact is the origin of the \(1/\RequiredComponentCount\) scaling of system MTBF in equation 5.

The fourth is the Poisson count. Under the same model, the number of failures in an interval \(\FailureObservationInterval\) is Poisson distributed with mean \(\FailureObservationInterval/\MeanTimeBetweenFailures\), so the probability of no failure is \(e^{-\FailureObservationInterval/\MeanTimeBetweenFailures}\). Equation 6 below is the complement of that event, and the expected number of failures per day in table 2 is the Poisson mean.

Component failure rates

A single data center GPU fails roughly once every 5.7 years of continuous operation, a rate of about 0.00002 failures per hour that is too small to write conveniently in engineering tables. Reliability engineers therefore characterize components using two complementary metrics. The failure in time (FIT) rate counts failures per \(10^9\) device-hours of operation, a unit chosen because individual components fail so rarely that failures-per-hour would produce inconveniently small numbers. The reciprocal quantity, mean time to failure (MTTF), calculates average lifetime in hours (equation 3):

\[ \MeanTimeToFailure = \frac{10^9}{\text{FIT}} \tag{3}\]

Reference FIT rates and MTTF values (table 1) characterize component reliability in typical GPU training nodes, grounded in large-fleet and warehouse-scale experience (Kokolis et al. 2025; Zu et al. 2024; Barroso et al. 2019). These values assume the steady-state “useful life” phase of the bathtub curve, where the failure rate is approximately constant, dominated neither by infant mortality (early life) nor by wear-out (end of life) (Klutke et al. 2003).

Klutke, G., P. C. Kiessler, and M. A. Wortman. 2003. “A Critical Look at the Bathtub Curve.” IEEE Transactions on Reliability 52 (1): 125–29. https://doi.org/10.1109/tr.2002.804492.

Table 1: Component Failure Rates: Order-of-magnitude reference FIT/MTTF in the steady-state useful-life phase. Informed by large-GPU research-cluster analysis (Kokolis et al. 2025), TPUv4 supercomputer resiliency and operations (Zu et al. 2024), and warehouse-scale machine design (Barroso et al. 2019).

Kokolis, Apostolos, Michael Kuchnik, John Hoffman, Adithya Kumar, Parth Malani, Faye Ma, Zach DeVito, Shubho Sengupta, Kalyan Saladi, and Carole-Jean Wu. 2025. “Revisiting Reliability in Large-Scale Machine Learning Research Clusters.” 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), 1259–74. https://doi.org/10.1109/hpca61900.2025.00096.

Zu, Y., A. Ghaffarkhah, H.-V. Dang, B. Towles, S. Hand, S. Huda, A. Bello, et al. 2024. “Resiliency at Scale: Managing Google’s TPUv4 Machine Learning Supercomputer.” 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), 761–74.

Barroso, Luiz André, Urs Hölzle, and Parthasarathy Ranganathan. 2019. The Datacenter as a Computer: Designing Warehouse-Scale Machines. Synthesis Lectures on Computer Architecture. Springer International Publishing. https://doi.org/10.1007/978-3-031-01761-2.

Component	FIT rate	MTTF	MTTF (years)	Typical failure mode
GPU	20,000	50,000 hours	5.7 years	Die defect, thermal fatigue
HBM	5,000	200,000 hours	22.8 years	Bit-flip accumulation, TSV
NIC	6,666.7	150,000 hours	17.1 years	Transceiver degradation
PSU	10,000	100,000 hours	11.4 years	Capacitor aging
PCIe Switch	5,000	200,000 hours	22.8 years	Solder joint, ESD damage
Optical Cable	20,000	50,000 hours	5.7 years	Fiber bend, connector wear
ToR Switch	3,333.3	300,000 hours	34.2 years	ASIC failure, fan bearing

Each component in isolation appears highly reliable; a GPU lasts 5.7 years on average. The difficulty begins with how a node behaves when many such components operate simultaneously.

The MTBF cascade

If one GPU or one network card inside an eight-GPU server fails during a synchronous collective, the entire server drops out of the training step just as surely as if its motherboard had lost power. The following model therefore treats a compute node as a series system, in which any modeled component failure interrupts it. A component enters the sum only when its failure alone stops the node, so redundant power supplies, spare links, and recoverable device errors stay out of it. For independent components with constant failure rates, equation 4 sums the individual rates into the node-level failure rate:

\[ \frac{1}{\NodeMtbf} = \frac{\AcceleratorsPerNode}{\GpuMttf} + \frac{\NicsPerNode}{\NicMttf} + \frac{\SeriesPsuCount}{\PsuMttf} + \cdots \tag{4}\]

Think of each component as a ticking clock counting down to failure. A DGX H100 node with 8 GPUs and 8 network interface cards (NICs) has 16 independent clocks, and the node fails when the first clock reaches zero. More clocks mean a shorter expected wait. The node’s 6 power supply units (PSUs) add no clocks. They run as a 4+2 redundant set, so the node keeps full performance with 2 of them failed, and a failed unit is normally replaced long before enough others fail to stop the node.

For a cluster of \(\ClusterNodeCount\) identical nodes, the same series-system logic applies one level up. As equation 5 shows, cluster MTBF divides the single-node MTBF by the number of nodes:

\[ \SystemMtbf = \frac{\NodeMtbf}{\ClusterNodeCount} \tag{5}\]

Reliability degrades linearly with component count at each level, and the levels compound, which is the MTBF cascade. A node with 4,687.5 hours MTBF sounds reliable. A cluster of 1,250 such nodes has an MTBF of 3.75 hours, so a failure every few hours is the expected steady state. Table 2 carries that arithmetic across fleet sizes, with aggregate MTBF falling in inverse proportion to GPU count.

Table 2: Cluster MTBF by Scale: As cluster size grows, the aggregate MTBF shrinks proportionally. At 10,000 GPUs, failures occur every few hours; at 100,000 GPUs, they occur continuously. Node configuration (DGX H100): 8 GPUs and 8 NICs per node in the series sum; its 6 PSUs are 4+2 redundant and add no term.

Cluster GPUs	Nodes	Cluster MTBF	Expected failures/day
256	32	146.5 hours	0.2
1,024	128	36.6 hours	0.7
2,048	256	18.3 hours	1.3
8,192	1,024	4.6 hours	5.2
10,000	1,250	3.8 hours	6.4
100,000	12,500	22.5 minutes	64

The transition from hundreds of GPUs to tens of thousands is a qualitative regime shift. About 6.1 days pass between failures at 256 GPUs. A 10,000-GPU cluster, in contrast, expects multiple failures per shift. At 100,000 GPUs, failures become a continuous background condition in which the system is never fully healthy.

Probability of failure during a job

A cluster MTBF of a few hours tells an operator how often nodes drop out on average, but a training team scheduling a 7-day or 30-day run needs the probability that at least one failure will interrupt that fixed window. Under the exponential failure model (constant failure rate), equation 6 gives the probability of at least one failure during a job of duration \(\JobDuration\) as the complement of the Poisson zero-failure probability over that window:

\[ \Pr(\geq 1\ \text{failure}) = 1 - e^{-\JobDuration/\MeanTimeBetweenFailures} \tag{6}\]

When \(\JobDuration\gg \MeanTimeBetweenFailures\), this probability approaches 1 rapidly, and it does so sooner on a larger cluster, whose MTBF is shorter (table 3).

Table 3: Probability of at Least One Failure: For large clusters and multi-day jobs, failure is a near-certainty. Any system operating in the bottom-right region of this table must treat fault tolerance as a core design requirement, not an optimization.

Cluster GPUs	1 day (24 h)	1 week (168 h)	30 days (720 h)
256	15.1%	68.2%	99.3%
1,024	48.1%	99.0%	> 99.9%
2,048	73.0%	> 99.9%	> 99.9%
8,192	99.5%	> 99.9%	> 99.9%
10,000	99.8%	> 99.9%	> 99.9%
100,000	> 99.9%	> 99.9%	> 99.9%

For any cluster above a few thousand GPUs running jobs longer than a day, the probability of experiencing at least one failure is effectively 100 percent. This is why Fault Tolerance treats fault tolerance as an architectural requirement rather than a defensive measure.

The exponential failure model assumes a constant failure rate, which holds during the steady-state useful-life phase. During burn-in (first few hundred hours) and wear-out (approaching end-of-life), failure rates are higher. In practice, fleet operators observe that newly deployed nodes show 2–3\(\times\) higher failure rates in their first week, making burn-in testing essential before admitting nodes to production clusters. Because failures will interrupt long training jobs, the practical question is how to minimize the lost progress.

---

Checkpoint Optimization

On a 10,000-GPU cluster that fails every few hours, pausing all 10,000 accelerators to flush a multi-terabyte state snapshot to shared storage stalls training, yet spacing snapshots hours apart sacrifices thousands of GPU-hours of progress whenever a node drops out. Choosing a checkpoint cadence therefore trades guaranteed write overhead against expected rework after a crash (Young 1974; Daly 2006).

Young, John W. 1974. “A First Order Approximation to the Optimum Checkpoint Interval.” Communications of the ACM 17 (9): 530–31. https://doi.org/10.1145/361147.361115.

The Young-Daly model

During a synchronous checkpoint, writing state to storage takes time \(\CheckpointWriteTime\) (the checkpoint cost), during which no useful training occurs. Extending the interval between checkpoints amortizes that pause, but a failure that arrives at a random point inside the interval destroys on average half the interval’s work. Equation 7 states the Young-Daly formula that minimizes the sum of those two expected losses:

\[ \OptimalCheckpointInterval = \sqrt{2 \times \CheckpointWriteTime \times \SystemMtbf} \tag{7}\]

where \(\CheckpointWriteTime\) is the checkpoint write time in seconds and \(\SystemMtbf\) is the cluster or job-level system MTBF in seconds.

The intuition behind equation 7 is geometric-mean-like. When checkpoints are cheap relative to the MTBF (\(\CheckpointWriteTime \ll \SystemMtbf\), the common case), the optimal interval sits between the two time scales. If checkpoints took zero time, the optimal cadence would be every step. If the system never failed, no checkpoints would be required. The square root interpolates between those extremes.

The formula assumes that failures follow an exponential distribution (memoryless property) and that checkpoint cost \(\CheckpointWriteTime\) is small compared to \(\SystemMtbf\). Correlated outages, changing hazard rates, long writes, and asynchronous persistence can require a more detailed model. Under the first-order assumptions, evaluating equation 7 for a concrete model requires only the cluster MTBF and the write duration \(\CheckpointWriteTime\), which is set by the serialized state volume and storage bandwidth.

Checkpoint sizing

Resuming a mixed-precision training run after a node failure requires restoring not only the low-precision weights used in the forward pass, but also the full-precision master weights and optimizer moments accumulated across prior steps, which multiplies the serialized footprint and sets the write time \(\CheckpointWriteTime\) in equation 7. For common mixed-precision training checkpoints with the adaptive moment estimation (Adam) optimizer, each parameter requires approximately 14 bytes of persistent state:

• 2 bytes for BF16 model weights
• 4 bytes for FP32 master weights
• 4 bytes for FP32 first moment (Adam \(\AdamFirstMoment\))
• 4 bytes for FP32 second moment (Adam \(\AdamSecondMoment\))

Equation 8 aggregates these per-parameter costs into the total checkpoint footprint. The resulting byte volume scales linearly with model parameter count:

\[ \text{Checkpoint Size} = \Parameters\times 14 \text{ bytes/parameter} \tag{8}\]

Here, \(\Parameters\) is the number of trainable model parameters whose persistent optimizer and weight state must be serialized. Transient activation buffers and unsharded temporary workspaces are excluded because they are recomputed after restore.

Mixed-precision Adam checkpoint sizing (table 4) demonstrates how state scales from manageable gigabytes to frontier-scale terabytes. Even with high-bandwidth parallel storage, serializing these footprints requires tens of seconds to several minutes per snapshot:

Table 4: Checkpoint Sizes for Mixed-Precision Adam Training: Each parameter requires about 14 bytes of persistent checkpoint state (model weights, FP32 master weights, and optimizer moments); gradients are normally transient and recomputed after restore rather than serialized as durable checkpoint state. Write times assume 100 GB/s aggregate storage bandwidth.

Model size	Checkpoint size	Write time at 100 GB/s
7B	98 GB	0.98 s
13B	182 GB	2 s
70B	980 GB	10 s
175B	2,450 GB	24 s
1T	14,000 GB	140 s

At frontier scale (175B+ parameters), these sizes reach the terabyte range. This makes checkpoint write time a significant cost that directly affects the Young-Daly optimal interval. The checkpoint strategies in Fault Tolerance discuss techniques for reducing \(\CheckpointWriteTime\) (asynchronous checkpointing, incremental deltas, and distributed storage), all of which improve the Young-Daly result by shrinking the \(\CheckpointWriteTime\) factor under the square root.

Calculating the optimal checkpoint interval

Combining the checkpoint write time with the 10,000-GPU MTBF gives a concrete checkpoint cadence. The worked example below walks through the two-step calculation and shows how doubling the cluster size shifts the optimum.

Napkin Math 1.1: Young-Daly: 175B model on a 10,000-GPU cluster

Setup: Consider training a 175B-parameter model on a 10,000-GPU cluster. The cluster MTBF is 3.75 hours (table 2). The checkpoint size is 2,450 GB, and the parallel storage system writes at 100 GB/s.

Step 1: Checkpoint write time (\(\CheckpointWriteTime\)). \[T_{\text{write}} = \frac{\text{Checkpoint Size}}{\text{Write Bandwidth}} = \frac{2,450 \text{ GB}}{100 \text{ GB/s}} = 24.5 \text{ s}\]

Step 2: Apply the Young-Daly formula (equation 7). \[T^\ast_{\text{ckpt}} = \sqrt{2 \times T_{\text{write}} \times \text{MTBF}_{\text{system}}} = \sqrt{2 \times 24.5 \text{ s} \times 3.75 \text{ h} \times 3{,}600 \text{ s/h}} = 13.6 \text{ min}\]

Interpretation: The optimal checkpoint interval is approximately 14 minutes. The overhead from checkpointing alone is \(\CheckpointWriteTime/\OptimalCheckpointInterval \approx\) 3.0 percent of training time.

Systems insight: If the cluster were doubled to 20,000 GPUs, the MTBF would halve and the optimal interval would shrink to 9.6 minutes. The system therefore checkpoints more frequently because failures happen more often. Larger clusters are faster but must interrupt themselves more often to protect progress, which is the tension at scale.

When \(\CheckpointWriteTime\) approaches \(\SystemMtbf\) (checkpoint cost approaches MTBF), checkpoint and rework overheads become so large that checkpoint/restart alone may fail to maintain useful forward progress, requiring the alternative strategies in section 1.4.1. Even when \(\CheckpointWriteTime \ll \SystemMtbf\), the Young-Daly cadence prices only the periodic write pause and the average step replay after a fault; a real cluster outage also stalls on failure detection, node replacement, and checkpoint reload before replay can begin.

---

Recovery Budgets

When a node drops out of a 10,000-GPU training step, every surviving accelerator idles while the control plane times out the dead heartbeat, allocates a replacement node, streams terabytes of checkpoint state back into GPU memory, and recomputes the steps lost since the last snapshot. Pricing how much wall-clock capacity those interruptions consume requires decomposing the outage into its constituent phases and translating the total stall time into runtime goodput.

The anatomy of recovery time

A single hardware fault triggers four sequential recovery phases between the instant a worker stalls and the moment the job returns to new training steps. Equation 9 sums them into total recovery time:

\[ \TotalFailureRecoveryTime= \FailureDetectionTime+ \JobRestartTime + \CheckpointLoadTime + \LostWorkReplayTime \tag{9}\]

The terms are all durations: failure detection, replacement scheduling, checkpoint reload, and replay of lost training work since the last checkpoint. Their sum is the wall-clock time before the job returns to productive training.

Table 5 breaks recovery into the phases that determine how long the cluster remains below full productivity after a failure. Because detection, rescheduling, checkpoint reload, and replay appear as separate rows, the table shows which operational bottlenecks dominate total downtime.

Table 5: Recovery Time Breakdown: Each phase contributes to the total time between failure and full-speed resumption. For the 10,000-GPU, 175B-model scenario, replay dominates because it recomputes work lost since the last checkpoint.

Phase	Typical duration	What happens
\(\FailureDetectionTime\)	30 s	Heartbeat timeout expires; worker declared failed (confirmation adds to the budget in Fault Tolerance)
\(\JobRestartTime\)	60 s	Replacement node allocated from spare pool
\(\CheckpointLoadTime\)	24.5 s	Checkpoint read from storage into GPU memory
\(\LostWorkReplayTime\)	~6.8 min	Recompute training steps since last checkpoint
Total \(\TotalFailureRecoveryTime\)	~9 min	System fully productive again

In this sequence, replay time \(\LostWorkReplayTime\) typically dominates and is controlled directly by the checkpoint interval, since on average half the interval must be replayed. This creates a reinforcing loop with the Young-Daly formula. Shorter intervals mean less replay but more checkpoint overhead, and the formula finds the minimum of this sum. The other phases offer distinct engineering levers. More aggressive heartbeat intervals shorten \(\FailureDetectionTime\), at the cost of false positives. Preallocated hot spare nodes shorten \(\JobRestartTime\) through the fleet orchestration policies covered in Fleet Orchestration. Higher storage bandwidth or smaller checkpoints shorten \(\CheckpointLoadTime\), which motivates the compression and sharding techniques discussed in Fault Tolerance.

Runtime goodput under failures

Every minute a cluster spends writing a periodic snapshot or executing the four-phase recovery pipeline of table 5 is billed wall-clock allocation that advances the final model by zero training steps. Comparing the committed step rate (the steps a checkpoint retains for the final model) against the fault-free step rate (the rate the configured cluster could sustain without checkpoint or recovery interruptions) defines the runtime goodput of a job whose only losses are checkpoints and failures:

\[ \RuntimeGoodput \approx \frac{\text{Committed Useful-Step Rate}}{\text{Fault-Free Step Rate}} \approx \frac{\UsefulTrainingTime}{\AllocatedWallTime} \]

Runtime Goodput defines runtime goodput as one factor of ML goodput and also charges startup and input stalls to it, which this appendix does not model. The modeled gap between the two rates comes from two sources:

1. Checkpoint overhead (\(\sim\) 3.0 percent): Training pauses during each checkpoint write.
2. Recovery and rework overhead (\(\sim\) 3.9 percent): Time lost to detection, rescheduling, reloading, and replaying work discarded after each failure.

At 10,000-GPU scale, failures and checkpointing together can consume a substantial fraction of wall time. If they consume a fraction \(\OperationalOverheadFraction\), a fixed amount of useful work takes \(1/(1-\OperationalOverheadFraction)\) times its fault-free duration, not merely that duration plus \(\OperationalOverheadFraction\) of it. The fleet orchestration strategies in Fleet Orchestration and ML Operations at Scale focus on narrowing this gap, because every percentage point of overhead recovered saves dollars and shortens training time.

Systems Perspective 1.1: The hidden cost of scale

A common misconception is that doubling cluster size halves training time. In practice, doubling from 5,000 to 10,000 GPUs halves the MTBF and increases failure-related overhead. The effective speedup is less than 2\(\times\), and at extreme scale, adding more GPUs can actually increase wall-clock time if the fault tolerance mechanisms cannot keep pace. The serial overhead of recovery bounds the benefit of parallelism, a reliability analogue of Amdahl’s Law.

---

Strategy Selection

Pausing a 10,000-GPU batch training job for several minutes to reload a checkpoint and replay lost steps is an acceptable throughput tax, whereas stalling a live inference service for even a few seconds violates user-facing latency contracts and drops production traffic. Because batch training and online serving tolerate interruption on completely different time scales, fleet architectures choose among checkpoint/restart, active redundancy, and elastic resizing according to the recovery latency, spare capacity, and state-management complexity each workload can afford.

Checkpoint/restart vs. redundancy vs. elastic training

Whether a fault-tolerance mechanism can pause the entire job to replay work, must fail over within milliseconds, or must continue on a shrunk allocation when replacement nodes are unavailable separates the three primary recovery architectures. Checkpoint/restart periodically saves full system state and rolls back to the last checkpoint after failure. It is the workhorse of large-scale training: conceptually simple, well-understood, and effective when MTBF is much larger than checkpoint cost. The weakness is that recovery requires stopping all workers and replaying lost computation.

Redundancy maintains alternate copies of state or computation. A surviving serving replica can take over only when routing, readiness, capacity, and the required request state meet the latency budget. Dedicated duplicate or triplicate execution costs 2–3\(\times\) the compute resources; active-active serving instead shares normal traffic across replicas while reserving failover headroom. The justified overhead depends on the workload and recovery contract.

Elastic training allows a job to resume with fewer workers when the smaller allocation can hold its state and support its parallel decomposition. Membership changes can still require a coordinated restart and checkpoint reload; TorchElastic restarts the worker group. Workers may be added back when capacity becomes available. The recovery path must restore missing shards, preserve data coverage, and either preserve the effective batch size or validate a changed optimization policy. Elasticity can reduce waiting for replacement capacity, but does not guarantee a shorter interruption. The three strategies trade recovery latency against resource overhead and implementation complexity, and each fits a different workload (table 6).

Table 6: Fault Tolerance Strategy Comparison: The latency and overhead ranges are illustrative, not guarantees; elasticity does not by itself require spare capacity. Real-world systems often combine strategies: checkpoint/restart for training with redundancy for the metadata service and checkpoint storage layer.

Criterion	Checkpoint/restart	Redundancy	Elastic training
Recovery latency	Minutes (replay)	Milliseconds (failover)	Restart and resize dependent
Resource overhead	~3–13% (storage + IO)	100–200% (replicas)	Resize and standby policy dependent
Workload fit	Training (batch)	Serving (online)	Training (long-running)
Implementation	Simple	Moderate	Complex
State management	Periodic snapshots	Continuous replication	Distributed with resharding
Failure mode	Job pauses, replays	Transparent to user	Pause, restore, then resume

The availability stacking formula

A single serving replica with 99 percent availability (“two nines”) still accumulates more than 87 hours of downtime per year, far exceeding the outage budget of production inference endpoints that target 99.9 percent (“three nines”) or 99.99 percent (“four nines”) uptime. Redundancy closes that gap by running \(\RedundantReplicaCount\) independent replicas so that surviving instances absorb traffic when one replica fails. Under an idealized model with reliable routing and enough capacity on any surviving replica, the system is unavailable only when all replicas are simultaneously down. Equation 10 captures this, with \(\ReplicaAvailability\) denoting the availability of a single replica and \(\RedundantReplicaCount\) the number of replicas:

\[ \RedundantServiceAvailability= 1 - (1 - \ReplicaAvailability)^\RedundantReplicaCount \tag{10}\]

Evaluating availability stacking with independent replicas (table 7) quantifies the gain from adding serving capacity. Each additional independent replica multiplies the outage probabilities together, cutting expected annual downtime by two orders of magnitude per replica at a 99 percent baseline.

Table 7: Availability Stacking with Independent Replicas: Starting from a single-replica availability of 99 percent, each additional replica dramatically reduces expected downtime. Assumes replica failures are independent.

Replicas \(\RedundantReplicaCount\)	System availability	Nines	Downtime per year
1	99.00%	2.0	87.6 hours
2	99.9900%	4.0	52.6 minutes
3	99.9999%	6.0	31.5 seconds

Stacking independent redundant replicas achieves large availability gains. Two replicas that are each available 99 percent of the time yield 99.99 percent availability, reducing annual downtime from 87.6 hours to 52.6 minutes. This is why inference serving systems almost universally deploy multiple replicas behind a load balancer. The cost of an extra replica is small compared to the business value of four-nines availability.

The result, however, depends on the independence assumption. Correlated failures, such as power outages affecting an entire rack, software bugs triggered by a specific input, or network partitions isolating a failure domain, defeat availability stacking. This is why Fault Tolerance emphasizes failure domain isolation: replicas must be placed in different racks, different power zones, and ideally different data centers to reduce shared failure modes. Placement alone does not eliminate common software, control-plane, or regional dependencies.

---

Checkpoint Intervals Beyond the First-Order Approximation

When a multi-terabyte checkpoint takes several minutes to flush to shared storage and a 100,000-GPU cluster fails every half hour, the checkpoint write is no longer negligible next to the failure interval. The cluster can then suffer a second crash while still writing the snapshot. Young’s first-order approximation is most accurate when checkpoint time is small relative to the failure interval; Daly’s analysis develops higher-order checkpoint models for regimes where write time is a substantial fraction of MTBF (Daly 2006). The simplified renewal model below shows why finite checkpoint time changes the optimum without claiming to reproduce Daly’s full recovery model.

Daly, J. T. 2006. “A Higher Order Estimate of the Optimum Checkpoint Interval for Restart Dumps.” Future Generation Computer Systems 22 (3): 303–12. https://doi.org/10.1016/j.future.2004.11.016.

Renewal model and small-cost expansion

Suppose a worker fails halfway through flushing a 60-second checkpoint snapshot; the job loses both the training steps computed since the previous snapshot and the 30 seconds spent writing the incomplete checkpoint. Let \(\CheckpointInterval\) be useful compute time between completed checkpoints. Assume Poisson failures with rate \(1/\SystemMtbf\), identical failure exposure during computation and writing, and zero detection, restart, or reload delay. A failed attempt loses its partial progress and repeats the interval. The expected elapsed time to complete one uninterrupted compute-plus-checkpoint attempt is:

\[ \mathbb{E}[\IntervalCompletionTime] = \SystemMtbf \left( e^{\frac{\CheckpointInterval + \CheckpointWriteTime}{\SystemMtbf}} - 1 \right) \tag{11}\]

Writing \(\CheckpointAttemptDuration= \CheckpointInterval + \CheckpointWriteTime\) for the attempt duration, the renewal equation is \(\mathbb{E}[\IntervalCompletionTime] = \mathbb{E}[\min(\NextFailureTime, \CheckpointAttemptDuration)] + \Pr(\NextFailureTime< \CheckpointAttemptDuration) \mathbb{E}[\IntervalCompletionTime]\), where \(\NextFailureTime\) is an exponential failure time. Since \(\mathbb{E}[\min(\NextFailureTime, \CheckpointAttemptDuration)] = \SystemMtbf (1 - e^{-\CheckpointAttemptDuration/\SystemMtbf})\), solving the equation gives equation 11. Expanding the exponential through its quadratic term then gives the wasted time per completed interval:

\[ \mathbb{E}[\IntervalCompletionTime] - \CheckpointInterval \approx \CheckpointWriteTime + \frac{(\CheckpointInterval + \CheckpointWriteTime)^2}{2 \cdot \SystemMtbf} \tag{12}\]

When \(\CheckpointWriteTime/\SystemMtbf\) is small, dividing equation 12 by \(\CheckpointInterval\) leaves the leading terms \(\CheckpointWriteTime/\CheckpointInterval + \CheckpointInterval/(2 \cdot \SystemMtbf)\) of the wasted-time fraction. Minimizing those terms recovers Young’s interval \(\sqrt{2 \cdot \CheckpointWriteTime \cdot \SystemMtbf}\). For finite checkpoint costs, differentiate the unexpanded elapsed time per unit useful work, \(\mathbb{E}[\IntervalCompletionTime]/\CheckpointInterval\), instead. Its stationary point satisfies:

\[ e^{(\OptimalCheckpointInterval + \CheckpointWriteTime)/\SystemMtbf}\left(1 - \frac{\OptimalCheckpointInterval}{\SystemMtbf}\right) = 1, \qquad 0 < \OptimalCheckpointInterval < \SystemMtbf \tag{13}\]

For positive checkpoint cost, the equivalent derivative numerator \(e^{(\CheckpointInterval + \CheckpointWriteTime)/\SystemMtbf}(\CheckpointInterval/\SystemMtbf - 1) + 1\) increases from a negative value at zero to a positive value at \(\SystemMtbf\). A one-dimensional root search on equation 13 therefore finds the unique optimum of this simplified model.

Choosing the appropriate approximation

The first-order formula is a useful planning estimate when writes are short compared with MTBF. When that condition fails, use a model that includes the deployment’s checkpoint semantics, recovery delays, and failure exposure. The numerical condition in equation 13 covers only the stated zero-restart-delay renewal model; asynchronous writes and correlated failures require different assumptions. All of these checkpoint and redundancy models assume that a hardware fault announces itself by stopping a worker or failing a health check, yet some of the most damaging faults at fleet scale corrupt tensor values silently without halting execution.

---

Silent Data Corruption in Attention

Not all hardware faults manifest as explicit system crashes, ECC double-bit errors, or kernel traps. Silent data corruption (SDC) occurs when single-bit errors or arithmetic logic unit (ALU) faults slip past parity checks, corrupting tensor values in SRAM, high-bandwidth memory (HBM), or register files without triggering an exception. In deep learning workloads, SDC is particularly dangerous because single corrupted floating-point values propagate through non-linear operations, destabilizing fleet-wide training (Dixit et al. 2021).

Dixit, H. D., S. Pendharkar, M. Beadon, C. Mason, T. Chakravarthy, B. Muthiah, and S. Sankar. 2021. Silent Data Corruptions at Scale. arXiv preprint arXiv:2102.11245.

Exponent bit flip model in FP16

Consider an IEEE 754 half-precision floating-point (FP16) tensor element representing a query entry \(\QueryComponent\), activation value, or weight matrix entry. FP16 allocates 16 total bits: 1 sign bit (\(\FloatSignBit\)), 5 exponent bits (\(\FloatExponentField\)), and 10 mantissa bits (\(\FloatMantissaField\)). For a finite normal value, with exponent field \(1 \le \FloatExponentField\le 30\), the scalar \(\FloatEncodedNumber\) represented by the bitstring \((\FloatSignBit, \FloatExponentField, \FloatMantissaField)\) is given by:

\[ \FloatEncodedNumber= (-1)^\FloatSignBit\times 2^{\FloatExponentField- 15} \times \left(1 + \frac{\FloatMantissaField}{1024}\right) \]

Suppose a cosmic ray, voltage drop, or SRAM failure causes a single bit flip in the most significant exponent bit (bit 14). The exponent value changes from \(\FloatExponentField\) to \(\FlippedFloatExponentField = \FloatExponentField\pm 16\):

\[ \BitFlippedFloatNumber= (-1)^\FloatSignBit\times 2^{(\FloatExponentField\pm 16) - 15} \times \left(1 + \frac{\FloatMantissaField}{1024}\right) = \FloatEncodedNumber\times 2^{\pm 16} \]

If both the original and corrupted exponent fields encode finite normal values, the magnitude increases by a factor of \(2^{16}=65{,}536\) or decreases by its reciprocal. Exponent fields of zero or all ones instead invoke the subnormal, zero, infinity, or NaN encodings, so the displayed normal-value formula does not apply.

Softmax saturation and error propagation

When a single-bit exponent flip inflates one query entry by a factor of \(65{,}536\), the dot products in that token’s attention row can jump by orders of magnitude and collapse its softmax distribution onto a single key. To trace how that perturbation propagates through one attention head, let \(\AttentionScoreMatrix = \AttentionQueryMatrix\AttentionKeyMatrix^T/\sqrt{\AttentionHeadDimension} \in \mathbb{R}^{\SequenceLength\times \SequenceLength}\) be the pre-softmax score matrix for sequence length \(\SequenceLength\) and head dimension \(\AttentionHeadDimension\). Suppose an SDC event corrupts one element of token row \(\MatrixRowIndex\) of the query matrix \(\AttentionQueryMatrix\), and write the resulting perturbation as \(\FiniteChange\AttentionQueryMatrix\), with only that element nonzero:

\[ \CorruptedAttentionScoreMatrix = \frac{(\AttentionQueryMatrix+ \FiniteChange\AttentionQueryMatrix) \AttentionKeyMatrix^T}{\sqrt{\AttentionHeadDimension}} = \AttentionScoreMatrix + \FiniteChange\AttentionScoreMatrix \]

where \(\FiniteChange\AttentionScoreMatrix\) is nonzero only in row \(\MatrixRowIndex\), with entries \(\FiniteChange \AttentionScoreEntry = \FiniteChange\QueryVector_i \cdot \KeyVector_j/\sqrt{\AttentionHeadDimension}\) for the corrupted query row \(\FiniteChange\QueryVector_i\) and key row \(\KeyVector_j\). Passing the perturbed score row through the normalization step shifts the entire row’s attention weights. The attention probability perturbation for token row \(\MatrixRowIndex\) is therefore:

\[ \FiniteChange\AttentionWeightVector_i = \operatorname{softmax}\left( \AttentionScoreRow + \FiniteChange\AttentionScoreRow \right) - \operatorname{softmax}\left( \AttentionScoreRow \right) \]

A large corrupted query component can strongly change several attention logits, with the signs and magnitudes determined by the corresponding key components. If one resulting logit exceeds every other logit by an increasingly large margin, the attention distribution approaches a one-hot vector:

\[ \operatorname{softmax}(\AttentionScoreRow + \FiniteChange\AttentionScoreRow)_j \to 1, \qquad \operatorname{softmax}(\AttentionScoreRow + \FiniteChange\AttentionScoreRow)_{k \neq j} \to 0 \]

This is a conditional saturation limit, not an inevitable consequence of a bit flip. The selected key depends on the corrupted logits; a corrupted query element does not identify one corrupted key. The softmax derivative is \(\partial \SoftmaxOutputProbability_j/\partial \AttentionScoreSummand = \SoftmaxOutputProbability_j(\KroneckerIndicator_{jk}-\SoftmaxOutputProbability_k)\), which tends toward zero under saturation rather than diverging. Other paths through the computation can still produce large or nonfinite activations and gradients. If invalid state enters the optimizer or a collective, it can propagate across ranks. Detection and recovery must therefore verify state before resuming from a clean checkpoint.

---

Summary

Failure is a physical constraint of scale. Under independent, constant component failure rates, a single GPU’s mean lifetime is 5.7 years. Yet the modeled 10,000-GPU cluster fails on average every 3.75 hours, because node failure rates sum across components and cluster MTBF divides by node count. The probability that a job is interrupted grows with its duration relative to that interval, so long runs on large clusters must plan for recovery rather than hope to avoid it.

Recovery has four modeled phases (detection, rescheduling, reloading, and replay), and replay is controlled by the checkpoint interval. The Young-Daly formula \(\OptimalCheckpointInterval = \sqrt{2 \CheckpointWriteTime \, \SystemMtbf}\) (principle 13) balances write overhead against expected rework when writes are short compared with the MTBF. Section 1.5 examines finite-cost effects under a simplified renewal model. Checkpoint/restart suits batch training, redundancy suits latency-sensitive serving, and elastic training sits between them. Independent replicas reduce unavailability multiplicatively, to \((1-\ReplicaAvailability)^\RedundantReplicaCount\) for \(\RedundantReplicaCount\) replicas, but only to the extent that failure-domain isolation keeps their failures independent.