The Scheduling Problem

Fleet Orchestration

Purpose

Why does resource allocation become the primary bottleneck when hardware is plentiful?

A thousand accelerators sitting idle waiting for scheduling decisions cost the same as a thousand accelerators computing useful work. At scale, the limiting factor shifts from having enough hardware to using it efficiently. Jobs wait in queues while resources sit idle, fragmentation leaves gaps too small for pending jobs, and deadlocks leave jobs holding partial resources while waiting for more. Orchestration extracts useful work from shared infrastructure by deciding which jobs run on which nodes, when to preempt, how to balance fairness against utilization, and how to keep the coordination mechanisms from becoming bottlenecks. Poor orchestration turns expensive hardware into waste; effective orchestration turns a collection of machines into a coherent computing resource. In C³ terms, orchestration deliberately pays a coordination tax, trading some peak utilization to prevent deadlock and fragmentation under load.

Learning Objectives

• Analyze scheduling objectives, fragmentation, and deadlock risks for distributed training jobs under shared fleet constraints
• Compare HPC, cloud-native, and hybrid orchestration paradigms for batch training and self-healing serving workloads
• Apply gang, bin-packing, and topology-aware placement policies across NVLink domains and switch hierarchies
• Design elastic training and preemption policies that trade throughput, recovery overhead, and spot-instance savings
• Evaluate ML-specific schedulers using job duration, iteration progress, fairness, and adaptive resource-allocation signals
• Design inference autoscaling, isolation, and routing policies using queue depth, tail latency, and GPU memory pressure
• Diagnose utilization losses from quota hoarding, noisy neighbors, fragmentation, topology mismatch, and data starvation

Imagine two research teams sharing a 1,000-GPU cluster: Team A submits a 512-GPU job that will run for a month, while Team B submits hundreds of 8-GPU experiments that each run for an hour. If the scheduler simply processes jobs as they arrive, Team B’s tiny experiments might indefinitely block Team A’s massive training run. The scheduling problem is the challenge of navigating this multi-dimensional conflict between throughput, fairness, and cluster utilization.

Partitioning computation, synchronizing it efficiently, and recovering from hardware failures solve the single-job scaling problem. Orchestration addresses the fleet-level challenge: deciding which jobs run, where they run, and how resources are shared among competing demands.

In the fleet stack (shown in the margin), orchestration operates at the distribution layer, but the infrastructure layer constrains its decisions. The scheduler must account for GPU heterogeneity, NVLink topology, rack power limits, and network bisection bandwidth. When scheduling goes wrong, debugging requires examining all four layers to identify whether the root cause is infrastructure (hardware constraints), distribution (algorithm limitations), serving (workload mismatch), or governance (policy misconfiguration). Orchestration connects physical infrastructure to distributed algorithms, translating resource requests into placements that respect both hardware topology and organizational policy.

To make the scheduling problem concrete, consider a research organization operating a 10,000-GPU cluster. Brown et al. (2020) report that training GPT-3 175B used \(3.14 \times 10^{23}\) floating-point operations on a V100 GPU cluster. Imagine 100 large-scale training jobs of that class queued, alongside thousands of smaller experiments and inference workloads all competing for the same resources. The system must decide which jobs run, when they run, and where they run. A naive first-come-first-served policy would let the first few large jobs monopolize the cluster for weeks, starving smaller experiments that researchers need to iterate quickly. A strict fair-share policy would fragment GPUs across many small allocations, preventing any large job from assembling the contiguous allocation it needs. Neither extreme works. The scheduler must navigate a multi-dimensional trade-off space where every decision affects throughput, fairness, cost, and researcher productivity simultaneously.

Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. “Language Models Are Few-Shot Learners.” Advances in Neural Information Processing Systems 33: 1877–901. https://doi.org/10.48550/arxiv.2005.14165.

The economic stakes make these decisions consequential at every scale. A 10,000-GPU cluster at $2/GPU-hour costs $480,000 per day to operate, whether the GPUs are computing useful work or sitting idle in a queue. If scheduling inefficiencies leave 30 percent of GPUs idle, that translates to $144,000 per day in wasted capacity, or about $52.6M annually. Conversely, improving utilization from 50 percent to 80 percent adds 3,000 fully utilized GPU-equivalents of productive capacity without purchasing additional hardware. That gain is equivalent to buying 6,000 raw GPUs if they ran at the old 50 percent utilization. At this scale, a 1 percentage-point improvement in utilization is worth more annually than the salary of the engineer who achieves it. Scheduling is not operational overhead; it is one of the highest-leverage engineering investments in ML infrastructure.

ML workloads present scheduling challenges that distinguish them from ordinary cloud service placement and from HPC workloads that can tolerate flexible resource counts. Gang scheduling¹ represents the most critical difference for synchronous distributed training: a job requiring 1,024 GPUs cannot make useful iteration progress with only 512. Many HPC jobs also need co-scheduled ranks, but synchronous data parallel training exposes the cost at every AllReduce. Every worker must participate in every collective, and a missing worker blocks all others. AllReduce develops the ring and tree AllReduce cost model that makes this constraint physical: the collective stalls until the last worker arrives, so allocating \(N\) GPUs but landing \(N-1\) leaves the missing rank’s peers burning fully idle on the synchronization barrier. A scheduler that partially allocates resources creates deadlocks where multiple jobs each hold some GPUs while waiting for more, with none able to proceed. This all-or-nothing requirement means the scheduler cannot simply “pack jobs tightly” as a traditional bin packer would; it must reason about atomic, multi-resource allocations across the entire cluster.

¹ Gang Scheduling: Formalized by Ousterhout (1982) for parallel systems, where the “Ousterhout matrix” co-schedules related threads across processors in synchronized time slices. In ML clusters, the constraint is stricter: synchronous AllReduce requires all workers simultaneously, so partial allocation wastes 100 percent of held resources rather than merely degrading throughput. Under the chapter’s cloud-equivalent $2/GPU-hour scenario, a 1,024-GPU job holding 900 idle GPUs while waiting for 124 more burns approximately $1,800/hour.

Ousterhout, John K. 1982. “Scheduling Techniques for Concurrent Systems.” Proceedings of the 3rd International Conference on Distributed Computing Systems (ICDCS), 22–30.

² Placement Group: A cloud-native abstraction (AWS, GCP) that requests instances be placed within a single high-bandwidth, low-latency network domain. For the ML Fleet, placement groups act as the topology contract that guarantees the bisection bandwidth \((\text{BW}_{\text{bisect}})\) required for AllReduce, preventing the “topology lottery” where nodes are scattered across different data center racks.

If gang scheduling is the first hard constraint, topology awareness via Placement Groups² is the second: to support tensor parallelism, the scheduler must ensure GPUs are placed within the same high-bandwidth NVLink domain rather than scattered across the data center.

GPU heterogeneity is the third dimension. Many clusters contain mixtures of A100 and H100 accelerators with different compute throughput, comparable high-bandwidth memory (HBM) capacity (80 GB on A100 vs. 80 GB on H100), and different interconnect bandwidth. An H100 provides roughly twice the training throughput of an A100 for transformer workloads but costs proportionally more. Some models fit in A100 memory with appropriate parallelism strategies; others require the larger H100 memory to avoid excessive model sharding. The scheduler must match workloads to appropriate hardware based on actual requirements, not user preferences, while maintaining high utilization across heterogeneous pools. When users request specific GPU types out of habit rather than necessity, the mismatch between requested and required resources can leave entire hardware pools idle while queues overflow on preferred hardware.

Job duration unpredictability compounds these challenges further. A training run may converge early and complete in days rather than weeks. Hardware failures may abort jobs unexpectedly, returning resources at unpredictable times. Hyperparameter searches may reveal early that certain configurations will not succeed, leading to voluntary termination. Traditional scientific computing workloads, such as weather simulations or molecular dynamics, have more predictable runtimes that enable better scheduling decisions: the scheduler can project when resources will become available and plan accordingly. ML workloads deny this luxury, forcing schedulers to operate with deep uncertainty about when current jobs will release resources.

The interaction between these challenges creates combinatorial complexity. A scheduler must simultaneously handle gang scheduling constraints (atomic allocation), heterogeneous resources (capability matching), topology requirements (NVLink locality for tensor parallelism), priority and fairness policies (organizational equity), preemption decisions (which running jobs to interrupt), and cost optimization (spot vs. on-demand placement). Each dimension constrains the others: a topology-optimal placement may violate fairness policies, a cost-optimal spot placement may violate reliability requirements, and a fairness-driven allocation may fragment resources in ways that prevent gang scheduling.

These constraints define the central problem of orchestration: the scheduler is not merely packing jobs onto GPUs, but maintaining a consistent, economically useful view of a failing, heterogeneous fleet. The first step is to see why cluster scheduling is fundamentally harder than single-machine process scheduling. At fleet scale, placement stops being only an optimization challenge and becomes a distributed systems problem of keeping state coherent across thousands of machines.

Distributed scheduling complexity

A single-machine scheduler enjoys luxuries that a cluster scheduler cannot: it has instantaneous, consistent visibility into all resource states; it can make atomic decisions that take effect immediately; and failures are binary (the machine is up or it is down). Cluster scheduling surrenders all three of these properties, and several fundamental distributed systems problems emerge as a result.

Partial failures pose the first challenge. A node can fail between allocation and job start, creating a gap between the scheduler’s decision and its effect. The scheduler may successfully allocate 32 GPUs across 4 nodes, only to have one node fail before the job launches. The remaining 24 GPUs sit idle while the scheduler detects the failure, updates its state, and re-plans the allocation. Meanwhile, other jobs that could have used those 24 GPUs have already been placed elsewhere, potentially on suboptimal hardware. The fault tolerance mechanisms from Fault Tolerance handle failures during execution; the scheduler must handle failures during placement, a fundamentally different problem because the job has not yet established any state to recover from.

Network partitions create a second problem that is unique to distributed scheduling. The scheduler may lose connectivity to a subset of nodes while those nodes continue operating normally. From the scheduler’s perspective, the nodes appear failed and their GPUs appear unavailable. From the nodes’ perspective, jobs may still be running and producing useful work. This ambiguity creates a dilemma with no universally correct resolution. Reallocating the GPUs to new jobs risks double-allocation if the partition heals; waiting for reconnection wastes capacity that may be perfectly functional. The duration of the partition is unknowable in advance, so any fixed timeout represents a guess about network behavior that may prove wrong.

State inconsistency emerges as a third challenge that compounds the first two. Resource state may differ between the scheduler’s view and reality on individual nodes. A GPU the scheduler believes is free may still be cleaning up from a previous job: flushing caches, deallocating device memory, running a zombie process from a failed container, or completing a CUDA driver reset after an error. Conversely, a GPU marked as “in use” may have been freed by a job that terminated between heartbeat intervals. This inconsistency means the scheduler operates on a model of cluster state that is always slightly stale, and the degree of staleness varies across nodes depending on heartbeat frequency, network latency, and the complexity of cleanup operations.

Ordering without global time presents the fourth fundamental issue. Without a global clock, determining whether allocation A happened before allocation B across different nodes requires careful protocol design using logical clocks or consensus algorithms. Two jobs may both believe they “own” the same GPU if the system does not enforce ordering through consensus protocols or centralized coordination. This scenario is not theoretical: during recovery from a network partition, allocation messages from before and after the partition may arrive in the wrong order at different nodes, creating conflicting resource assignments that must be detected and resolved.

These four challenges echo the consistency, availability, partition-tolerance trade-off described by the CAP theorem.³ CAP is a theorem about distributed data systems, not a literal proof about every scheduler, but the analogy is useful: a control plane must choose how much stale or uncertain state it is willing to tolerate before making allocation decisions. Centralized batch control systems like Slurm, an HPC workload manager, emphasize an authoritative allocation view before jobs start. Reconciliation loops like those in Kubernetes, a container orchestration platform, emphasize continuously driving the actual cluster toward declared desired state. Custom ML schedulers often accept bounded inconsistency in exchange for scheduling throughput, using optimistic concurrency where conflicts are detected and resolved after the fact rather than prevented through locking.

³ CAP (Consistency, Availability, Partition Tolerance) Theorem: Conjectured by Brewer (2000) and proven by Gilbert and Lynch (2002), CAP establishes that no distributed data system can simultaneously guarantee Consistency, Availability, and Partition tolerance. For ML schedulers, use CAP as a design analogy rather than a product claim: the concrete policy question is how much uncertainty the control plane accepts before it allocates, reclaims, or reschedules expensive accelerators.

Brewer, Eric A. 2000. “Towards Robust Distributed Systems (Abstract).” Proceedings of the Nineteenth Annual ACM Symposium on Principles of Distributed Computing, 7. https://doi.org/10.1145/343477.343502.

Gilbert, Seth, and Nancy Lynch. 2002. “Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services.” ACM SIGACT News 33 (2): 51–59. https://doi.org/10.1145/564585.564601.

PyTorch Contributors. 2026b. Torchrun (Elastic Launch). PyTorch documentation.

PyTorch Contributors. 2026a. Torch Distributed Elastic Rendezvous. PyTorch documentation.

The control-plane tension becomes concrete when the scheduler interacts with the ML framework’s own distributed coordination plane during a partition. A scheduler may decide that a subset of nodes is unavailable while workers on those nodes are still blocked inside a collective operation. That creates a brief window of conflicting intent: the scheduler wants to recover the allocation, but the training job has not yet converged on a new membership view. Elastic launch and rendezvous mechanisms such as TorchElastic reduce this ambiguity by using heartbeats and re-rendezvous so the framework can either reform a viable worker group or terminate cleanly, giving the scheduler a clearer recovery signal than an opaque collective timeout (PyTorch Contributors 2026b, 2026a). This layered coordination—scheduler-level policy for coarse allocation, framework-level membership for workers—is the characteristic architecture of production distributed training clusters.

Failure rates at scale

At scale, failure is normal operation, not exceptional. This principle, established in the reliability analysis of Fault Tolerance, has direct consequences for scheduling: the scheduler must not merely tolerate failure but actively plan for it in every allocation decision. Component reliability does not change with cluster size, but aggregate system reliability degrades multiplicatively. Counting GPU hardware faults alone understates the rate; with 99.9 percent annual GPU reliability (typical for data center hardware), the hardware-only expectation for a 4,096-GPU cluster is modest:

Normal component failure rates make fleet failures routine.

\[ \text{Expected failures per day} = 4096 \times \frac{0.001}{365} \approx 0.01 \text{ GPU failures/day} \]

This calculation captures only GPU hardware failures. Including software failures (driver crashes, CUDA context corruption), thermal events (throttling, emergency shutdowns), network interface failures, host OS issues, and container runtime errors, large clusters can see 1 to 4 failures per day per 1,000 GPUs. A 10,000-GPU cluster would then experience 10 to 40 component failures daily. A multi-week training run on 4,096 GPUs is therefore very likely to encounter multiple failures. Component failure rates tabulates the FIT rates and mean time to failure (MTTF) figures behind the 99.9 percent annual reliability assumption, letting an operator substitute measured hardware rates and confirm whether the empirical 1-to-4 failures-per-day band still holds.

The scheduling implications are profound and shape nearly every design decision in the scheduler. The scheduler cannot treat the cluster as a static resource pool where resources, once allocated, remain available until voluntarily released. Instead, it must anticipate that allocated resources will disappear during job execution, maintain spare capacity or implement rapid rescheduling to keep jobs running through the constant churn of hardware entering and leaving operational status, and distinguish between transient failures (which may self-resolve within minutes and should not trigger expensive reallocation) and permanent failures (which require new resource allocation and checkpoint recovery) (Tiwari et al. 2015).

Tiwari, Devesh, Saurabh Gupta, James Rogers, Don Maxwell, Paolo Rech, Sudharshan Vazhkudai, Daniel Oliveira, et al. 2015. “Understanding GPU Errors on Large-Scale HPC Systems and the Implications for System Design and Operation.” 2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA), 331–42. https://doi.org/10.1109/hpca.2015.7056044.

Consider the scheduler’s decision when a node heartbeat is missed. The scheduler has three options:

• Wait for recovery: The scheduler waits to see if the heartbeat resumes, risking wasted GPU time if the node is truly dead.
• Reallocate immediately: The scheduler assigns the node’s resources to other jobs, risking conflict if the node recovers and its original jobs are still running.
• Mark suspect: The scheduler marks the node as suspect and begins prepositioning replacement resources while waiting for confirmation, consuming spare capacity that could be used for other work.

Each option trades off between responsiveness and correctness, and the effective choice depends on the failure mode distribution for the specific hardware in the cluster, information the scheduler must learn from historical failure data. The anatomy of recovery time decomposes recovery time into its detect, reschedule, reload, and replay phases, which bound how long the scheduler can afford to wait on a missed heartbeat before committing to reallocation rather than betting on a transient self-resolution.

These requirements connect the scheduler directly to the checkpoint and recovery infrastructure from Fault Tolerance: scheduling decisions about spare capacity, preemption policies, and elastic training support determine how quickly the system recovers from statistically expected failures. A scheduler that reserves spare capacity or supports elastic recovery can restore throughput much faster than one that requires every failed job to wait for a full fresh allocation and reload, but the correct reserve fraction and recovery time are workload- and fleet-specific parameters rather than universal constants.

Scheduling objectives and their conflicts

Every scheduling decision represents a trade-off between four fundamental and often contradictory objectives:

• Throughput: The total useful work completed per unit time naturally favors large, long-running jobs that saturate hardware and minimize context-switching and data-movement overhead.
• Fairness: The equitable distribution of resources across users or teams prevents a single large team from monopolizing the fleet. Dominant resource fairness (Ghodsi et al. 2011) equalizes each user’s largest normalized share across resources such as GPUs, CPU, and memory.
• Latency: Rapid completion of short jobs maintains researcher velocity, but aggressive preemption can starve large training jobs.
• Cost Efficiency: Interruptible spot instances, off-peak scheduling, and tight packing reduce budget burn, but often increase failure rates or completion time.

Ghodsi, Ali, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker, and Ion Stoica. 2011. “Dominant Resource Fairness: Fair Allocation of Multiple Resource Types.” 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI 11), 323–36.

The objectives conflict because each one rewards a different scheduling behavior. A throughput-maximal policy fills the cluster with massive training runs, achieving near-perfect utilization but forcing every other user to wait weeks for a slot. Fairness-driven allocation fragments resources, leaving small gaps that cannot be filled by large jobs and sacrificing aggregate throughput for social harmony. A latency-optimal scheduler, similar to shortest job first, aggressively preempts long-running training jobs to service interactive debugging sessions or small experiments. Cost optimization often means accepting higher failure rates and longer completion times, trading researcher productivity for budget preservation.

Optimizing all four simultaneously is impossible; every scheduling policy represents a specific point in this four-dimensional trade-off space. The most direct conflict exists between throughput and latency, a tension analogous to the throughput-latency trade-off in computer architecture. A throughput-optimal scheduler prioritizes the largest, most parallelizable jobs because they use the hardware most efficiently. However, this policy is latency-catastrophic for small jobs: a researcher submitting a 10-minute debugging task might wait days for the large job to finish. Conversely, a latency-optimal scheduler minimizes average wait time but potentially starves large jobs indefinitely, reducing aggregate cluster throughput by leaving large blocks of resources idle while waiting for “just one more” small job to finish.

Past about 70 percent utilization, queue wait time explodes.

Consider a 64-GPU fine-tuning run for our 175B model, a scheduling-demo workload distinct from the full 1,024-GPU pretraining run that recurs elsewhere in the fleet discussion. From a throughput perspective, this is an ideal job: it runs for days with high utilization and zero scheduling overhead once started. From a latency perspective, it is a boulder in the stream. To schedule it, the system might need to drain 64 GPUs of all other work, forcing hundreds of smaller jobs to wait. Once running, it occupies those resources immovably. If the scheduler prioritizes this run (throughput), the P99 latency for small jobs explodes. If it prioritizes small jobs (latency) by allowing them to preempt or fragmentation-fill the cluster, the 64-GPU job may never assemble the contiguous block it needs to start. This zero-sum game forces organizations to make explicit policy choices. In scheduler terminology, those choices often appear as quality-of-service classes: job-priority tiers that decide which objective wins when throughput, latency, fairness, and cost collide.

Napkin Math 1.1: The queuing theory of GPU clusters

A GPU cluster can be modeled as an M/G/1 queue, where jobs arrive according to a Poisson process \((\lambda_{\text{job}})\) and service times follow a general distribution \((G)\) with mean \(1/\mu\) and standard deviation \(\sigma\). The Pollaczek-Khinchine formula defines the expected waiting time in the queue \((W_q)\):

\[ \frac{W_q}{1/\mu} = \frac{\rho}{1-\rho} \cdot \frac{1+C_s^2}{2} \]

Here, \(\rho = \lambda_{\text{job}}/\mu\) represents cluster utilization, \(1/\mu\) is the average job duration, and \(C_s = \sigma \mu\) is the Coefficient of Variation of job duration. The right-hand side is therefore the waiting time expressed as a multiple of the average job duration.

In standard web serving, request service times are often approximated by an exponential distribution, yielding \(C_s \approx 1\). In ML clusters, job durations follow a heavy-tailed distribution: a vast number of short debugging jobs (minutes) mixed with a few massive training runs (weeks). The representative \(C_s=3\) calculation in this callout captures that variance without claiming a universal fleet constant. The impact on wait times is multiplicative. For the scheduling argument here, the single-server multiplier is sufficient; Queuing theory for batched inference generalizes the result to the M/G/c/K queue and validates it against Little’s Law for readers sizing a scheduler’s admission queue under heavy-tailed arrivals.

At 80 percent utilization \((\rho = 0.8)\): with an exponential workload \((C_s = 1)\), \(W_q =\) 4 \(\times\) the average job duration. With a typical ML workload \((C_s = 3)\), \(W_q =\) 20 \(\times\) the average job duration. This explains the “utilization wall” in ML infrastructure. A web server cluster feels responsive at 80 percent load, but an ML cluster at the same utilization feels broken, with jobs languishing in the queue for days. The heavy tail of the service distribution acts as a latency multiplier, forcing operators to run ML clusters at lower utilization (often 60 to 70 percent) to maintain acceptable responsiveness for interactive users.

Bin packing

The most fundamental scheduling algorithm is bin packing:⁴ fitting jobs of varying sizes into fixed-capacity nodes. This problem is NP-hard in its general form, meaning no known algorithm finds optimal solutions in polynomial time. Fortunately, practical heuristics such as first-fit decreasing and best-fit decreasing achieve near-optimal results for the workload distributions typical of ML clusters, where job sizes follow a heavy-tailed distribution (many small jobs, few large ones).

⁴ Bin Packing: The one-dimensional version is NP-hard; ML scheduling adds four to five dimensions (GPU, CPU, memory, network, topology), making exact solutions intractable for clusters above a few hundred nodes. First-fit-decreasing heuristics achieve within 11/9 of optimal for typical workloads, but the real cost in ML clusters is not suboptimality in packing but fragmentation: stranded GPUs that individually satisfy no pending job yet collectively represent millions of dollars in idle capacity.

Consider a 64-node cluster with 8 GPUs per node, totaling 512 GPUs. If jobs request 6 GPUs each, each job occupies one full node but wastes 2 GPUs per node, reducing effective capacity to 75 percent. The remaining 2 GPUs per node cannot be combined across nodes because GPU workloads require local memory access. This Fragmentation grows worse with heterogeneous job sizes: a mix of 1-GPU, 3-GPU, and 7-GPU jobs creates irregular gaps distributed across many nodes, where no single pending job can fit into any individual gap, yet the total free capacity would be sufficient if the gaps were contiguous.

ML workloads make bin packing multi-dimensional in ways that traditional scheduling rarely encounters. Each job requires GPUs together with CPU cores for data preprocessing, host memory for data loading and augmentation pipelines, local SSD storage for dataset caching, and network bandwidth for gradient synchronization. A job requesting 4 GPUs, 32 CPU cores, and 256 GB of RAM may not fit on a node that has 4 free GPUs but only 16 free CPU cores because other jobs have consumed the host CPU for preprocessing. The scheduler must simultaneously satisfy all resource dimensions, and the job fits only if every dimension has sufficient capacity on the selected node. This multi-dimensional constraint dramatically reduces the solution space compared to single-dimensional packing, because a bottleneck in any single resource dimension can strand capacity in all others.

Locality constraints further restrict placement beyond simple resource availability. A training job using tensor parallelism requires GPUs connected via NVLink within the same node, since cross-node communication over InfiniBand is an order of magnitude slower (450 GB/s per direction intra-node vs. 50 GB/s per port inter-node). A job requesting “4 GPUs with NVLink connectivity” cannot use 2 GPUs from node A and 2 from node B, even if all 4 GPUs are individually available and the total capacity is sufficient. These topology constraints transform a packing problem into a placement problem where which specific resources matter as much as how many resources are available. The placement problem is strictly harder than the packing problem because it adds spatial constraints to the existing capacity constraints.

Schedulers address fragmentation through several complementary strategies, each targeting a different aspect of the problem. Backfill scheduling allows smaller jobs to fill gaps while larger jobs wait for contiguous resources, improving utilization without violating priority ordering. The insight is that small jobs can execute and complete in the gaps, freeing those resources before the large job’s target start time. Backfill scheduling requires estimating when current jobs will complete (to determine whether a backfill candidate will finish before the large job can start), which is why accurate runtime estimates are so important.

Periodic Defragmentation migrates or preempts low-priority jobs to consolidate free resources into contiguous blocks, analogous to memory compaction in operating systems. The scheduler identifies nodes where partial allocations leave stranded resources, preempts the jobs occupying those resources, and re-schedules them on nodes where they can be packed more efficiently. The cost of defragmentation (preempting running work, which wastes compute between the last checkpoint and the preemption event) must be weighed against the benefit (enabling large jobs to start sooner, improving overall throughput). Operators often schedule defragmentation during low-demand periods when the impact of preemption is lower.

A third strategy, Over-Subscription, allows more jobs to be admitted than strictly fit, relying on statistical multiplexing to avoid simultaneous peak usage across all resource dimensions. If each job’s GPU utilization averages 70 percent (alternating between compute-intensive and data-loading phases), the cluster can support approximately 1.4\(\times\) as many jobs as the strict GPU count would allow. Over-subscription works well when resource utilization is genuinely bursty and jobs’ peak usage periods are uncorrelated, but can cause severe performance degradation (thrashing, memory pressure, network congestion) when multiple jobs peak simultaneously. The key to safe over-subscription is monitoring actual utilization in real time and throttling admission when contention is detected. Figure 1 shows the combined effect: although 19 GPUs (roughly 30 percent of the cluster) sit idle, no single node has a contiguous block of eight, so a full-node job cannot be placed.

Figure 1: The Fragmentation Problem: The heatmap shows a snapshot of an 8-node cluster (rows) with 8 GPUs each (columns). Different colors represent distinct active jobs, while gray slots represent free GPUs. Although 19 GPUs (~30 percent of capacity) are available globally, they are fragmented across nodes. A new job requiring 8 contiguous GPUs (a full node) cannot be scheduled because no single row is completely empty, illustrating how bin-packing inefficiencies reduce effective cluster capacity.

Gang scheduling

Partial allocation of multi-GPU jobs creates a failure mode unique to ML clusters: two jobs can each hold half the resources they need, blocking each other indefinitely while every allocated GPU sits idle. Figure 2 contrasts this deadlock scenario with the atomic allocation approach that eliminates it.

Figure 2: The Distributed Deadlock (Hold and Wait): In a non-gang scheduler (left), job J1 acquires half the cluster and waits for the rest, while job J2 acquires the other half and waits for J1 to release its GPUs. Neither can proceed, and 100 percent of the cluster sits idle. Gang scheduling (right) solves this by mandating atomic allocation: a job is either granted its entire \(N\)-GPU requirement simultaneously or it waits in the queue without holding any resources.

Definition 1.1: Gang scheduling

Gang Scheduling is the ML cluster scheduling policy that allocates all of a job’s accelerators atomically (all-or-nothing) because synchronous distributed training makes the marginal value of a partial allocation zero: every worker must reach every collective, so a job cannot trade fewer GPUs for proportionally slower progress.

1. Significance: A 1,024-GPU synchronous job holding 512 GPUs completes zero training steps, not half as many: each AllReduce stalls until every rank arrives, so the held GPUs burn power and capital at 0 percent useful output. Without atomic allocation, two such jobs can each acquire half the cluster and deadlock, both at zero progress while consuming the full cluster’s operating cost.
2. Distinction: Unlike cloud service placement (where a service granted half its requested replicas serves roughly half its traffic) and unlike flexible HPC workloads (which degrade gracefully with fewer ranks), synchronous training has step-function utility: full allocation or nothing, which forces the scheduler to reason about atomic multi-resource allocations rather than incremental bin packing.
3. Common pitfall: A frequent misconception is that priorities or quotas alone solve this. Atomicity must extend to every co-scheduled resource the job needs to make progress (accelerators, high-bandwidth interconnect domains, network capacity for collectives); enforcing it at GPU granularity alone recreates the deadlock one resource layer down.

Example 1.1: The physics of deadlock

Scenario: A 1,024-GPU cluster receives 2 large distributed training jobs requiring 1,024 GPUs each. A naive scheduler allocates 512 GPUs to Team A and 512 GPUs to Team B.

Diagnosis: Neither job can proceed without its full node allocation, creating a hold-and-wait deadlock: Team A waits for Team B’s GPUs while Team B waits for Team A. Aggregate throughput drops to zero while consuming $2048/hour in idle compute cost.

Systems lesson: Distributed training workloads require atomic all-or-nothing (gang) scheduling. Fleet orchestrators must enforce job-level admission boundaries so that jobs remain queued until the entire requested GPU quota is available simultaneously.

The partial-allocation pattern is a Hold-and-Wait Deadlock⁵ in which two jobs each hold half the cluster and neither can proceed. The implementation guarantee is straightforward: for a job \(J\) requesting \(N\) GPUs, the scheduler must guarantee that either all \(N\) GPUs are allocated atomically, or the job remains in the queue without holding any resources. This binary outcome eliminates deadlock by construction, since a job that holds no resources cannot block other jobs. Implementing this guarantee efficiently, however, is the challenge that defines much of cluster scheduling algorithm design.

⁵ Hold-and-Wait Deadlock: One of the four classical conditions jointly sufficient for deadlock in resource-allocation systems. In GPU clusters, this condition is uniquely expensive: two jobs each holding 500 GPUs while waiting for 200 more strand 1,000 GPUs indefinitely, burning approximately $2,000/hour. Gang scheduling eliminates hold-and-wait by construction, requiring atomic all-or-nothing allocation at the cost of reduced packing flexibility.

Lifka, David A. 1995. “The ANL/IBM SP Scheduling System.” In Job Scheduling Strategies for Parallel Processing, edited by Dror G. Feitelson and Larry Rudolph, vol. 949. Lecture Notes in Computer Science. Springer. https://doi.org/10.1007/3-540-60153-8_35.

Naive gang scheduling is wasteful in predictable ways. If the cluster has 900 free GPUs and a job requests 1,024, the 900 GPUs sit idle until 124 more become available, potentially wasting thousands of GPU-hours. Backfill scheduling addresses this by identifying jobs in the queue that can fit within the 900 available GPUs without delaying the large job’s expected start time, an idea popularized in the ANL/IBM SP scheduling system (Lifka 1995). A 128-GPU job with an estimated runtime of 2 hours can safely backfill if the 124 additional GPUs needed by the large job will remain unavailable for at least 2 hours and the backfill job is expected to finish before the large job’s reservation. The backfilled job uses resources that would otherwise be idle, improving utilization without violating the large job’s scheduling guarantee.

The accuracy of runtime estimates determines backfill effectiveness, and this creates a game-theoretic challenge. If users consistently overestimate runtimes, backfill slots are artificially narrow and fewer jobs fit, reducing utilization. If users underestimate, backfilled jobs may still be running when the large job’s resources become available, forcing preemption that wastes the backfilled job’s progress. In practice, users have strong incentives to overestimate (to avoid being killed before completion) and weak incentives to estimate accurately, leading to systematic inflation that degrades backfill performance. Research schedulers like Tiresias (Gu et al. 2019) address this fundamental problem by eliminating the requirement for runtime estimates entirely, instead using observed resource consumption to dynamically adjust priority. This approach, discussed in detail in section 1.5.1, turns the runtime estimation problem from a user-facing burden into a system-level observation.

A quick cost estimate shows why even modest improvements in this balance justify engineering effort.

Systems Perspective 1.1: The economics of idle GPUs

The cluster economics introduced at the start of this chapter set the stake; the gang-scheduling decision is where a scheduler first earns or burns it. Taking the same 10,000-GPU cluster:

• Operating cost: $480,000/day ($175.2M/year)
• Lower Utilization (60 percent): 6,000 productive, 4,000 idle = $192,000/day wasted
• Higher Utilization (80 percent): 8,000 productive, 2,000 idle = $96,000/day wasted
• Improvement value: Moving from 60 percent to 80 percent saves $96,000/day, or about $35.0M/year

Gang scheduling delivers this utilization gain by preventing deadlock. A single hold-and-wait deadlock can strand the whole cluster, erasing more than a full day of the gain in one event.

Gang scheduling removes the hold-and-wait deadlock from the utilization example, but another wait-for pattern remains: priority inversion can still strand reserved GPUs behind jobs that cannot finish their own exit path.

Priority inversion is a wait-for chain, not a root-failure tree.

Deadlock prevention and detection

Gang scheduling eliminates the hold-and-wait condition, one of the four formal Coffman conditions for deadlock, but it does not immunize the cluster against all forms of resource contention. Deadlocks can still emerge from the interaction of priority rules, preemption policies, and auxiliary resource dependencies. In a cluster with thousands of GPUs and petabytes of state, these edge cases transition from theoretical curiosities to daily operational incidents.

The most pernicious of these is priority inversion, a scenario borrowed from real-time systems where a high-priority job is indefinitely blocked by a low-priority job. Consider a 175B parameter training run (high priority) requesting a gang of 64 GPUs. The scheduler has reserved 60 available GPUs, but the remaining 4 are held by a low-priority data processing job. Normally, the scheduler would preempt the low-priority job to satisfy the high-priority request. However, if a stream of medium-priority development jobs saturates the cluster’s CPU or network bandwidth, starving the low-priority job of the resources it needs to checkpoint and exit, the low-priority job stalls. It cannot release the GPUs because it cannot complete its exit sequence. The high-priority job waits for the low-priority job, which is effectively blocked by the medium-priority jobs. The result is a 60-GPU idle block that persists until an operator manually intervenes.

Definition 1.2: Priority inversion

Priority Inversion is an ML cluster scheduling pathology in which a high-priority task is forced to wait for a lower-priority task to release a shared resource.

1. Significance: It reduces the progress rate of the entire fleet to that of the lowest-priority job. In ML clusters, this typically occurs when a low-priority job holding GPUs is starved of auxiliary resources (for example, \(\text{BW}\) for checkpointing), preventing it from finishing and releasing the accelerators needed by high-priority workloads.
2. Distinction: Unlike standard queuing (where tasks wait their turn), priority inversion involves an active blockage: the high-priority task is ready to run but is transitively dependent on a task that the scheduler does not prioritize.
3. Common pitfall: A frequent misconception is that strict priority levels solve this. In reality, without holistic preemption (reserving all resources needed for a task to exit), increasing priority levels can actually increase the likelihood of inversion by creating more complex dependency chains.

Solving this requires a choice between prevention and detection. Prevention mechanisms, like strict gang scheduling and aggressive timeouts, eliminate deadlock states by construction but often sacrifice utilization. Detection mechanisms allow the system to enter potentially unsafe states to maximize throughput, relying on monitoring to identify and resolve deadlocks when they occur.

For large-scale fleets, exact placement search is computationally nontrivial. A cluster with 10,000 GPUs and 500 pending jobs has a state space exceeding \(10^{15}\) possible allocations, making exhaustive allocation search intractable. By contrast, detecting a cycle in an already constructed wait-for graph is linear in the graph’s vertices and edges. The practical difficulty is maintaining a complete, current global graph while resource state changes across a distributed control plane. Large-scale schedulers therefore supplement wait-for detection with mechanisms that preserve liveness under incomplete state. A Lease-Based Resource gives every allocation a time-to-live, allowing the scheduler to reclaim GPUs when a job stops renewing its claim. Progress Monitoring then checks whether the job is actually advancing, using signals such as GPU utilization, step counters, or heartbeat timestamps to identify zombie allocations. When the scheduler must preempt a lower-priority job, Holistic Preemption reserves the CPU, network, and storage bandwidth needed for that job to checkpoint and exit cleanly; otherwise, the victim can be too starved to release the accelerators. These strategies accept a nonzero rate of false positives to preserve cluster liveness.

Heterogeneous gang scheduling

The traditional gang scheduling model assumes a homogeneous bulk synchronous parallel workload: \(N\) identical workers executing the exact same compute graph. Some training workflows break this assumption because one logical job contains several roles with different resource profiles. In preference-training pipelines such as reinforcement learning from human feedback (RLHF), for example, one component may update the policy while another scores outputs or supplies reference behavior. The scheduler still sees one job, but the gang contains trainers, evaluators, and inference-style workers that need different GPU counts, memory footprints, and lifetimes.

Systems Perspective 1.2: Routing over synchrony

Heterogeneous gang scheduling reframes this allocation. The scheduler must atomically allocate a diverse constellation of resources: high-HBM nodes optimized for memory-bound inference generation, alongside high-compute nodes for compute-bound backpropagation. The orchestration challenge shifts from static topological bin-packing to dynamic data routing. Activations and gradients are no longer just reduced within a homogeneous ring; instead, data must stream continuously between inference nodes generating rollouts and training nodes computing policy updates. If the heterogeneous gang allocation is not perfectly balanced across these distinct workload profiles, the entire cluster falls out of sync, leaving expensive accelerators starved for data.

Heterogeneous RLHF gangs are the exception; for common homogeneous training jobs, the recurring tension is gang scheduling’s safety against backfill’s utilization. Gang scheduling prevents deadlock but wastes resources; backfill improves utilization but requires runtime estimates that users cannot accurately provide. Two common orchestration paradigms resolve this tension through fundamentally different architectural philosophies.

Orchestration Paradigms

The gang-scheduling/backfill tension does not have a tool-neutral answer: the orchestration paradigm determines which guarantee the platform protects first. Slurm, shaped by HPC batch systems, favors explicit resource requests and predictable allocation. Kubernetes, shaped by cloud-native service management, favors declared desired state and continuous reconciliation. The choice is therefore not Slurm vs. Kubernetes as products; it is which control model best matches the workload mix, failure behavior, and operational culture of the fleet.

The fundamental distinction is between imperative and declarative resource management. In an imperative system, users specify exactly what resources they need and the system allocates them directly. In a declarative system, users specify what they want running and the system converges toward that state. This distinction, familiar from programming language design, determines how ML workloads are scheduled, monitored, and recovered from failure.

Slurm: The HPC heritage

Slurm Workload Manager⁶ originated at Lawrence Livermore National Laboratory as a scalable, fault-tolerant resource manager for Linux clusters (Yoo et al. 2003). Its batch-oriented design fits the distinctive requirements of scientific computing: long-running jobs, expensive shared hardware, and users who can specify resource needs in advance. In ML infrastructure, the same model maps to GPU training through partitions for heterogeneous accelerator pools and predictable allocations for long-running distributed jobs.

⁶ Slurm Workload Manager: Originally “SLURM,” an acronym for Simple Linux Utility for Resource Management, the project dropped the acronym capitalization in 2012. Developed at Lawrence Livermore National Laboratory beginning in 2002, Slurm’s centralized controller (slurmctld) maintains a single authoritative view of resource allocations. At very large scale, operators may partition fleets or federate schedulers to keep scheduling latency, policy isolation, and failure domains manageable.

Yoo, A. B., M. A. Jette, and M. Grondona. 2003. “SLURM: Simple Linux Utility for Resource Management.” In Job Scheduling Strategies for Parallel Processing. Springer Berlin Heidelberg. https://doi.org/10.1007/10968987_3.

Slurm uses an Imperative Scheduling Model: users submit job scripts specifying exact resource requirements, and the scheduler places jobs into partitions based on priority, fairness, and resource availability. This directness makes reasoning about allocation guarantees straightforward. When a user submits sbatch --gres=gpu:8 --nodes=4, Slurm guarantees that if the job starts, it will have exactly 32 GPUs across 4 nodes. The user knows precisely what they are getting, and the scheduler knows precisely what it must provide. The cost of this directness is that users must understand their resource needs precisely; requesting more than needed wastes resources, while requesting less causes out-of-memory errors or reduced performance.

Slurm’s architecture consists of a central controller daemon (slurmctld) that maintains the global view of cluster state and makes scheduling decisions, and per-node daemons (slurmd) that manage local resources and execute jobs. This centralized architecture provides the authoritative allocation model discussed in section 1.0.1: the controller is the source of truth for resource assignments, reducing the risk of conflicting allocations. The cost is a potential single point of failure (addressed through active-passive failover) and a scheduling-throughput limit. As fleets grow, organizations may partition the cluster or federate schedulers to bound scheduling latency and failure impact.

A typical ML cluster configuration defines Slurm partitions by accelerator type and interconnect (table 1).

Table 1: Slurm Partition Configuration: Partitions organize heterogeneous accelerators into logical pools matched to workload characteristics. NVLink-connected partitions support tensor parallelism, while PCIe partitions serve workloads that rely primarily on data parallelism. Separating inference and debug partitions prevents experimental workloads from impacting production serving.

Partition	GPUs/Node	Interconnect	Typical Use
dgx-a100	8\(\times\) A100	NVLink + IB NDR	Large LLM training
a100-pcie	4\(\times\) A100	PCIe + IB HDR	Medium training
inference	2\(\times\) A10G	Ethernet	Model serving
debug	1\(\times\) V100	Ethernet	Development

Those partitions make GPU allocation policy concrete, and Slurm provides several mechanisms for controlling placement. The --gres=gpu:N flag requests N GPUs per node, while --gpus=N requests a total GPU count for the job. Naive allocation can fragment nodes: if jobs request 6 GPUs on 8-GPU nodes, each job wastes 2 GPUs per node, reducing effective capacity to 75 percent. Slurm’s select/cons_tres plugin enables GPU-aware consumable-resource scheduling, tracking individual GPUs rather than treating nodes as indivisible units. The --gpus-per-node flag requests a fixed GPU count on each allocated node when NVLink communication patterns make partial-node allocation counterproductive, while --gpus-per-task distributes GPUs evenly across tasks for data-parallel workloads.

The interaction between GPU allocation and node selection creates subtleties that affect both performance and utilization. A job requesting --gpus=16 --gpus-per-node=8 will always receive exactly 2 complete nodes, ensuring all 8 GPUs within each node communicate via NVLink. The same job requesting --gpus=16 without the per-node constraint might receive GPUs spread across 3 or 4 partially occupied nodes, degrading intra-node communication performance. For training jobs using tensor parallelism, the per-node constraint is essential; for data-parallel jobs that communicate only through AllReduce, the flexibility of unconstrained placement improves the scheduler’s ability to find valid allocations and reduces fragmentation.

The Fair-Share Scheduling mechanism prevents any single user or project from monopolizing resources over time. The core insight is that past usage should affect future priority: heavy users should yield to lighter users when the cluster is contended. The effective-priority heuristic combines base priority, the user’s target fair share (\(F_{\text{target}}\)), the user’s recent actual consumption (\(F_{\text{actual}}\)), and a small stabilizing constant (\(\epsilon\)):

\[\text{Priority}_{\text{effective}} = \text{Priority}_{\text{base}} \times \frac{F_{\text{target}}}{F_{\text{actual}} + \epsilon} \tag{1}\]

The fair-share formula in equation 1 naturally deprioritizes heavy users while allowing burst access when resources are idle. A researcher who has consumed twice their fair share sees their priority halved, pushing new submissions behind colleagues who have used less. Conversely, a researcher who has used no resources recently receives maximum priority, allowing rapid access when they return to active work.

The time decay of usage history determines how quickly the system “forgives” past heavy usage. A half-life of one week means that a researcher’s heavy usage from two weeks ago contributes only 25 percent to their current usage calculation. Short half-lives create rapid rebalancing but can lead to oscillatory behavior where users alternate between starving and gorging on resources. Long half-lives create stable priority ordering but may penalize researchers who completed a legitimate large project and now want resources for a smaller one. Most production clusters configure half-lives between 3 and 14 days, balancing responsiveness against stability.

Preemption policies enable high-priority jobs to reclaim resources from running workloads, and for ML training, this requires careful coordination with the checkpoint infrastructure. Slurm can notify selected jobs with SIGTERM and use GraceTime to delay final termination, typically giving 60 to 300 seconds for a checkpoint before the eventual SIGKILL. PreemptMode=REQUEUE requeues eligible batch jobs (for example, jobs submitted with --requeue or clusters with JobRequeue=1), and the job’s startup path must resume from the latest checkpoint. The checkpoint and recovery infrastructure developed in Fault Tolerance makes this preemption practical; without reliable checkpointing, preemption would lose all progress since the last manually triggered save, making preemption economically ruinous for long-running training jobs.

Preemption introduces a scheduling paradox: it improves the scheduler’s ability to serve high-priority work but can degrade overall cluster throughput. Every preemption wastes the compute between the last checkpoint and the preemption event, plus the time to restart and reload from checkpoint. If checkpoint intervals are long (for example, hourly) and preemptions are frequent, the wasted compute can be substantial. Production systems balance preemption frequency against checkpoint overhead, often configuring preemption cooldown periods that prevent the same job from being preempted more than once within a configurable interval.

Slurm’s strengths for ML training are clear: predictable allocation guarantees, mature fair-share policies, straightforward integration with MPI-based distributed training frameworks, and decades of operational experience in managing large-scale scientific computing. Its weaknesses emerge for inference workloads and mixed training-serving clusters, where the batch-oriented model struggles with the dynamic, latency-sensitive nature of serving traffic. Slurm has no native concept of “a service that should always be running,” making it awkward for inference endpoints that must respond to requests continuously. Adding or removing inference replicas requires submitting or canceling Slurm jobs, a much heavier operation than scaling a Kubernetes deployment.

Advanced Slurm configuration for ML

Once Slurm owns batch training, the remaining question is how much ML semantics the batch system can see. Standard partitions and fair-share policies schedule ordinary jobs; large-scale deep learning needs configuration that exposes sweep structure, asymmetric pipeline resources, hardware health at job boundaries, and the economic weight of different accelerator types.

Job arrays transform the chaotic submission of thousands of experimental hyperparameter sweep trials into a coherent, schedulable unit. Instead of flooding the controller with individual sbatch requests, a user submits a single array job: #SBATCH --array=0-99. Slurm treats this as a single object for parsing and queuing overhead but schedules each element as an independent task, allowing the scheduler to backfill small holes in the cluster with individual trials. Each task receives a unique SLURM_ARRAY_TASK_ID environment variable, which the training script uses to index into a hyperparameter configuration file (for example, selecting learning rate \(\eta = 10^{-4}\) for task 0 and \(\eta = 3 \times 10^{-4}\) for task 1). This mechanism is common for large-scale ablation studies, enabling researchers to launch 1,000 experiments with a single command while preserving scheduler throughput.

Heterogeneous jobs address the preprocessing bottleneck where expensive GPUs sit idle while CPUs prepare data. Traditional jobs allocate symmetric resources to every node, forcing users to reserve GPU nodes for the entire pipeline. Slurm heterogeneous job support lets a single submission span disparate resource types by separating components with : on the command line or #SBATCH hetjob inside a script. The 64-GPU fine-tuning run for our 175B model (the scheduling-demo workload, not the full 1,024-GPU pretraining run) uses this pattern: it requests a heterogeneous allocation of 8 GPU nodes (64 A100s) for the model and 2 CPU-only nodes for on-the-fly data tokenization. The srun --het-group=0 command launches the training process on the GPUs, while srun --het-group=1 launches the data workers, allowing the expensive accelerators to focus purely on gradient computation while cheaper CPU nodes feed them data.

Prolog and epilog scripts serve as the cluster’s immune system, running administrative code before a job starts and after it finishes. In ML clusters, a typical prolog script performs a preflight check on allocated hardware: it validates NVLink topology (using nvidia-smi topo -m), checks ECC error counters, and verifies InfiniBand link width. If a node fails these checks, the prolog returns a nonzero exit code, causing Slurm to drain the node and requeue the job elsewhere, preventing a silent hardware fault from corrupting a week-long training run. The epilog script ensures hygiene by killing orphaned Python processes, clearing shared memory segments (/dev/shm), and logging final GPU utilization metrics to the accounting database. For the 64-GPU fine-tuning run, the prolog script specifically validates that all 8 GPUs on each node have full P2P bandwidth access, preventing a single degraded NVLink lane from bottlenecking the entire tensor parallel group.

Finally, Trackable Resources (TRES) extend Slurm’s accounting beyond simple CPU/memory tracking to assets like GPU-hours, license tokens, or power budget. The capability that matters is economic-weight accounting per accelerator class: each tracked resource carries a billing weight, so fair-share can be driven by actual economic cost rather than raw core counts. This ensures that a team using 100 older GPUs does not deplete its budget as fast as a team using 100 flagship H100s, and it lets organizations implement project billing for expensive resources directly inside the scheduler’s fairness calculation.

These extensions make Slurm more ML-aware while preserving its batch-scheduler model. Kubernetes starts from a different contract: users declare the desired state of services and jobs, and controllers continuously reconcile the cluster toward that state.

Kubernetes: Declarative orchestration

Kubernetes is a common platform for ML infrastructure, particularly for organizations requiring unified management of training and serving workloads (Burns et al. 2016; Cloud Native Computing Foundation 2024). Where Slurm’s model is imperative (“run this job on these resources”), Kubernetes uses a Declarative Model (“ensure this state exists”), using control loops that continuously reconcile desired state with actual state. This fundamental difference shapes every aspect of ML workload management, from job submission to failure recovery.

Burns, Brendan, Brian Grant, David Oppenheimer, Eric Brewer, and John Wilkes. 2016. “Borg, Omega, and Kubernetes.” Communications of the ACM 59 (5): 50–57. https://doi.org/10.1145/2890784.

Cloud Native Computing Foundation. 2024. Kubernetes: Production-Grade Container Orchestration.

The declarative model’s power lies in its approach to failure handling. Slurm’s recovery is configuration-dependent and batch-oriented: slurmctld can detect node failures through slurmd heartbeats and requeue jobs when Requeue=1 is set, but the recovery semantics require explicit configuration to behave like a continuously-reconciling control loop. In Kubernetes, the control loop continuously reconciles desired state (“4 replicas of this serving worker should be running” for a Deployment, or the configured retry policy for a Job) against actual state, automatically creating a replacement pod on a healthy node when divergence is detected. For distributed training specifically, replica-replacement semantics typically come from a higher-level operator such as Kubeflow’s PyTorchJob or MPIJob rather than core Kubernetes primitives. This self-healing behavior reduces operational burden but introduces complexity: the system’s actions are emergent from control loop interactions rather than explicitly commanded, making debugging more challenging when things go wrong.

Kubernetes’ control loop architecture is built on the Controller Pattern:⁷ a controller watches the cluster’s actual state (stored in etcd, Kubernetes’ distributed key-value store for API state), compares it to desired state (specified by users through API objects like Deployments, Jobs, and StatefulSets), and takes corrective action to close the gap. This pattern repeats at every level: the Deployment controller ensures the right number of pods exist, the scheduler assigns pods to nodes, the kubelet on each node ensures containers are running, and the node controller detects node failures. Each controller operates independently, communicating only through shared state in etcd, which makes the system resilient to individual controller failures but can create emergent behaviors when multiple controllers interact.

⁷ Controller Pattern (Reconciliation Loop): A “level-triggered” design where the controller continuously compares desired state to actual state, as opposed to “edge-triggered” systems that react to individual events. The distinction matters for ML clusters: if a controller crashes and restarts, it re-reads current state and self-heals without needing an event log. The trade-off is that emergent interactions between multiple independent controllers can create unexpected scheduling behaviors that are difficult to debug sequentially.

Native Kubernetes lacks ML-aware scheduling, but extensions address this gap. GPU scheduling relies on Device Plugins that expose accelerators as schedulable extended resources. The NVIDIA device plugin registers GPUs with the kubelet (the node-level agent), enabling pod specifications that request GPU resources declaratively through the standard Kubernetes resource model. Listing 1 shows the resulting pod fragment.

Listing 1: Kubernetes GPU Allocation: Pod resource specification requesting GPUs through the NVIDIA device plugin. The nvidia.com/gpu resource name follows Kubernetes extended resource conventions, where the domain prefix identifies the device plugin vendor. This declarative syntax enables portable GPU workload definitions across any Kubernetes cluster with the NVIDIA device plugin installed.

# Kubernetes pod resource specification for GPU allocation
resources:
  limits:
    nvidia.com/gpu: 4  # Request exactly 4 GPUs for this pod
  requests:
    cpu: "32"
    memory: "256Gi"

This binary allocation model creates a significant inefficiency for inference workloads. A small model serving occasional requests might need only a fraction of a GPU’s compute and memory capacity, yet Kubernetes allocates an entire GPU, wasting the remaining capacity. For training workloads that saturate GPU compute, full-device allocation is appropriate. For inference workloads with variable and often modest resource needs, it creates the same kind of fragmentation that partial-node allocation creates in Slurm.

Multi-Instance GPU (MIG)⁸ technology addresses this inefficiency by partitioning A100 (80 GB) and H100 (80 GB) GPUs into hardware-isolated instances. Unlike software-based GPU sharing, MIG dedicates memory, cache, and compute resources to each instance, preventing one workload from interfering with another and eliminating the noisy neighbor⁹ effect. The A100 MIG profiles in table 2 show the scheduling consequence: a single physical A100 becomes several fixed-size resources that the device plugin can expose as independently schedulable accelerators.

⁸ Multi-Instance GPU (MIG): Introduced with the A100 (2020), MIG partitions a single GPU into up to 7 hardware-isolated instances with dedicated memory controllers, L2 cache slices, and compute units (NVIDIA Corporation 2020). The isolation is enforced at the hardware level, not by time-slicing, reducing noisy-neighbor interference relative to software-only sharing. The trade-off is rigidity: partition profiles cannot change without draining all workloads, making MIG poorly suited for clusters with rapidly shifting workload mixes.

NVIDIA Corporation. 2020. NVIDIA A100 Tensor Core GPU Architecture. NVIDIA Whitepaper, V1.0.

⁹ Noisy Neighbor: A metaphor from multi-tenant housing where one resident’s activity (loud music) disturbs another’s peace. In GPU clusters, this occurs when one job’s bursty network traffic or memory bus utilization slows down a co-located job on the same node. For distributed training, noisy neighbors are fatal to efficiency because the slowest flow dictates the global AllReduce time.

Table 2: A100 MIG Profiles: Multi-Instance GPU configurations trade GPU memory and compute resources for increased sharing density. Each profile provides hardware-isolated resources, making MIG suitable for multi-tenant inference clusters where workloads from different teams or customers must not interfere. The streaming multiprocessor (SM) count determines compute throughput within each instance.

MIG Profile	GPU Memory	SM Count	Typical Workload
1g.10gb	10 GB	14 SMs	Small inference
2g.20gb	20 GB	28 SMs	Medium inference
3g.40gb	40 GB	42 SMs	Large inference
7g.80gb	80 GB	98 SMs	Training

As section 1.0.5 explains, default Kubernetes pod scheduling is independent, so ML platforms add admission or gang semantics when all-or-nothing startup matters. The default scheduler evaluates pods one at a time, placing each on the best available node. For a distributed training job with 64 pods, this means 64 independent scheduling decisions, with no guarantee that all 64 will succeed. If the cluster has resources for 60 pods but not 64, the default scheduler may place 60 pods and leave the remaining 4 pending, creating exactly the partial-allocation waste that gang scheduling prevents. PodGroup-style abstractions, whether implemented by scheduler plugins or higher-level batch controllers, give the platform a unit of admission that matches the training job rather than the individual pod.

The Volcano¹⁰ batch scheduler and Coscheduling scheduler plugin address this gap by implementing gang semantics through PodGroup abstractions. A PodGroup declares that a set of pods must be scheduled together, specifying a minimum member count that must be satisfiable before any pod in the group starts. The scheduler evaluates the entire group atomically: either all minimum members can be placed, and they are placed simultaneously, or none are placed and the group waits in the queue. This transforms Kubernetes’ pod-level scheduling into job-level scheduling that respects the all-or-nothing requirements of distributed training.

¹⁰ Volcano: Open-sourced by Huawei and now a Cloud Native Computing Foundation (CNCF) incubating project, Volcano replaces the Kubernetes default scheduler entirely to add gang scheduling via its PodGroup CRD. The replacement approach provides strong atomicity guarantees for multi-pod ML training jobs but carries operational risk: a bug in Volcano affects all scheduling decisions on the cluster, not just batch workloads.

Priority classes control preemption behavior in Kubernetes, establishing a hierarchy that determines which workloads yield resources to which others. A typical production configuration assigns inference workloads high priority (ensuring serving service-level objectives (SLOs) are met), training workloads medium priority, and development jobs low priority. The default PriorityClass behavior, preemptionPolicy: PreemptLowerPriority, allows a higher-priority pending pod to evict lower-priority pods when that would make room for it. A PriorityClass with preemptionPolicy: Never is non-preempting: its pods can sit ahead of lower-priority pods in the scheduling queue, but they do not evict running pods and can still be preempted by still-higher-priority pods. Organizations must carefully balance the priority hierarchy: overly aggressive inference preemption can thrash training jobs (repeatedly preempting and restarting them), while overly permissive training priorities can starve inference during demand spikes. The Kubernetes priority and preemption system integrates with checkpoint-aware preemption patterns developed later in Fault Tolerance, where preempted training jobs save state before yielding resources, minimizing wasted computation.

Kueue,¹¹ a newer Kubernetes-native job queueing system, represents an emerging approach that separates admission control from scheduling. Rather than replacing the Kubernetes scheduler entirely (as Volcano does), Kueue manages when jobs enter the cluster, while the default scheduler (or any compatible scheduler) handles where pods are placed. This separation of concerns has practical advantages: Kueue can be deployed alongside existing Kubernetes infrastructure without disrupting running workloads, and it integrates naturally with the ecosystem of scheduler plugins for topology-aware placement and other features.

¹¹ Kueue: Developed by Google, Kueue separates admission control (when jobs enter the cluster) from scheduling (where pods are placed), unlike Volcano which replaces the scheduler entirely. This less-invasive design can be deployed alongside existing Kubernetes infrastructure without disrupting running workloads, but it lacks Volcano’s strong gang scheduling guarantees, forcing teams to choose between deployment safety and scheduling atomicity.

Kueue provides fair-share scheduling through ClusterQueues that represent organizational teams or projects. Each queue has resource quotas, borrowing policies that allow queues to use idle capacity from other queues, and preemption policies that reclaim borrowed resources when the owning queue needs them. A research team queue might borrow idle capacity from a production team queue during off-peak hours, automatically returning resources through preemption when the production team submits urgent work. This borrowing mechanism mirrors the hierarchical fair-share approaches in Slurm but expressed through Kubernetes’ declarative model rather than Slurm’s imperative configuration.

Kubernetes ecosystem for ML training

Kubernetes provides the orchestration primitives, but a specialized ecosystem of operators and plugins is required to bridge the gap between microservice orchestration and high-performance training. Each extension makes one hidden operational constraint visible to the platform: job atomicity, network bypass, checkpoint storage, or telemetry. This ecosystem augments Kubernetes with the batch processing, hardware acceleration, and observability capabilities found in HPC environments.

The Training Operator Ecosystem simplifies distributed training by extending Kubernetes with job-specific Custom Resource Definitions (CRDs). The Kubeflow Training Operator provides controllers for PyTorchJob, TFJob, and MPIJob resources, automating the complex lifecycle of distributed workloads. A PyTorchJob specification defines the number of workers, the container image, and resource requirements; the operator then creates the necessary pods and configures distributed environment variables such as MASTER_ADDR, MASTER_PORT, WORLD_SIZE, and RANK. If a worker pod fails, the operator can detect the deviation and create a replacement pod to restore the desired replica count, but training-state recovery still depends on the framework’s checkpoint and rendezvous logic.

Achieving high network performance in Kubernetes often requires bypassing networking layers that were designed for ordinary microservices rather than tightly synchronized collectives. Container Network Interface configuration, overlay networking, and CPU switching can add latency or reduce effective bandwidth when they sit on the critical gradient-synchronization path. For bandwidth-sensitive training, Host Networking (hostNetwork: true) allows pods to bypass container networking entirely, granting direct access to the host’s network namespace and remote direct memory access (RDMA) devices. The NVIDIA Network Operator automates the low-level plumbing required for this performance, managing RDMA device plugins, SR-IOV virtual function assignment, and GPUDirect RDMA configuration. This bypass capability allows containerized workloads to approach bare-metal interconnect performance when the remaining stack is configured correctly.

Storage integration addresses the “checkpoint storm” problem where thousands of GPUs simultaneously write terabytes of state. Kubernetes PersistentVolumeClaims (PVCs) abstract the underlying storage fabric, which may be a parallel file system like Lustre or GPFS, or high-performance object storage via Container Storage Interface drivers. The critical architectural challenge is preventing these synchronized writes from saturating the storage backend. Storage classes with quality-of-service policies and client-side throttling are essential to ensure that a 1,024-GPU training job writing a 2 TB checkpoint does not starve other workloads or crash the storage metadata server.

The Observability Stack for training clusters combines general-purpose cluster monitoring with specialized hardware telemetry. Prometheus and Grafana provide the collection and visualization layer, while the Data Center GPU Manager (DCGM) exporter exposes granular GPU metrics: utilization, memory bandwidth, temperature, and ECC errors. This stack enables the utilization dashboards essential for multi-tenant efficiency, allowing operators to distinguish between true compute saturation and I/O-bound idleness.

The 1,024-GPU pretraining run for our 175B model uses this full Kubernetes stack. The workload is defined as a PyTorchJob with 128 worker pods, each requesting 8 GPUs. Host networking is enabled to allow direct InfiniBand access for the NVLink-connected nodes. A PVC backed by a high-throughput Lustre file system provides access to the 100 TB training dataset and absorbs periodic checkpoints. DCGM metrics stream to Prometheus, triggering alerts if any GPU reports uncorrectable ECC errors or if NVLink bandwidth drops below the expected baseline.

Choosing between paradigms

The choice between Slurm and Kubernetes is rarely binary; it depends on workload composition, organizational context, and the relative importance of different system qualities. Understanding where each paradigm excels guides the architectural decision.

Slurm excels for pure training clusters where predictability and bare-metal performance matter most. Its direct resource management avoids the container overhead that Kubernetes introduces (container networking adds microseconds of latency and reduces effective network bandwidth by 1 to 5 percent), and its batch scheduling model aligns naturally with long-running training jobs that require guaranteed, uninterrupted access to specific hardware. Slurm’s centralized scheduler has global visibility into cluster state, enabling sophisticated multi-factor priority decisions that account for fair-share, job size, queue depth, and resource availability simultaneously. National laboratories, university research clusters, and dedicated training infrastructure typically favor Slurm because these environments prioritize maximum hardware utilization and minimal overhead for a relatively homogeneous set of batch workloads.

Kubernetes excels for mixed training-serving clusters where unified management, rolling updates, and service mesh integration reduce operational complexity. Organizations running both training pipelines and production inference endpoints benefit from a single control plane that manages both workload types through a consistent API. Kubernetes’ declarative model simplifies operational workflows: deploying a new model version means updating a Deployment specification, and Kubernetes handles rolling updates, health checks, and rollback automatically. Cloud-native organizations, teams deploying ML as microservices, and organizations with existing Kubernetes expertise typically favor Kubernetes because the marginal cost of adding ML workloads to an existing platform is lower than operating a separate scheduling system.

Many production environments recognize that neither paradigm alone satisfies all requirements, and they deploy both systems in complementary roles. A common architecture uses Slurm for dedicated training clusters where raw performance and scheduling sophistication matter, and Kubernetes for inference serving, pipeline orchestration, and lighter training workloads (fine-tuning, small-scale experimentation). The emerging pattern is to use Kubernetes as the outer orchestration layer, submitting Slurm jobs for large training runs through Kubernetes operators. This hybrid architecture achieves unified management and observability without sacrificing Slurm’s batch scheduling capabilities for the workloads that need them most.

The Slurm vs. Kubernetes trade-off organizes around the dimensions most relevant to ML workloads (table 3).

Table 3: Slurm vs. Kubernetes for ML Workloads: The two paradigms reflect different design philosophies optimized for different workload characteristics. Most production ML platforms use elements of both, with Slurm managing dedicated training clusters and Kubernetes managing inference serving and pipeline orchestration.

Dimension	Slurm	Kubernetes
Scheduling model	Imperative: user specifies exact resources and scheduler allocates them	Declarative: user specifies desired state, system reconciles continuously
Failure handling	Configuration-dependent requeue and checkpoint workflows	Self-healing through control loops
Gang/atomic multi-node start	Atomic job allocation plus backfill; Slurm “gang scheduling” specifically refers to time-slicing/preemption	Production clusters commonly use PodGroup-based support such as Volcano, Coscheduling, Kueue, or upstream alpha gang scheduling
Fair-share	Mature multi-factor priority with configurable decay	Kueue provides basic fair-share with ClusterQueue borrowing
Container overhead	None from a container runtime when jobs run directly on nodes	Configuration-dependent networking overhead and container startup time
Service management	No native support for long-running services	Native Deployment, rolling updates, health checks
Ecosystem	MPI, OpenMP, scientific computing libraries	Cloud-native, microservices mesh, observability stack

The choice of scheduling paradigm also reflects organizational trajectory (table 3). Teams starting with HPC-focused training infrastructure may begin with Slurm and add Kubernetes for serving. Teams starting from cloud-native application infrastructure may begin with Kubernetes and add HPC-oriented extensions (Volcano, Kueue) for training. Either path can reach a functional hybrid architecture, but the starting point determines which capabilities are strongest and which require the most investment to develop.

Hybrid architectures in practice

While the choice between Slurm and Kubernetes often appears binary, sophisticated engineering organizations increasingly deploy Hybrid Architectures that use the strengths of both systems. These architectures acknowledge a fundamental reality: training workloads behave like HPC applications (batch, synchronous, monolithic), while serving workloads behave like microservices (continuous, asynchronous, elastic). Forcing both into a single paradigm often results in a “lowest common denominator” system that serves neither well.

The Kubernetes-over-Slurm pattern wraps the HPC scheduler within a cloud-native control plane. In this architecture, Kubernetes acts as the outer orchestration layer, managing the lifecycle of training jobs through custom operators while delegating the actual pod placement to Slurm. A developer submits a TrainingJob CRD to Kubernetes, which the operator translates into a Slurm sbatch script. The operator then monitors the job status via Slurm’s REST API, streaming logs and metrics back to the Kubernetes control plane. This approach provides the best of both worlds: the unified observability, CI/CD integration, and access control of Kubernetes, combined with the gang scheduling sophistication, topology awareness, and bare-metal performance of Slurm.

The Slurm-alongside-Kubernetes pattern takes a coarser-grained approach, maintaining separate clusters for training and serving that draw from a shared pool of physical nodes. A meta-scheduler or capacity broker acts as an arbitrator, dynamically reassigning nodes between the Slurm partition and the Kubernetes cluster based on demand signals. During intensive training campaigns, nodes are drained from Kubernetes and joined to Slurm; during product launches or traffic spikes, the flow reverses. The primary challenge is the mode-switching cost: draining a Kubernetes node, re-imaging or reconfiguring it, and joining it to Slurm (or vice versa) typically requires 5 to 15 minutes. This latency limits the granularity of sharing: the system can adapt to diurnal patterns or weekly cycles but cannot use Slurm capacity to absorb second-by-second inference bursts.

Emerging unified platforms like Ray attempt to eliminate the dichotomy entirely by providing a single runtime for both training and serving. Ray implements its own distributed scheduler that handles actor placement, task scheduling, and resource management natively. This eliminates the impedance mismatch between training (stateful actors) and serving (stateless replicas), allowing a single Python script to define a training loop that seamlessly transitions into a deployment pipeline. The trade-off is maturity: while Ray excels at orchestration flexibility, its scheduling logic lacks the decades of hardening found in Slurm and the massive ecosystem support of Kubernetes, effectively requiring the platform team to assume more responsibility for reliability and isolation.

Threading these patterns together, consider the lifecycle of our 175B parameter model. The training phase runs on Slurm to maximize NVLink topology efficiency and gang scheduling reliability for the multi-month run. Once trained, the model artifacts are handed off to Kubernetes for serving, where rolling updates and autoscaling ensure high availability for end users. The bridge between them is a Kubernetes operator that monitors training progress; when validation loss stabilizes, it automatically triggers a quantization pipeline and deploys the resulting model to the inference cluster.

The operational impact of these choices is quantifiable, but the numbers depend on workload mix and local platform maturity. A pure Slurm-only environment minimizes batch-job overhead and can keep dedicated training clusters highly utilized, but it requires extra engineering to build production serving infrastructure. A Kubernetes-only environment reduces operational fragmentation by using standard cloud-native tools, but distributed training may need batch extensions to avoid scheduling fragmentation. The hybrid approach targets the efficient frontier: preserving strong batch-training placement guarantees while keeping a unified management surface for serving and pipeline workloads, at the cost of increased architectural complexity.

Systems Perspective 1.3: The convergence of HPC and cloud

The sharp distinction between HPC and cloud-native scheduling is rapidly blurring as both communities adopt features from the other. Kubernetes is evolving batch capabilities through the Volcano and Kueue projects, introducing concepts like queues, priorities, and gang scheduling that were previously unique to HPC. Conversely, Slurm is adopting cloud features like REST APIs, container runtime support, and elasticity. The industry is converging toward a unified fleet operating system: a system that offers the declarative API and fault tolerance of the cloud with the strict placement guarantees and performance of the supercomputer.

This convergence turns the paradigm choice into a workload-specific design decision: teams must decide which scheduling guarantees matter before layering on topology constraints.

Checkpoint 1.1: Scheduling paradigm trade-offs

Before the chapter turns to topology-aware scheduling, review the core trade-offs:

• Gang scheduling: explain why gang scheduling is essential for distributed training but not for inference workloads, including the property of synchronous data parallelism that creates the all-or-nothing requirement.
• Slurm vs. Kubernetes: identify the CAP theorem trade-off Slurm makes differently from Kubernetes, and describe how this difference appears during a network partition between the scheduler and a subset of nodes.
• MIG partitioning: determine when MIG partitioning improves cluster utilization through concurrent inference workloads on fewer GPUs and when it reduces utilization for training jobs that need full GPU memory.
• Backfill incentives: explain why backfill scheduling depends on accurate job runtime estimates and why users have game-theoretic incentives to provide inaccurate estimates.

The orchestration paradigms treat GPUs as interchangeable units within a partition, but physical location matters as much as count: the same 64 GPUs run far slower scattered across racks than packed into one. Topology-aware scheduling exploits that structure.

Self-Check: Question

1. What is the fundamental difference in control philosophy between Slurm and Kubernetes when managing distributed ML training workloads?

   1. Slurm relies on continuous reconciliation loops against desired state stored in etcd, whereas Kubernetes executes imperative one-time allocation scripts via a centralized daemon.
   2. Slurm uses peer-to-peer gossip protocols to schedule tasks across nodes, whereas Kubernetes enforces hierarchical master-slave token ring passing.
   3. Slurm employs an imperative allocation model where users request explicit physical resources guaranteed at launch, whereas Kubernetes uses a declarative model where controllers continuously reconcile observed cluster state toward declared desired state.
   4. Slurm schedules containerized microservices with dynamic horizontal auto-scaling, whereas Kubernetes only supports uncontainerized bare-metal MPI processes.

2. How does NVIDIA Multi-Instance GPU (MIG) partitioning on A100/H100 accelerators differ from software-based GPU time-slicing in multi-tenant serving environments?

   1. MIG uses OS-level software schedulers to alternate kernel execution times, but shares the same L2 cache and memory controllers dynamically.
   2. MIG compiles user PyTorch models into specialized FPGA bitstreams, preventing simultaneous execution of untrusted kernels.
   3. MIG allows dynamic resizing of GPU partitions on a millisecond basis without draining running containers or resetting the device.
   4. MIG partitions the physical accelerator into hardware-isolated instances with dedicated streaming multiprocessors (SMs), memory controllers, and L2 cache slices, eliminating noisy-neighbor interference at the cost of static partition reconfiguration.

3. In Slurm’s fair-share priority formula (\(\text{Priority}_{\text{effective}} = \text{Priority}_{\text{base}} \times \frac{F_{\text{target}}}{F_{\text{actual}} + \epsilon}\)), what role does the usage decay half-life play, and what is the trade-off between setting a short versus a long half-life?

4. Why do high-performance distributed training jobs in Kubernetes typically enable hostNetwork: true and leverage the NVIDIA Network Operator instead of relying on default Container Network Interface (CNI) overlay networks?

5. True or False: Kueue replaces the native Kubernetes kube-scheduler binary entirely to implement custom gang scheduling algorithms, whereas Volcano acts purely as an admission controller layered on top of the default scheduler.

6. Order the following steps in the lifecycle of a distributed training job submitted to a Kubernetes cluster configured with Kubeflow Training Operator and Volcano gang scheduling:

1. The PodGroup is created and Volcano verifies that the cluster can simultaneously satisfy the minimum worker requirement (\(N_{\text{min}}\)).
2. The developer submits a PyTorchJob custom resource manifest defining worker replicas, GPU limits, and rendezvous parameters.
3. Volcano binds all worker pods to specific healthy nodes simultaneously, and kubelets start the training containers with GPUDirect RDMA.
4. Workers establish distributed communication via TorchElastic rendezvous (setting MASTER_ADDR and WORLD_SIZE) and begin synchronized training steps.
5. The Kubeflow Training Operator reconciles the PyTorchJob, creating the individual worker pods and a corresponding Volcano PodGroup.

See Answers →

Topology-Aware Scheduling

Randomly assigning 64 GPUs across a massive data center cripples a synchronous training job, causing it to run 15 to 30 percent slower than if those same 64 GPUs were packed into a single rack. The scheduling algorithms discussed so far treat GPUs as interchangeable units, but topology-aware scheduling recognizes that physical proximity in the network fabric is as critical as raw compute.

The topology hierarchy

Modern GPU clusters exhibit a multi-level communication hierarchy, where bandwidth decreases and latency increases at each level. This hierarchy is not an implementation detail that the scheduler can ignore; it is a physical constraint that directly determines training throughput for communication-intensive workloads. Understanding the hierarchy is essential for making placement decisions that translate allocated resources into useful work.

Bandwidth cliffs from NVLink to spine across the fabric.

Within a single node, GPUs communicate via NVLink, providing 600 GB/s aggregate bidirectional bandwidth (300 GB/s per direction) per GPU on A100 systems and 900 GB/s (450 GB/s per direction) on H100 systems. This high-bandwidth, low-latency interconnect enables efficient tensor parallelism, where matrix operations are split across GPUs that must exchange intermediate results (activations and gradients) at every transformer layer. The NVLink bandwidth is sufficient to overlap communication with computation for most model architectures, meaning tensor parallel operations can proceed at near-ideal throughput when confined to a single node.

Between nodes within the same rack or Leaf Switch Domain, communication traverses InfiniBand or RoCE at 400 Gb/s (NDR InfiniBand), roughly 50 GB/s per link. This represents an order of magnitude reduction from NVLink bandwidth. Data parallelism, which requires AllReduce of gradients after each training step, operates efficiently at this level because bucketed gradient synchronization can overlap with the remaining backward computation. The latency penalty is modest (microseconds vs. nanoseconds), and the aggregate bandwidth is sufficient for gradient tensors that are typically smaller than the activation tensors exchanged in tensor parallelism.

Between racks connected through Spine Switches, communication crosses additional switch hops, introducing both higher latency and potentially reduced effective bandwidth due to network congestion. The fat-tree topologies analyzed in Fat-tree (Clos) networks provide full bisection bandwidth in theory, but in practice, congestion from concurrent flows, routing inefficiencies, and switch buffer limitations reduce effective cross-rack bandwidth by 10 to 30 percent compared to intra-rack communication. Cross-rack communication also adds tail latency variability: while median latency may be only slightly higher, P99 latency can spike during congestion events, creating straggler effects in synchronous training where the slowest communicator determines the pace for all workers.

This hierarchy means that a 64-GPU job allocated entirely within a single rack (8 nodes, all under one leaf switch) will outperform the same job spread across 8 racks (1 node per rack, crossing spine switches for every AllReduce). The performance difference stems directly from the communication analysis in Collective Communication: ring AllReduce latency scales with the slowest link in the ring, and cross-rack hops introduce both higher latency and increased contention risk. For synchronous training, every iteration waits for the slowest worker to complete its AllReduce, so even occasional cross-rack congestion events create persistent throughput degradation.

The scheduler must therefore model the cluster not as a flat list of compute slots, but as a hierarchical graph with distinct bandwidth cliffs at each level (GPU, Node, Rack, and Pod) corresponding to NVLink, InfiniBand, and spine-switch boundaries. Each boundary crossing introduces an order-of-magnitude bandwidth reduction, and at the rack and pod levels, network oversubscription (often 2:1 or 4:1 at the spine switches) further restricts bisection bandwidth beyond what the raw link speed suggests. A topology-aware scheduler explicitly optimizes for these constraints, treating “locality” as a first-class resource. For a distributed training job requiring 1,024 GPUs, a fragmented placement that scatters workers across random racks can degrade end-to-end training throughput by 15 to 30 percent compared to a compact placement within a single pod.

Figure 3 contrasts optimal and scattered placement for a 4-GPU tensor parallel group, showing how the communication path determines whether GPUs use a single leaf switch or must traverse slower leaf-spine-leaf hops.

Figure 3: Topology-Unaware vs. Topology-Aware Placement: A 4-GPU job placed across two racks (left, topology-unaware) must traverse leaf-spine-leaf hops: 3 switch hops per AllReduce message at about 12 GB/s effective bandwidth. The same 4-GPU job packed within one rack (right, topology-aware) uses a single leaf switch: 1 hop at about 50 GB/s, roughly 4\(\times\) faster. A quantitative summary beneath the figure reports switch-hop counts and effective bandwidth.

Placement algorithms

The bandwidth gap shown in figure 3 is the key constraint: the roughly 4\(\times\) reduction from intra-rack leaf paths to cross-rack leaf-spine-leaf paths means that scattered placement forces tensor parallel groups to spend more time communicating than computing, a penalty that compounds at every transformer layer in the model. Topology-aware placement algorithms assign jobs to nodes that minimize communication cost, transforming the abstract bin packing problem into a graph optimization problem on the cluster’s physical topology. The simplest approach uses a Locality Cost, defined in equation 2, that penalizes allocations spanning multiple topology domains:

\[ \text{Cost}_{\text{locality}} = \sum_{i < j} w(d(g_i, g_j)) \tag{2}\]

where \(g_i\) and \(g_j\) are GPUs assigned to the job, \(d(g_i, g_j)\) is their topological distance (0 for same node, 1 for same rack, 2 for different racks), and \(w(\cdot)\) is a weighting function that increases with distance. The scheduler minimizes this score when selecting an allocation from the set of feasible placements. The weighting function encodes the performance impact of each topology level: \(w(0) = 0\) (same node, NVLink, no penalty), \(w(1) = 1\) (same rack, one switch hop), \(w(2) = 10\) (different racks, multiple switch hops). The tenfold weight difference between same-rack and cross-rack placements reflects the disproportionate performance impact of crossing spine switches.

More sophisticated algorithms distinguish between parallelism strategies, recognizing that different communication patterns have different bandwidth and latency requirements. Tensor parallelism requires the highest bandwidth and lowest latency because it exchanges activation tensors at every layer; it should be confined to NVLink domains within a single node. Pipeline parallelism tolerates moderate latency because it sends activations between stages less frequently (once per microbatch per stage boundary rather than at every layer); it can span nodes within a rack where InfiniBand provides sufficient bandwidth. Data parallelism, with its bulk gradient synchronization that can overlap with computation, operates efficiently across racks because the communication is less latency-sensitive and can use the available bandwidth effectively.

Algorithm 1 combines these pieces into a single decision: the all-or-nothing gang constraint of section 1.0.5, the locality cost of equation 2, and the parallelism-to-topology mapping just described. The scheduler enumerates the allocations that fit, scores each by communication cost, and takes the cheapest, leaving the job queued rather than stranding accelerators when none does.

\begin{algorithm} \caption{Topology-aware placement} \begin{algorithmic} \Require job requesting $N$ accelerators with tensor-, pipeline-, and data-parallel groups; cluster topology with per-level link costs $w(\cdot)$ \Ensure an all-or-nothing placement of least communication cost, or the job stays queued \State enumerate the feasible slots: sets of $N$ free accelerators that satisfy every resource dimension \Comment{gang: all or nothing} \If{no feasible slot exists} \State leave the job queued without holding any accelerators \Comment{prevents hold-and-wait deadlock} \EndIf \For{each feasible slot} \State map tensor-parallel ranks within an NVLink domain, pipeline stages within a rack, data-parallel replicas across racks \State score the slot by $\text{Cost}_{\text{locality}} = \sum_{i < j} w(d(g_i, g_j))$ \EndFor \State \Return the feasible slot of least $\text{Cost}_{\text{locality}}$ \Comment{topology-optimal, atomic} \end{algorithmic} \end{algorithm}

Scattered placement runs AllReduce 4.8\(\times\) slower than a topology-aware slot, a gap figure 4 breaks out across four placement strategies from random to topology-optimal.

Figure 4: Topology Placement Impact: AllReduce latency across four placement strategies (Random, Rack-Aware, Rail-Optimized, and Topology-Optimal) for a 1 GB gradient sync on 64 GPUs. Topology-aware scheduling reduces sync latency from 120 ms to 25 ms, yielding a 4.8\(\times\) speedup.

A Hierarchical Placement algorithm assigns resources in layers that match this parallelism hierarchy. First, it assigns tensor parallel groups to individual nodes, ensuring that all GPUs in a tensor parallel group share NVLink connectivity. Second, it groups those nodes into pipeline stages within racks, placing consecutive pipeline stages on nodes connected through the same leaf switch. Third, it distributes data parallel replicas across racks, ensuring that each data parallel group has access to the cluster’s full bisection bandwidth for AllReduce operations. This three-level placement mirrors the three-level topology and the three-level parallelism strategy, creating a natural alignment between logical and physical structure.

Napkin Math 1.2: The topology placement impact

The physical location of GPUs on the network switch fabric dramatically impacts collective performance. Consider an AllReduce operation across 64 GPUs:

• Random placement: Spread across many racks \(\rightarrow\) Traffic traverses core switches \(\rightarrow\) 120 ms latency.
• Rack-aware: Confined to a single rack (Top-of-Rack switch) \(\rightarrow\) 85 ms.
• Rail-optimized: Aligned to specialized “rails” (dedicated standard IB switches) \(\rightarrow\) 40 ms.
• Topology-optimal: All GPUs on the same leaf switch \(\rightarrow\) 25 ms. Proper scheduling yields a 4.8× speedup purely by respecting the network topology, with zero changes to the model code.

The computational cost of topology-aware placement is nontrivial. Evaluating all possible placements for a 256-GPU job across a 10,000-GPU cluster is combinatorially intractable. Production schedulers use heuristic approaches: greedy algorithms that build placements bottom-up (first fill NVLink domains, then fill racks, then spread across racks), tree-search algorithms with pruning that explore the most promising placements first, or scoring-based approaches that evaluate a sample of feasible placements and select the best. The scheduling latency introduced by topology-aware algorithms (milliseconds to seconds, depending on cluster size and algorithm complexity) is negligible compared to job runtimes of hours to weeks, making the throughput improvement worth the scheduling overhead.

Napkin Math 1.3: Placement impact on 3D-parallel training

Scenario: Consider a 256-GPU training job using 3D parallelism: 8-way tensor parallel, 4-way pipeline parallel, 8-way data parallel.

Good Placement (topology-aware):

• Tensor parallel groups: 32 groups of 8 GPUs, each confined to one 8-GPU node (NVLink: 900 GB/s)
• Pipeline stages: 4 consecutive nodes within the same rack (1 switch hop, InfiniBand: 50 GB/s)
• Data parallel replicas: 8 racks (2 switch hops for cross-rack AllReduce)

Bad Placement (topology-unaware):

• Tensor parallel groups split across 2 nodes (InfiniBand instead of NVLink)
• Pipeline stages scattered across racks (2 switch hops instead of 1)
• Data parallel replicas randomly distributed

Math: The tensor parallel communication alone illustrates the impact. With NVLink delivering 450 GB/s per direction, a 1 GB activation transfer takes approximately 2.2 ms. Over InfiniBand at 50 GB/s, the same transfer takes approximately 20 ms, a 9× slowdown.

Systems insight: The 4.8× AllReduce-latency gap in figure 4 is a communication-only measurement; it does not become a 4.8× end-to-end penalty because synchronous training overlaps much of that communication with the next iteration’s compute. Since tensor parallelism communicates at every transformer layer (typically 80+ layers for large models), the residual exposed communication compounds to 15 to 30 percent total training throughput degradation, the figure to carry forward for 3D-parallel jobs.

Rail-optimized scheduling

Large GPU clusters increasingly use Rail-Optimized Topologies where each GPU in a node connects to a different network switch, creating parallel “rails” through the network fabric. This design, analyzed in Rail-optimized topology, provides balanced bandwidth across all GPUs by distributing traffic across independent switch hierarchies. However, it also creates a scheduling constraint: AllReduce operations are most efficient when communicating GPUs are on the same rail (connected to the same switch chain), because same-rail communication avoids cross-rail traffic that could create congestion at shared switch ports.

Rail-aware scheduling assigns data parallel replicas such that corresponding GPUs across nodes share the same rail. For an 8-GPU-per-node cluster with 8 rails, GPU 0 on every node connects to rail 0, GPU 1 to rail 1, and so on. The scheduler places data parallel rank \(r\) on GPU \(r \bmod 8\) across selected nodes, ensuring that AllReduce for each data parallel group traverses only a single rail switch hierarchy rather than crossing between rails. This alignment means that AllReduce traffic is confined to independent switch trees, eliminating contention between data parallel groups and providing each group with the full bandwidth of its dedicated rail.

The interaction between rail-aware scheduling and topology-aware placement creates a multi-dimensional optimization problem. The scheduler must simultaneously optimize for intra-node NVLink locality (tensor parallelism requires GPUs on the same node), intra-rack switch locality (pipeline parallelism benefits from minimal switch hops between stages), and rail alignment (data parallelism benefits from same-rail communication). These three objectives are partially conflicting: constraining data parallel replicas to specific GPU positions within each node (for rail alignment) reduces the scheduler’s flexibility to choose nodes based on rack locality, and requiring full-node allocations for NVLink locality prevents sharing nodes between jobs that could improve packing efficiency.

Production schedulers resolve these conflicts using weighted scoring functions that prioritize constraints based on workload characteristics. For large training jobs using 3D parallelism, tensor parallel locality receives the highest weight (because NVLink-to-InfiniBand performance degradation is the largest), followed by rail alignment (because it eliminates congestion for the most frequent communication pattern), followed by rack locality (because the performance benefit is smaller and can be partially compensated by compute-communication overlap). For pure data parallel jobs, rail alignment receives the highest weight. For single-node jobs, topology constraints are irrelevant and the scheduler optimizes purely for packing efficiency.

Topology discovery and dynamic adaptation

Static configuration files like Slurm’s topology.conf or Kubernetes node labels provide a baseline map of the cluster, but they represent the intended state, not the effective state. True Topology Discovery requires runtime introspection to validate that the physical hardware matches the architectural diagram. Libraries like NCCL bypass static configurations entirely, parsing the Linux virtual filesystem at /sys/class/infiniband to map the PCIe bus hierarchy, enumerate NVLink connections, and verify network interface card (NIC) proximity. This creates a ground-truth graph where edges represent verified, operational links rather than theoretical cables.

Cluster topology is an operating state, not a fixed schematic. Hardware failures, maintenance windows, and thermal throttling constantly reshape the effective topology available to the scheduler. A single spine switch failure in a Clos network does not sever connectivity entirely, but it drastically reduces the bisection bandwidth available to jobs spanning the affected racks. If a spine switch serving our 175B parameter model’s data-parallel AllReduce group fails, cross-rack bandwidth might drop by 50 percent. The scheduler faces a critical optimization decision: continue training at reduced throughput, effectively wasting 25 percent of the expensive GPU cycles waiting for straggling gradients, or initiate a topology-aware migration.

Migration is not free; it requires a coordinated checkpoint, a job kill, a rescheduling event on healthy nodes, and a warmup phase. This process typically consumes 5 to 15 minutes of idle time. However, for a multi-week training run, the math often favors migration. If the network degradation creates a 20 percent throughput penalty, the break-even point for a 10-minute migration is less than an hour of training. Sophisticated orchestrators continuously monitor effective bandwidth metrics reported by the collective communication library. When the delta between the assigned topology score and the realized throughput exceeds a threshold, the scheduler triggers a drain-and-migrate workflow, treating network degradation with the same severity as a GPU hardware fault.

Example 1.2: The limping switch

Scenario: A multi-node foundation model training job exhibits a 30 percent throughput drop while static topology maps show full bandwidth and GPU hardware health checks report green.

Diagnosis: A single leaf switch transceiver experienced high CRC error rates and packet retransmissions. Binary up/down probes passed, but degraded bisection bandwidth stalled AllReduce communication across the entire training job.

Systems lesson: Binary health checks are insufficient for distributed training fabrics. Schedulers require performance-aware topology monitoring that tracks retransmissions and collective bandwidth, dynamically quarantining degraded network links.

The general scheduling idea is quarantine: when runtime evidence says a node, rack, or link is degraded, the scheduler should stop placing sensitive work there even if the nominal resource count still looks available. Kubernetes implements this pattern through Taints and Tolerations,¹² while Slurm clusters commonly use drain states, node features, or partition rules. The complementary problem is how jobs adapt when available resources change dynamically. Rather than treating resource allocation as fixed for a job’s lifetime, elastic training allows jobs to grow, shrink, and recover without full restarts.

¹² Taints and Tolerations: Kubernetes node-repulsion primitives used in ML fleets to isolate specialized hardware. By tainting A100 or H100 nodes, the orchestrator prevents general-purpose web microservices from consuming expensive GPU capacity, ensuring those nodes remain reserved exclusively for training jobs that declare explicit tolerations.

Self-Check: Question

1. When scheduling a 3D-parallel large language model across a multi-rack GPU cluster, how should Tensor Parallelism (TP), Pipeline Parallelism (PP), and Data Parallelism (DP) be mapped across the topology hierarchy to maximize throughput?

   1. Map DP strictly within intra-node NVLink domains, TP across inter-rack spine switches, and PP across rail-aligned network switches.
   2. Map PP strictly within intra-node NVLink domains, DP across Top-of-Rack switches, and TP across multi-hop spine switches.
   3. Map TP strictly within intra-node NVLink domains (highest bandwidth), PP within the same rack or adjacent switches (point-to-point activations), and DP across nodes with rail-aligned network switches.
   4. Map TP, PP, and DP uniformly at random across all available nodes to balance switch thermal loads across the datacenter fabric.

2. In a multi-node cluster where binary up/down health probes report all switches healthy but an AllReduce collective experiences a \(30\%\) throughput degradation due to high CRC error rates on a single leaf switch, what is the optimal orchestration response?

   1. Continue running the job indefinitely because a 10-minute checkpoint-and-reschedule migration would cost more compute than a 30% slowdown over a multi-week run.
   2. Rely on the GPU driver to disable NVLink and route all tensor-parallel traffic through host memory over PCIe.
   3. Trigger a performance-aware topology quarantine via taints or drain states, checkpoint the running job, and migrate it to healthy nodes where the break-even time is under an hour.
   4. Terminate the job and permanently de-allocate all 10,000 GPUs in the datacenter until all network transceivers are manually replaced.

3. In an 8-GPU-per-node cluster with 8 independent network fabric rails, how does rail-aware scheduling place data-parallel ranks, and why does this prevent network switch congestion?

4. Why are static topology configuration files (such as Slurm’s topology.conf or static Kubernetes node labels) insufficient on their own for optimal collective placement in large-scale ML clusters?

5. Order the steps executed by a topology-aware scheduler when placing a multi-node distributed training job across a multi-tier datacenter fabric:

1. Filter out candidate nodes that have active taints, hardware ECC alerts, or degraded network link flags.
2. Allocate contiguous GPU ranks within each node to satisfy intra-node NVLink requirements for Tensor Parallel groups.
3. Group nodes into the smallest common network domain (such as a single leaf switch or rack) to minimize switch hops for Pipeline Parallel stages.
4. Receive job submission specifying total GPU count, node granularity, and parallelism dimensions (\(TP, PP, DP\)).
5. Assign Data Parallel ranks to matching GPU rail indices across nodes to ensure non-interfering AllReduce traffic.

6. In cloud orchestration environments such as AWS and GCP, a user specifies a ____ to ensure that allocated virtual machine instances are placed within a single high-bandwidth, low-latency network spine domain rather than scattered across the datacenter.

See Answers →

Elastic Scheduling

When cluster demand fluctuates, rigid allocation creates two categories of waste. During off-peak hours, a job running on 256 GPUs cannot absorb idle resources to speed up training. During peak demand, the same job cannot release excess resources to accommodate higher-priority work without being fully preempted. The scheduler’s only options are “let it keep all its resources” or “kill it entirely,” with nothing in between.

Elastic scheduling eliminates this binary choice. The recovery mechanics of elastic training developed in Elastic Recovery show how a distributed job can adjust its worker count, recalibrate batch size and learning rate, and reconstruct its communication group after a failure. From the scheduler’s perspective, the same capability solves a resource-allocation problem under fluctuating demand. A training job might start with 128 GPUs (the minimum needed for reasonable throughput), scale up to 256 when resources become available, and scale back to 192 when higher-priority work arrives, all while maintaining continuous training progress. This flexibility transforms the scheduler’s action space from a binary {continue, preempt} to a continuous spectrum of resource allocation levels.

The scheduler-framework contract

Elastic scheduling requires a well-defined contract between the scheduler and the training framework. The scheduler guarantees that resource changes are communicated through defined signals, with sufficient advance notice for graceful adaptation. The framework guarantees that it can handle resource changes without corrupting model state, using the batch size adjustment, learning rate recalibration, and communication group reconstruction mechanisms developed in Recovery adaptation mechanisms.

From the scheduler’s side, the contract has three parameters:

• Elastic Range: \([N_{\text{min}}, N_{\text{max}}]\) specifies the worker counts between which the job can operate.
• Transition Cost: The rendezvous cycle, typically 10 to 60 seconds, determines how frequently the scheduler can resize without the overhead negating the throughput gain.
• Scaling Efficiency Curve: The throughput gained from each added worker determines whether assigning idle GPUs to this job is more productive than holding them for a pending gang-scheduled job.

The contract gives the scheduler enough information to resize a job without treating elasticity as free capacity.

The scheduler must also account for the constraint that elastic scaling changes the effective batch size (\(B_{\text{effective}} = B_{\text{per-worker}} \times N\)). The two strategies for handling this (constant global batch size vs. adaptive learning rate) impose different limits on how aggressively the scheduler can resize: constant global batch size preserves convergence but may violate per-GPU memory limits at low worker counts, while adaptive learning rate simplifies worker management but introduces convergence risk during rapid scaling.

Frameworks like TorchElastic expose the worker-membership range in their launch configuration, as listing 2 illustrates.

Listing 2: TorchElastic Launch Configuration: The --nnodes=4:16 range configures the elastic worker group to admit 4 to 16 nodes, or 32 to 128 workers. A separate cluster scheduler or operator must acquire, launch, and release nodes within that range.

# Launch elastic training with min 32 to max 128 workers
torchrun \
  --nnodes=4:16 \
  --nproc_per_node=8 \
  --rdzv_backend=c10d \
  --rdzv_endpoint=head-node:29500 \
  --max_restarts=3 \
  train.py

The --nnodes=4:16 specification configures TorchElastic’s rendezvous and worker-membership range as 4 to 16 nodes (32 to 128 workers with 8 processes per node). It does not acquire or resize cluster resources. A separate cluster scheduler or operator must place the nodes and start or stop elastic agents as capacity changes.

Systems Perspective 1.4: Elastic vs. rigid scheduling

Consider a congested cluster where a 128-GPU job waits 8 hours for a full gang allocation.

Rigid Scheduling: Wait 8 hours, then train for 24 hours at full throughput. Total wall-clock: 32 hours.

Elastic Scheduling (min 32, max 128):

• Start immediately with 32 GPUs (throughput = 25 percent of full)
• After 2 hours, scale to 64 (throughput = 50 percent)
• After 4 hours, scale to 128 (throughput = 100 percent)
• Training progresses during the entire wait period

Approximate work completed during the 8-hour “queue wait”:

• Hours 0 to 2: \(25\%\) throughput \(\times\) 2 hours = 0.5 hours equivalent
• Hours 2 to 4: \(50\%\) throughput \(\times\) 2 hours = 1.0 hours equivalent
• Hours 4 to 8: \(100\%\) throughput \(\times\) 4 hours = 4.0 hours equivalent
• Total: 5.5 hours of equivalent full-scale work

The elastic job completes in approximately 26.5 hours (8 hours of scaled-up training plus 18.5 hours at full scale), saving 5.5 hours compared to rigid scheduling. This 17 percent improvement comes entirely from using resources productively during what would otherwise be idle queue time.

The scheduler cannot treat every job as elastic. The most significant constraint is incompatibility with model parallelism. Elastic scaling is fundamentally a data-parallel operation: the scheduler adds or removes replicas of the model, changing the global batch size but keeping the model architecture constant. For a 175B-parameter job that relies on 8-way tensor parallelism within each node, elasticity operates strictly at the node level. The job can scale between 128 nodes (1,024 GPUs) and 32 nodes (256 GPUs) in whole-node steps, changing the data-parallel degree. It cannot elastically remove a single GPU from a tensor-parallel group; doing so would require re-sharding the model weights, equivalent to a full checkpoint-and-restart cycle. Pipeline parallelism permits limited elasticity in principle, but adding or removing stages forces a recalculation of the pipeline schedule that often negates the benefit.

The transition cost also constrains the scheduler’s resize frequency. Each scaling event incurs 10 to 60 seconds of reduced throughput while the framework reconstructs its communication group. The scheduler must amortize this cost over the subsequent run duration, so small resize increments are only worthwhile if the job will run at the new scale for significantly longer than the transition window; section 1.3.3 works through the thrashing regime this rule rules out.

Scheduler integration

Elastic training’s value depends critically on scheduler integration. Without scheduler awareness, elastic training is merely a fault tolerance mechanism (replacing failed workers). With scheduler integration, it becomes a scheduling optimization that fundamentally changes the economics of cluster utilization. The scheduler must understand that elastic jobs can operate within a range of resource allocations, specified as \([N_{\text{min}}, N_{\text{max}}]\) workers, enabling three key scheduling optimizations.

The most immediately valuable benefit is faster job start. An elastic job requesting 32 to 128 GPUs can start as soon as 32 GPUs are available, rather than waiting in the gang scheduling queue for 128 contiguous GPUs. In congested clusters where large-job queue times can be hours to days, starting immediately at reduced scale often completes the job sooner than waiting for full-scale allocation, a trade-off notebook 1.4 works through quantitatively. The scheduler can compute it dynamically, choosing the start time and initial scale that minimizes expected total time (wait time plus execution time).

Opportunistic scaling enables the scheduler to use elastic jobs as flexible capacity absorbers. When resources become available (other jobs complete, spot instances are provisioned, preempted jobs release resources), the scheduler can expand running elastic jobs to improve their throughput. When resources are needed (higher-priority job arrives, reserved capacity is reclaimed), the scheduler can shrink elastic jobs without preempting them entirely. This bidirectional flexibility converts elastic jobs into a buffer that absorbs utilization variance, smoothing the gap between allocated and productive capacity.

Graceful preemption transforms the scheduler’s response to resource pressure. Instead of killing a training job outright to reclaim resources, the scheduler can request a scale-down, reducing the job’s resource allocation to release capacity for higher-priority work. The job continues with fewer workers, maintaining progress at reduced throughput rather than losing all progress since the last checkpoint and incurring restart overhead. This provides the scheduler with a continuous spectrum between “full resources” and “fully preempted,” rather than the binary choice of traditional scheduling. Graceful preemption composes well with checkpoint-based preemption: if the scheduler needs to reclaim all resources, it first scales the job down to its minimum size, then initiates a checkpoint-and-preempt sequence, minimizing the amount of lost work.

The trade-off for all of these benefits is complexity. Elastic training requires framework support (not all training frameworks support it), introduces potential convergence concerns from batch size changes (requiring validation that elastic scaling does not degrade model quality), and complicates performance modeling (a job’s throughput is no longer constant, making capacity planning harder). Not all workloads benefit equally from elastic scaling: jobs with high communication-to-computation ratios see diminishing returns from additional workers (because communication overhead grows with worker count) and thus benefit less from scale-up, while compute-bound jobs with low communication overhead scale more linearly and benefit more from opportunistic scaling. The scheduler must model these scaling characteristics per-job to make optimal elastic scaling decisions.

Elastic scaling policies

Adding resources to a running job does not always accelerate time-to-convergence. The naive assumption that throughput scales linearly with worker count collapses under the weight of communication overhead, requiring the scheduler to enforce elastic scaling policies based on empirical efficiency curves rather than resource availability alone. A job’s scaling efficiency \(\eta_{\text{scaling}}(k)\), defined as the ratio of observed throughput at \(k\) workers to the ideal linear speedup, typically exhibits three distinct phases: a linear phase where compute dominates communication, a sublinear phase where gradient synchronization latency begins to mask compute, and a saturation phase where the AllReduce ring becomes the bottleneck. A scheduler that blindly assigns available GPUs to a job in the saturation phase wastes cluster capacity that could have cleared the queue; conversely, running a job below its Minimum Viable Scale forces it to hold onto memory and network leases for disproportionately low progress.

Elastic scaling’s benefit flattens into saturation.

The decision to scale is therefore a function of marginal utility, not vacancy alone. For our 175B parameter model, the efficiency curve is derived from profiling: it scales linearly from 64 to 512 GPUs, becomes sublinear between 512 and 1,024, and hits diminishing returns beyond 1,024. Consequently, the scheduler enforces an elastic range of \([256, 1024]\). If the cluster has free resources but the job is already at 1,024 GPUs, the scheduler withholds the extra nodes, predicting that the 5 percent marginal throughput gain does not justify the re-sharding overhead or the opportunity cost of starving a pending job. Similarly, if only 128 GPUs are available, the scheduler may choose to queue the job rather than run it, as the training intensity per GPU would be too low to hide the communication latency.

This logic extends to the temporal dimension, where the scheduler must weigh the immediate progress of a smaller allocation against the deferred gratification of waiting for a larger one. If a 512-GPU job can start immediately with 128 GPUs, or wait two hours for the full request, the optimal choice depends on the integration of throughput over time. The preemption overhead (the cost of stopping, checkpointing, and restarting to resize) must be amortized over the subsequent run duration. A policy of “opportunistic scaling” that resizes a job every time a single node becomes free often yields net negative progress due to the constant thrashing of the distributed process group.

Napkin Math 1.4: The elastic scaling decision

Consider a large language model training job with a target batch size that requires 512 A100s for peak efficiency. Three scheduling strategies are evaluated over a window of 24 hours.

Given: Throughput at 128 GPUs is 1 epoch/hour (baseline). Throughput at 256 GPUs is 1.8 epochs/hour (90 percent scaling efficiency). Throughput at 512 GPUs is 3.2 epochs/hour (80 percent scaling efficiency). Re-scaling cost is 10 minutes (0.17 hours) to checkpoint, re-shard, and restart.

Strategy 1 – immediate start (no scaling): The job starts immediately with 128 GPUs and runs for the full 24 hours. Total work: \(24 \times 1.0 = 24\) epochs.

Strategy 2 – wait for capacity (gang scheduling): The job waits in the queue for 4 hours until 512 GPUs become available, then runs for 20 hours. Total work: \(20 \times 3.2 = 64\) epochs.

Strategy 3 – elastic scaling (step-up): The job starts immediately with 128 GPUs. After 4 hours, 256 GPUs become available. The job pauses, scales up, and resumes. Phase 1: \(4 \times 1.0 = 4\) epochs. Phase 2: \((24 - 4 - 0.17) \times 1.8 \approx 35.7\) epochs. Total work: 4 + 35.7 = 39.7 epochs.

The elastic strategy completes nearly 1.7× the epoch-equivalent work of the immediate small-scale run, but still trails gang scheduling by about 38 percent. That gap shrinks when queue waits grow or peak capacity stays scarce; re-scaling overhead is only 10 minutes, and if that tax exceeded 2 hours (due to slow checkpoint transfer or compilation), Strategy 3 would fall further behind Strategy 2.

The placement and elasticity mechanisms discussed so far address individual job efficiency, optimizing how each job uses its allocated resources. Fleet economics shifts the objective to the collective level: an allocation policy must maximize useful work across all jobs when every placement, preemption, and reservation creates opportunity cost.

Self-Check: Question

1. When an elastic scheduler resizes a distributed training job from 64 GPUs to 128 GPUs, what trade-off exists between maintaining a constant global batch size versus scaling batch size proportionally with worker count?

   1. Constant global batch size preserves mathematical convergence dynamics but halves per-GPU micro-batch size (potentially lowering hardware compute efficiency or hitting memory limits when shrinking), whereas proportional scaling maintains compute efficiency per GPU but alters optimization trajectory and requires learning rate recalibration.
   2. Constant global batch size requires restarting the model weights from epoch zero, whereas proportional batch size allows continuous gradient descent without learning rate adjustment.
   3. Constant global batch size disables NVLink communication across workers, whereas proportional batch size enables ring AllReduce.
   4. Constant global batch size doubles the learning rate automatically via PyTorch hooks, whereas proportional batch size keeps the learning rate fixed to prevent divergence.

2. An elastic job running on 64 GPUs at 100 steps/minute can scale to 128 GPUs, reaching 180 steps/minute. If resizing triggers a dynamic rendezvous and checkpoint-sync transition pause of 60 seconds (0 steps during pause), what is the minimum duration the job must remain at 128 GPUs to achieve a net gain in total steps completed?

   1. At least 15 seconds.
   2. At least 30 seconds.
   3. At least 45 seconds.
   4. At least 75 seconds.

3. What three parameters define the operational contract between an elastic scheduler and an elastic training framework (such as TorchElastic), and why is each necessary?

4. True or False: When high-priority workloads surge on a shared cluster, an elastic scheduler can reclaim capacity from lower-priority distributed training jobs by downsizing them toward their \(N_{\text{min}}\) worker limit rather than terminating and requeuing them, thereby avoiding total progress loss.

5. Order the sequence of events that occurs when a worker node fails during an elastic training run managed by TorchElastic:

1. The rendezvous backend (e.g., c10d / etcd) detects the dead worker heartbeat and signals a rendezvous state change to remaining workers.
2. The scheduler detects the unrecoverable node, provisions a replacement node (or authorizes running with reduced workers if within \([N_{\text{min}}, N_{\text{max}}]\)), and joins the new worker.
3. Active workers pause gradient computation at the step boundary and enter the rendezvous barrier.
4. All surviving and new worker ranks establish a new communication world, assign new ranks (\(0 \dots N-1\)), re-initialize process groups, load the latest checkpoint, and resume training.
5. A hardware fault causes an unhandled CUDA kernel crash on worker rank 3, stopping its heartbeats.

See Answers →

Cost Optimization

The utilization-to-dollars relationship established in The Scheduling Problem now becomes the orchestrator’s objective function. A 10,000-GPU cluster represents a capital investment of roughly $300 million and consumes millions more in monthly power and cooling. If poor scheduling leaves 20 percent of those GPUs idling in fragmentation gaps or waiting for data, roughly $60 million in deployed capital sits unused, and at cloud-equivalent rates the same idleness burns approximately $35 million per year in operating cost. Cost optimization forces the fleet orchestrator to treat the cluster as a massive financial asset whose return on investment (ROI) must be maximized.

Figure 5 makes this waste concrete by plotting annual wasted cost as a function of cluster size and utilization level. At 10,000 GPUs with 50 percent utilization, the annual waste exceeds $87 million under the chapter’s $2/GPU-hour accounting scenario. Published measurements from production clusters show that underutilization is a real operational problem: Microsoft’s Philly cluster study (Jeon et al. 2019) measured mean GPU utilization of approximately 52 percent, and Li et al. (2023) found that 50 percent of jobs on NERSC’s Perlmutter supercomputer used less than 25 percent of allocated GPU memory. The 30 to 70 percent shaded band in the figure is an illustrative operating range for the cost model, not a universal empirical claim about all production clusters. Moving from 50 percent to 70 percent utilization on a 10,000-GPU cluster saves over $35 million annually in this scenario without purchasing a single additional GPU.

Figure 5: The Cost of Idle GPUs: Annual wasted cost as a function of cluster size for different GPU utilization levels, assuming $2/GPU-hour (cloud equivalent). The shaded region between 30 percent and 70 percent is an illustrative operating range for the scenario analysis. At 10,000 GPUs with 50 percent utilization, the model yields $87.6M in annual wasted capacity. The Microsoft Philly data point marks the published mean GPU utilization of 52 percent. Data sources: (Jeon et al. 2019; Li et al. 2023).

Jeon, Myeongjae, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, and Fan Yang. 2019. “Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads.” 2019 USENIX Annual Technical Conference (USENIX ATC 19), 947–60.

Li, Jie, George Michelogiannakis, Brandon Cook, Dulanya Cooray, and Yong Chen. 2023. “Analyzing Resource Utilization in an HPC System: A Case Study of NERSC’s Perlmutter.” Lect. Notes Comput. Sci., Lecture notes in computer science, vol. 13948: 297–316. https://doi.org/10.1007/978-3-031-32041-5_16.

The idle-capacity figure establishes the economic target; the remaining cost controls explain how the scheduler moves toward it. Interruptible capacity lowers the price of tolerant work, reservation and quota policy decide which jobs deserve scarce guaranteed capacity, and accountability mechanisms make idle GPUs visible to the teams that create demand.

Spot instances and preemptible VMs

Cloud providers offer Spot Instances¹³ (AWS) or Preemptible VMs (GCP) at 60 to 90 percent discounts compared to on-demand pricing, with the caveat that instances can be reclaimed with as little as 30 seconds to 2 minutes notice when the provider needs the capacity for on-demand customers. At $0.70/GPU-hour instead of $2/GPU-hour, spot instances transform the economics of large-scale training, potentially reducing the cost of a multi-week pretraining run from hundreds of thousands of dollars to under one hundred thousand.

¹³ Spot Instances: Named after commodity trading’s “spot price” (the current market price for immediate delivery), spot instances sell spare cloud capacity at 60 to 90 percent discounts with the provider retaining reclamation rights. The critical asymmetry for ML training is the interruption notice window: AWS provides 2 minutes, GCP provides 30 seconds. A 175B-parameter model requires 3 to 5 minutes to checkpoint, so GCP’s 30-second notice forces either continuous checkpointing overhead or acceptance of lost work on every interruption.

The viability of spot instances for ML training depends on two factors that determine whether the discount translates into actual savings: the cost of each interruption (checkpoint overhead plus restart time plus lost work since the last checkpoint) and the frequency of interruption (which varies by instance type, region, time of day, and overall cloud demand). Equation 3 gives the Effective Cost of spot training:

\[ C_{\text{effective}} = C_{\text{spot}} \times \frac{T_{\text{total}}}{T_{\text{productive}}} \tag{3}\]

where \(T_{\text{total}}\) includes productive training time plus checkpoint overhead plus restart overhead after interruptions, and \(T_{\text{productive}}\) is the time spent advancing training. If interruptions are rare (less than once per day) and checkpointing is efficient (less than 5 percent overhead), effective cost remains close to the spot price and the savings are nearly the full discount. If interruptions are frequent (every few hours) and restarts are expensive (large model, slow checkpoint loading, cold GPU cache), effective cost can approach or even exceed on-demand pricing, turning the apparent discount into a hidden premium.

The relationship between spot savings and fault tolerance infrastructure is circular in an important way. The fault tolerance mechanisms from Fault Tolerance (frequent asynchronous checkpointing, fast checkpoint loading, elastic recovery) directly enable spot instance usage by minimizing the cost of each interruption. Elastic training allows jobs to continue with reduced worker count when some spot instances are reclaimed, avoiding full restart entirely. Organizations that invested in fault tolerance for reliability reasons discover that the same infrastructure unlocks significant cost savings through spot instance utilization, providing a return on their fault tolerance investment that goes far beyond avoiding lost work from hardware failures.

The strategic insight is that fault tolerance infrastructure has a double return: it both protects against involuntary losses (hardware failure) and enables voluntary cost savings (spot instances). This double return often justifies fault tolerance investments that would be hard to justify on reliability grounds alone, because the cost savings from spot utilization are concrete and measurable while the cost avoidance from hardware failure protection is statistical and harder to quantify.

Spot instance strategies for ML training

To maximize spot instance utility while minimizing disruption, sophisticated schedulers employ diversification and predictive strategies that go beyond simple “retry on failure” logic. Availability zone diversification reduces the probability of simultaneous fleet-wide reclamation. Cloud providers typically manage spot pools independently per availability zone (AZ). If a training job requires 1,024 GPUs, allocating them as a single block in one zone exposes the job to a complete stop if that specific zone’s spot pool is reclaimed. Spreading the job across multiple AZs ensures that a reclamation event in one zone affects only a fraction of the fleet. The scheduler still must model correlation carefully: independent per-instance interruptions make large simultaneous losses unlikely, but pool-level or AZ-level reclamations can remove hundreds of GPUs at once and must be treated as elastic-shrink or failover events rather than ordinary isolated failures.

Instance type diversification exploits independent spot markets across GPU SKUs. A job that strictly requests one instance type competes in a single, crowded market. A job that accepts multiple equivalent instance types dramatically increases its scheduling probability. The scheduler should maintain a priority-ordered list of acceptable instance types, falling back to larger-memory instances when the primary type is unavailable. While this mix requires careful peer-to-peer bandwidth management, it effectively uses the “upgrade” to maintain progress at a blended cost still far below on-demand rates.

Spot capacity signals can guide placement but do not necessarily predict interruption frequency. For example, the AWS Spot Placement Score estimates the current likelihood that a capacity request will succeed in a region or availability zone; it explicitly does not predict interruption risk. A scheduler can use this signal to choose where to request capacity, but estimating interruptions per day requires separate historical measurements for the instance type and region. If the resulting workload-specific recovery model shows that interruption overhead erases the discount, the scheduler should migrate the job to on-demand capacity or a different region.

Checkpoint frequency optimization for spot instances differs from the standard reliability logic. The Young-Daly formula developed in The Young-Daly law: Optimal checkpointing optimizes for unpredicted hardware failures. Spot interruptions, however, come with a warning: 2 minutes on AWS, 30 seconds on GCP. A Two-Tier Checkpointing approach combines a periodic checkpoint at the Young-Daly interval with a best-effort checkpoint triggered by the termination notice. The notice-triggered checkpoint reduces lost work only if it can finish within the warning window. When a large model’s full checkpoint takes longer, the job must rely on an already-running asynchronous or incremental checkpoint and accept rollback to the last durable state.

The same need for contingency planning becomes acute when spot interruptions are correlated rather than isolated.

Systems Perspective 1.5: Correlated spot risk

Context: Spot capacity is cheap because it is reclaimable. That risk is not independent across instances: cloud regions, instance types, product launches, and deadline cycles can create correlated demand.

Failure mode: A fleet that concentrates all interruptible training in one region and one instance type can lose a large fraction of workers at once, even if each individual worker appears restartable.

Consequence: Checkpointing handles isolated interruptions; it does not solve fleet-wide preemption when every retry targets the same exhausted pool.

Systems insight: A robust scheduler must diversify across regions and instance types, monitor interruption risk, and maintain a “break glass” path to on-demand capacity when the spot market becomes toxic.

Capacity reservation strategies

Organizations that rely on cloud infrastructure (or that think about on-premise infrastructure in economic terms) balance three tiers of compute capacity, each offering a different trade-off between cost, availability, and commitment. Table 4 compares those tiers across cost, availability, interruption risk, and workload fit.

Table 4: Cloud Capacity Reservation Tiers: Capacity tiers trade cost and commitment against availability guarantees and interruption risk, giving the scheduler a price ladder for routing workloads by urgency and restart tolerance.

Capacity tier	Cost and commitment	Availability and interruption risk	Best-fit workloads
Reserved capacity	One- to three-year commitments, with 30–60% discounts vs. on-demand.	Guaranteed availability for the committed baseline.	Predictable continuous workloads: 24/7 inference, scheduled training, CI and regression testing, SLA-bound jobs.
On-demand capacity	Full price with no long-term commitment.	Immediate provisioning when queuing or interruption is unacceptable.	Urgent retraining, interactive debugging, and short-lived jobs that do not justify spot management.
Spot capacity	Discounted interruptible capacity.	Availability varies with cloud demand, and workers can be reclaimed.	Restart-tolerant jobs: hyperparameter sweeps, ablations, checkpointed pretraining, and exploratory research.

The optimal mix depends on workload characteristics and organizational priorities. An organization with 60 percent steady-state utilization and 40 percent burst demand might reserve capacity for 60 percent of peak need, use on-demand for latency-sensitive bursts that require immediate provisioning, and use spot for the remainder. The scheduling system must understand these capacity tiers and route workloads accordingly: inference workloads to reserved capacity for guaranteed availability, large training jobs to a mix of reserved (for the base allocation) and spot (for additional workers via elastic scaling), and exploratory work to spot exclusively where the cost savings are maximized and the interruption tolerance is highest.

Systems Perspective 1.6: Spot vs. on-demand training

Consider training a 70B parameter model requiring 512 GPUs for 14 days.

On-Demand: 512 GPUs \(\times\) 336 hours \(\times\) $2/GPU-hour = $344,064

Spot (65 percent discount, 5 percent checkpoint overhead, 14 interruptions/day with 30 minutes restart):

• Base cost: 512 GPUs \(\times\) 336 hours \(\times\) $0.70/GPU-hour = $120,422
• Checkpoint overhead: \(5\%\) \(\times\) 336 hours = 16.8 hours
• Restart overhead: 196 interruptions \(\times\) 0.5 hours = 98 hours
• Total time: 336 hours + 16.8 hours + 98 hours = 450.8 hours
• Total cost: 512 GPUs \(\times\) 450.8 hours \(\times\) $0.70/GPU-hour = $161,567

Savings: $182,497 (53.0 percent), at the cost of approximately 34.2 percent longer wall-clock time.

The economics are compelling when fault tolerance infrastructure is already in place. Without checkpointing, a single interruption forces restart from scratch, potentially wasting days of compute and negating all savings.

Scheduling for cost efficiency

Cost-aware scheduling extends the scheduler’s optimization objective beyond utilization and fairness to include financial efficiency. Traditional schedulers minimize a single objective (for example, average job completion time or maximum wait time), but cost-aware schedulers must navigate multi-objective trade-offs where time, money, and reliability are all decision variables. The scheduler must routinely choose among three decision classes:

• Wait vs. run now: Run a job on reserved A100s now, or wait 2 hours for spot H100s that would complete the job 3\(\times\) faster at half the cost.
• Preempt vs. pay more: Preempt a low-priority hyperparameter sweep to make room for a high-priority training run on reserved capacity, or place the training run on more expensive on-demand instances.
• Fallback vs. elastic shrink: When spot capacity is reclaimed, migrate the job to on-demand to maintain throughput at higher cost, or scale down elastically to maintain low cost at reduced throughput.

Each decision prices time, money, and reliability differently, so cost-aware scheduling is policy design rather than simple bin packing.

These decisions require the scheduler to model both the time-value of computation (the organizational cost of a 2-hour delay in researcher productivity, product launch schedules, or competitive position) and the financial cost of different resource allocations. The time-value varies enormously by workload type: delaying a production model retrain by 2 hours might violate a service-level agreement (service level agreement (SLA)) and cost thousands in penalty fees, while delaying an exploratory experiment by 2 hours costs nothing but researcher patience.

Production systems typically define Cost Policies per workload class that encode these trade-offs as configuration rather than requiring per-decision human judgment. Inference workloads are pinned to reserved capacity for guaranteed availability and predictable latency, since the cost of an inference outage (lost revenue, degraded user experience) far exceeds the premium of reserved pricing. Large training runs use a mix of reserved capacity (for the minimum viable allocation) and spot capacity (for elastic expansion), with automatic fallback to on-demand if spot is reclaimed and the job is within a configurable deadline. Exploratory workloads are restricted to spot to minimize cost, with the understanding that they may be interrupted and must tolerate variable completion times.

Technical optimization must be paired with financial accountability through ML Finops. In many organizations, GPU costs are pooled into a generic “infrastructure” bucket, creating a tragedy of the commons where teams lack visibility into their resource consumption and have no incentive to optimize. Effective governance requires granular tagging to implement Showback (reporting costs to teams) or Chargeback (billing teams’ budgets). Metrics must evolve from “GPU hours” to business-aligned units: “Cost per Model Version,” “Cost per Experiment,” and “Cost per 1M Predictions.” This shift transforms cost from an opaque infrastructure expense into a measurable input to product decisions, enabling teams to make informed trade-offs between model quality, training speed, and infrastructure cost. Cost visibility and attribution develops the platform machinery that makes these costs visible and attributable; the concern here is narrower, namely how that price signal reshapes the scheduling and quota behavior of the teams competing for the fleet.

The total cost of ownership

Optimizing for spot instance availability or scheduling density addresses only the visible tip of the infrastructure iceberg. The full cost of a machine learning fleet extends far beyond GPU-hours: in high-performance clusters the compute silicon typically accounts for only 40 to 60 percent of total system cost. The remainder is consumed by the supporting infrastructure required to keep that silicon fed and cool: high-bandwidth networking (InfiniBand switches, optics, and cabling), parallel file systems for checkpointing, and specialized power and cooling delivery. A single rack of H100 nodes can draw upwards of 40 kW, ten times the density of a standard web server rack, forcing facilities to deploy liquid cooling or rear-door heat exchangers that reshape the facility’s capital requirements. A scheduler that optimizes GPU-hours alone is therefore blind to most of what the fleet actually costs.

Whether that fixed capacity is worth owning is a question of utilization. Fleet Economics and Utilization develops the full build-vs.-buy analysis, amortizing three years of capital and operational expenditure against cloud rental to find the break-even utilization that decides between them. The consequence for the scheduler is direct: owned hardware only amortizes when it stays busy, so every point of utilization the scheduler reclaims moves the fleet toward the break-even point that justifies its capital, while a cluster left idle by poor packing or fragmentation makes owned infrastructure more expensive than the cloud it was meant to undercut.

Checkpoint 1.2: Cost-aware scheduling trade-offs

Before moving to ML-specific schedulers, review how the cost-optimization mechanisms interact:

• Spot break-even: determine the checkpoint frequency under which spot training becomes more expensive than on-demand for a 512-GPU, 14-day training job with a 65 percent spot discount, 1 interruption per day, and 30-minute restart overhead.
• Double return: explain why fault tolerance infrastructure has a “double return” for reliability and cost savings, and identify the minimum fault tolerance investment needed to make spot instances viable.
• Reservation sizing: determine whether increasing reserved capacity from 40 percent to 60 percent would save or waste money for an organization.

Spot pricing and reservation strategy squeeze the cost of running a job, but they leave a second source of efficiency untouched: the learning dynamics inside the training loop. The cost models so far optimize time-to-completion, yet the metric that matters to a research organization is time-to-accuracy, the wall-clock time until a run reaches a target validation metric. A scheduler blind to convergence cannot tell whether a run is in its steep early phase or its diminishing-returns tail, so it cannot redirect resources to where they buy the most progress. Closing that gap is the job of the ML-aware schedulers examined in section 1.5.

Self-Check: Question

1. When training large foundation models on preemptible/spot cloud GPU instances discounted by \(70\%\) compared to on-demand pricing, what operational vulnerability poses the greatest threat to training goodput?

   1. Spot instances limit maximum GPU memory to half of on-demand instances due to hypervisor virtualization overhead.
   2. Spot preemption events within a single Availability Zone or accelerator pool are often highly correlated during market capacity crunches, simultaneously terminating all nodes in a multi-node gang and causing repetitive re-rendezvous overhead unless checkpoint frequency is tuned.
   3. Spot instances cannot communicate over InfiniBand networks and must route all AllReduce collectives through public HTTP REST APIs.
   4. Cloud providers prohibit running PyTorch or TensorFlow frameworks on spot instances under standard acceptable use policies.

2. How do Cloud Capacity Blocks (or contiguous reserved GPU instances) differ from standard on-demand and spot instance allocations for large-scale training?

   1. Capacity Blocks guarantee contiguous topological co-location within a single network fabric for a committed time window, eliminating preemption risk and placement fragmentation at the cost of upfront commitment.
   2. Capacity Blocks allow unlimited instantaneous bursting beyond physical cluster limits without financial commitment.
   3. Capacity Blocks dynamically convert idle inference GPUs into spot training nodes with sub-second preemption notices.
   4. Capacity Blocks provide 99% cost discounts by sharing GPU streaming multiprocessors across unrelated cloud tenants.

3. Beyond raw accelerator capital expenditure (CapEx) or hourly cloud VM rates, what major operational and infrastructural factors constitute the Total Cost of Ownership (TCO) of an ML training cluster?

4. In a distributed training architecture with parameter servers or a central coordinator/master node alongside data-parallel workers, how should roles be partitioned between on-demand and spot instances to optimize cost while maintaining system stability?

5. True or False: Because each VM instance in a cloud provider has an independent lifecycle, placing worker nodes of a 64-node distributed training job across different instance types within the same Availability Zone guarantees that preemption events will remain uncorrelated and statistically independent.

See Answers →

Custom ML Schedulers

General-purpose schedulers treat training jobs as opaque boxes, so they cannot tell whether a model is in the steep early phase of learning or the final diminishing-returns phase. Custom ML schedulers trade implementation complexity for that visibility: they peer inside the training loop to optimize time-to-accuracy rather than only time-to-completion. The selection question is which internal signal addresses the fleet’s binding bottleneck, and the four systems differ less by scheduler sophistication than by the signal they ask the platform to trust: consumed service, iteration boundaries, completion value, or measured goodput.

Tiresias: Duration-agnostic scheduling

Tiresias (Gu et al. 2019) addresses the fundamental problem identified in section 1.0.5: ML job durations are unpredictable, and scheduling policies that rely on exact duration estimates can make poor decisions when those estimates are unavailable or unreliable. Rather than fighting this information asymmetry, Tiresias eliminates the requirement for duration estimates entirely. It uses a Two-Dimensional Attained Service¹⁴ scheduler that makes priority decisions based on what a job has already consumed, not what it claims it will consume. Jobs accumulate “service” based on GPU-time consumed, with priority decreasing as service increases. The two dimensions are time (how long the job has been running) and resources (how many GPUs the job uses), capturing both elapsed duration and resource intensity.

Gu, J., M. Chowdhury, K. G. Shin, Y. Zhu, M. Jeon, J. Qian, H. H. Liu, and C. Guo. 2019. “Tiresias: A GPU Cluster Manager for Distributed Deep Learning.” Proceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’19), 485–500.

¹⁴ Attained Service Scheduling: A discipline from OS queuing theory where priority decreases with cumulative resource consumption, approximating the theoretically optimal shortest remaining processing time (SRPT) policy without requiring the future knowledge that SRPT demands. In ML clusters, this approximation is particularly effective because job durations are heavy-tailed: the majority of submitted jobs are short experiments, so deprioritizing high-service jobs correctly identifies the long-running outliers without requiring users to provide runtime estimates they cannot accurately compute.

A discretized version groups jobs into service bins (for example, less than 1 GPU-hour, 1 to 10 GPU-hours, 10 to 100 GPU-hours, greater than 100 GPU-hours), promoting jobs in lower bins to the front of the queue. This bin structure means that all short jobs (the majority of submitted work) receive high priority and run quickly, while the few long jobs that consume most cluster resources gradually lose priority and are deprioritized relative to new arrivals. The key insight is that this behavior approximates the theoretically optimal shortest remaining processing time (SRPT) policy without requiring the future knowledge that SRPT demands: jobs that have consumed little service are statistically likely to be short jobs, so prioritizing them is a good heuristic for minimizing average completion time.

Experiments on production cluster traces from Microsoft and Alibaba show 40 to 60 percent reduction in average job completion time compared to FIFO scheduling, with the largest improvements for short jobs that previously waited behind long-running training runs. The improvement is not free: long-running training jobs experience increased completion times because they are systematically deprioritized. However, the aggregate benefit is positive because there are many more short jobs than long ones, and the short-job improvement exceeds the long-job penalty in total.

Gandiva: Iteration-aware scheduling

Gandiva (Xiao et al. 2018) exploits iteration-boundary time slicing, enabling higher utilization through controlled oversubscription. A cluster with 100 GPUs might support 120 concurrent jobs if each spends 20 percent of its iteration time waiting for data, because the scheduler can interleave data loading from one job with GPU computation from another on the same device. The critical insight that makes this practical is that iteration boundaries provide natural preemption points where GPU state is minimal. Between iterations, only model weights and optimizer state reside on the GPU; the intermediate activations from forward and backward passes have been freed. This minimal state makes context switching cheap (seconds rather than minutes), unlike preempting mid-iteration when the full activation memory is in use.

Xiao, Wencong, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Prabal Patel, et al. 2018. “Gandiva: Introspective Cluster Scheduling for Deep Learning.” 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), 595–610.

Gandiva also implements Grow-Shrink Elasticity at a finer granularity than the framework-level elastic training discussed in section 1.3. Gandiva automatically adjusts data parallelism degree based on real-time resource availability, using profiled iteration times to predict the throughput impact of adding or removing workers. When a high-priority job arrives, Gandiva shrinks lower-priority jobs by reducing their worker count rather than killing them entirely, preserving their progress while freeing resources. When resources become available again, it grows jobs back to their preferred parallelism degree. This fine-grained elasticity, informed by actual iteration-level profiling data, enables scheduling decisions that general-purpose systems cannot make because they lack visibility into job internals.

War Story 1.1: The limping switch (2021)

Context: In a 16,384 GPU distributed cluster, a foundation-model training job experienced severe throughput degradation despite healthy GPU execution metrics (Sigler 2021).

Mechanism: A single transceiver on an inter-rack spine switch experienced elevated bit-error rates, degrading link throughput from 100 Gbps to 1 Gbps (\(100\times\) bandwidth collapse). Standard ping/up health checks reported green, but AllReduce gradient synchronization \(T_{\text{sync}}\) stalled across all 16,384 GPUs waiting for the straggling link.

Impact: Cluster-wide hardware efficiency \(\eta_{\text{hw}}\) collapsed from 45 percent to 8 percent, wasting \(>15,000\) GPU-hours per day until network-level telemetry flagged the degraded switch port.

Fix: Engineers implemented automated micro-benchmark probes and packet-error telemetry to automatically isolate and quarantine degraded switch ports before scheduling job allocation.

Systems lesson: Binary up/down health monitoring is insufficient for distributed scaling. Schedulers require continuous, performance-aware telemetry (switch counters, packet retransmission rates, collective benchmarks) to quarantine degraded network paths automatically.

Themis: Finish-time fairness

Traditional fair-share scheduling treats all GPU-seconds equally: a job consuming 100 GPU-hours is treated the same whether those hours represent the first 10 percent of a long training run or the last 10 percent of a nearly complete one. From a resource accounting perspective, 100 GPU-hours is 100 GPU-hours regardless of context. Themis (Mahajan et al. 2020) argues this resource-centric view is fundamentally unfair because it ignores the sunk cost of work already performed.

Mahajan, Kshiteej, Arjun Balasubramanian, Arjun Singhvi, Shivaram Venkataraman, Aditya Akella, Amar Phanishayee, and Shuchi Chawla. 2020. “Themis: Fair and Efficient GPU Cluster Scheduling.” 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), 289–304.

Themis defines a Finish-Time Fairness metric that allocates resources to minimize the maximum slowdown any job experiences relative to exclusive access (the hypothetical scenario where the job has the entire cluster to itself). Under this metric, a job that is 90 percent complete and needs only 10 more GPU-hours to finish receives higher priority than a job that is 10 percent complete and needs 90 more GPU-hours. The reasoning is economic: delaying the nearly-complete job by an hour wastes the 90 GPU-hours already invested (because those hours only produce value when the job completes), while delaying the early-stage job by an hour has a proportionally smaller impact on the total investment’s return.

This approach benefits shorter jobs and nearly-complete jobs without excessive penalty to longer ones, and it aligns scheduling decisions with the economic value of completing work rather than the simple accounting of resource consumption. Themis implements this metric through an auction mechanism where jobs bid for resources based on their current marginal value (how much their completion time improves per GPU-hour allocated), creating a market-like dynamic that naturally directs resources to their highest-value use.

Pollux: Adaptive resource allocation

Pollux (Qiao et al. 2021) takes the most aggressive approach of the four research schedulers by jointly optimizing resource allocation and training hyperparameters. Where Tiresias, Gandiva, and Themis make scheduling decisions about jobs (when to run, how to share), Pollux makes decisions within jobs (how many GPUs and what batch size). The key observation is that the optimal number of GPUs for a training job depends on the current batch size, learning rate, and gradient noise level, all of which change during training as the model moves through different phases of convergence.

Qiao, Aurick, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R. Ganger, and Eric P. Xing. 2021. “Pollux: Co-Adaptive Cluster Scheduling for Goodput-Optimized Deep Learning.” 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21), 1–18.

Pollux dynamically adjusts each job’s GPU allocation and batch size to maximize Cluster-Wide Goodput, defined as the rate of useful training progress across all jobs simultaneously. The goodput metric folds both sides of the trade-off into one number: statistical efficiency (how much each gradient step advances convergence, which depends on batch size) and system throughput (how many gradient steps per second, which depends on GPU count and communication overhead). A job experiencing diminishing returns from its current GPU count (because communication overhead is growing superlinearly) may have GPUs reassigned to a job that would benefit more (because it is in a phase of training where larger batches improve statistical efficiency).

This co-optimization of scheduling and hyperparameters achieves 37 to 50 percent improvement in average job completion time compared to scheduling with fixed resource allocations. The improvement comes from two sources: better resource allocation (GPUs go to jobs where they produce the most progress) and better batch size tuning (each job runs at a batch size that balances statistical and computational efficiency for its current training phase). The combined effect is greater than either optimization alone.

Scheduler comparison framework

Custom ML schedulers represent a progressive trade-off between implementation complexity and scheduling precision, so the comparison in table 5 matters as a deployment filter rather than a ranking. General-purpose schedulers see declared resources and coarse runtime metadata; these research systems add ML-specific signals such as iteration structure, elasticity, convergence phase, and measured goodput.

Table 5: Custom ML Scheduler Comparison: A taxonomy of research schedulers based on the specific workload characteristic they exploit. Tiresias and Themis optimize for queuing metrics (job completion time, fairness) using external signals, while Gandiva and Pollux optimize for efficiency using internal profiling data.

Scheduler	Key Insight	Scheduling Signal	Optimization Target	Overhead	Production Readiness
Tiresias	Attained service predicts remaining time	GPU-hours consumed	Minimize avg JCT	Low (no profiling)	Medium
Gandiva	Iteration boundaries are preemption points	Iteration timing	Maximize utilization	Medium (profiling)	Medium
Themis	Sunk cost matters for fairness	Completion percentage	Minimize max slowdown	Low (job metadata)	Low (auction complexity)
Pollux	Batch size and allocation are coupled	Goodput (stat + sys)	Maximize cluster goodput	High (continuous profiling)	Low (framework integration)

This design space reveals a fundamental tension between scheduler complexity and scheduling quality (table 5). Tiresias operates with the least information, essentially guessing that “young” jobs will finish soon, yet achieves significant gains simply by preventing large jobs from blocking small ones. It is robust because it requires no cooperation from the user or the training framework. At the other extreme, Pollux requires deep bidirectional integration: the scheduler must know the job’s scaling curve and gradient noise scale, and the job must accept commands to change its batch size and learning rate on the fly. When this integration works, it produces the highest cluster-wide throughput; when the assumptions are violated (for example, a model with unstable convergence dynamics that cannot tolerate batch size changes), the optimization collapses.

Failure modes generally track this complexity gradient. Tiresias fails gracefully: if its assumption (that past usage predicts future usage) is wrong, it simply degrades to a standard fair-share policy. Gandiva’s failure mode is performance jitter: if the profiling phase misestimates iteration variance, it may pack incompatible jobs onto the same node, causing interference. Pollux has the most brittle failure mode because it modifies the training semantics themselves; an incorrect goodput model could theoretically harm model convergence, a risk some production teams are unwilling to take even for its reported job-completion gains.

For our 175B parameter model, these trade-offs dictate a hybrid lifecycle. Pollux offers the highest potential improvement during the early, unstable phase of training where batch sizes are small and the job is elastic; it could dynamically resize the job to fit available holes in the cluster. However, effectively using Pollux requires modifying the training loop to be elasticity-aware. Tiresias offers the simplest deployment path: it would treat the 175B model as a “heavy” job and deprioritize it relative to small experiments, ensuring the cluster remains responsive to researchers without requiring any changes to the 175B model’s code. In practice, most infrastructure teams start with Tiresias-like policies to solve the “blocked queue” problem before attempting the deep integration required by Pollux.

The consistent theme across all four systems is that exploiting domain-specific knowledge about ML workloads (predictable iteration times, diminishing returns from parallel scaling, and inherent checkpoint-restart capability) enables dramatically better scheduling outcomes than static resource requests alone. A generic scheduler sees a job requesting 64 GPUs for 3 days; an ML-aware scheduler sees a stochastic gradient descent process that converges nonlinearly, releases resources if performance plateaus, and can be preempted with minimal loss. The decision framework for selecting among these schedulers depends on the primary bottleneck: for exploratory clusters with high churn, Gandiva’s time-slicing minimizes queuing delay; for production training where job completion time is the primary SLA, Tiresias’s age-based prioritization prevents starvation; for large-scale training where resource efficiency is paramount, Pollux’s adaptive scaling reclaims the “elasticity gap” that static allocation leaves on the table.

The challenge for production systems is incorporating these insights while maintaining the operational reliability, policy flexibility, and organizational governance that production environments require. Production platforms usually adopt scheduler ideas selectively because reliability, governance, heterogeneous hardware, and user policy constrain pure research designs.

From research to production

Research schedulers become useful in production only when their core signal survives contact with a heterogeneous data center. In research, the cluster is often assumed to be a homogeneous pool of accelerators with uniform interconnect topology, and jobs are well-behaved entities that declare their resource needs accurately. In production, the “cluster” is a geological formation of hardware generations: V100s alongside A100s and H100s, connected by a patchwork of InfiniBand and Ethernet, running jobs that frequently crash, hang, or misrepresent their memory requirements. Consequently, no major hyperscaler simply “installs” a research scheduler; instead, they selectively transplant specific algorithms into standard orchestrators via Kubernetes operators or Slurm plugins.

Consider our 175B-parameter model training run. A pure implementation of Pollux might aggressively resize the job based on global cluster load, scaling the number of GPUs up and down to maximize goodput. However, changing the worker count for a synchronous training job requires a checkpoint-restart cycle, which for a model of this size involves writing terabytes of optimizer state to persistent storage. If the re-scaling interval is too short, the I/O overhead of checkpointing negates the throughput gains. In practice, production teams adopt a hybrid approach: they use Pollux-style adaptive allocation primarily during the volatile early phase of training or for hyperparameter sweeps, but lock the 175B model into a static, topology-aware allocation once the learning rate warmup completes and the job enters its months-long steady state.

The build-vs.-buy decision for a custom scheduler is a substantial capital investment. Developing a robust, fault-tolerant scheduler that outperforms the default bin-packing behavior of Kubernetes requires a specialized team of 3 to 5 systems engineers working for 6 to 12 months. The ROI math is what justifies the effort: if the team can improve overall cluster utilization by 10 percent on a fleet of 10,000 H100 GPUs (approximately $25,000 per hour in capital depreciation and power), the savings amount to roughly $22 million per year. This stark return on investment drives the development of internal scheduling platforms at every major AI lab, transforming the scheduler from a mere utility into a primary lever for operational efficiency.

The risk is not only implementation cost; scheduler behavior can also fail to match the execution semantics of distributed training.

War Story 1.2: OpenAI's gang-scheduling deadlock (2021)

Context: OpenAI scaled Kubernetes clusters to 7,500 nodes for workloads including GPT-3, CLIP, and DALL-E. Their large training jobs used MPI-style execution, where all workers in a group had to be scheduled simultaneously before training could make progress (Sigler 2021).

Mechanism: Default Kubernetes scheduling placed pods incrementally without all-or-nothing constraints. When two large experiments requested cluster capacity, Kubernetes scheduled half of each job’s pods across nodes.

Impact: Both training jobs stalled indefinitely while holding allocated GPUs, producing zero training progress and deadlocking cluster resources.

Fix: OpenAI developed custom Kubernetes gang-scheduling operators that enforce atomic all-or-nothing pod placement.

Systems lesson: Distributed ML scheduling must encode the job’s execution semantics. For MPI-style training, “some pods scheduled” is not partial progress; it is wasted allocation. Production schedulers need gang scheduling, quota controls, and workload-aware policies rather than generic pod placement alone.

Sigler, Eric. 2021. Scaling Kubernetes to 7,500 Nodes. OpenAI Blog.

A scheduler designed for general-purpose pod placement will mishandle workloads that require gang semantics, and adding gang awareness alone does not resolve priority conflicts among competing training jobs. Each of the four research schedulers examined in this section addresses one facet of this problem (runtime estimation, iteration-boundary preemption, fairness across finish times, or joint resource-hyperparameter tuning), but none addresses all facets simultaneously. The design space is therefore a set of explicit trade-offs rather than a single optimal policy.

Checkpoint 1.3: Custom scheduler design space

Consider the trade-offs between the four research schedulers:

• Estimate-free scheduling: identify the information Tiresias sacrifices by eliminating runtime estimates, and describe when that sacrifice hurts scheduling quality.
• Time-slicing limits: identify workloads for which Gandiva’s iteration-boundary time-slicing fails, and explain why.
• Job starvation: assess whether Themis’s preference for nearly complete jobs can starve long-running pretraining jobs indefinitely.
• Goodput-model risk: describe the failure mode when Pollux’s goodput model is wrong.

Custom schedulers maximize throughput for long-running training jobs, but the operational reality changes drastically when models are deployed for inference. The objective function flips from maximizing cluster utilization to guaranteeing millisecond-level responsiveness. Serving resource management must handle bursty, unpredictable traffic patterns while adhering to strict SLOs.

Self-Check: Question

1. How does the Tiresias scheduler solve the problem of inaccurate and inflated user runtime estimates in GPU cluster scheduling?

   1. It uses a Two-Dimensional Gittins Index (Least Attained Service) based on observed spatial (GPU count) and temporal (elapsed execution time) resource attainment, prioritizing jobs that have consumed less cumulative compute without asking users for duration estimates.
   2. It requires users to pay monetary deposits that are forfeited if their job runtime deviates by more than \(10\%\) from their initial estimate.
   3. It parses Python ASTs at submission time to statically calculate the exact loop iteration count and FLOP requirements of every training script.
   4. It forces all jobs to run for exactly 60 minutes before automatically killing and requeueing them.

2. In the Pollux scheduling architecture, how is the Goodput metric (\(G_j\)) defined to co-optimize cluster resource allocation and job-level training dynamics?

   1. Goodput is the total number of raw GPU clock cycles executed divided by the cluster’s electricity consumption in kilowatt-hours.
   2. Goodput is the ratio of network packets transmitted over InfiniBand to the number of retransmissions caused by packet drops.
   3. Goodput is the product of job training speed (step throughput \(T_j\), steps per second) and statistical efficiency (\(E_j\), progress made per step toward convergence given the current batch size and worker allocation).
   4. Goodput is the percentage of time GPUs spend running CUDA kernels compared to CPU data loading time.

3. What mechanisms does Gandiva use to achieve fine-grained, low-overhead time-slicing and job migration across GPUs?

4. The custom ML scheduler Themis optimizes for ____ fairness, ensuring that each job completes in a time comparable to what it would achieve in a dedicated cluster share, thereby preventing gaming and multi-resource allocation monopolies.

5. Order these custom ML scheduler milestones chronologically by their introduction of specialized ML scheduling primitives:

1. Pollux: Co-optimizing cluster resource allocation with dynamic batch size tuning via statistical Goodput modeling.
2. Gandiva: Iteration-aware scheduling enabling sub-second time-slicing and migration at mini-batch boundaries.
3. Traditional HPC schedulers (Slurm / PBS): Static batch queue allocation with priority and fair-share decay.
4. Tiresias: Duration-agnostic 2D Least Attained Service (LAS) scheduling eliminating reliance on user runtime estimates.

See Answers →

Serving Resource Management

Training jobs are predictable, batch-oriented workloads that run for weeks; serving workloads are volatile, user-facing services that must respond in milliseconds to unpredictable traffic spikes. When an e-commerce platform launches a major sale, the serving fleet must instantly autoscale to handle thousands of requests per second. Serving resource management forces the orchestrator to balance these aggressive latency constraints against raw hardware utilization.

For orchestration, the important serving fact is that inference capacity is a live envelope around user traffic rather than a batch allocation around a training job. The fleet orchestrator manages that envelope by scaling replica counts up and down with demand, isolating serving workloads from interference, and balancing inference resource needs against training’s claims on the same shared infrastructure.

Autoscaling for inference

Kubernetes Horizontal Pod Autoscaling (HPA) adjusts replica counts based on observed metrics, adding instances when load increases and removing them when load decreases. The default autoscaling metric for general cloud workloads is CPU utilization, with targets of 50 to 70 percent. This default poorly reflects GPU inference workloads for two reasons. First, GPU utilization is a noisy metric that can be high even when the system is not processing user requests (background maintenance, model warmup). Second, the relationship between GPU utilization and inference latency is highly nonlinear: latency can spike dramatically when utilization crosses a threshold (typically 70 to 80 percent for GPU inference), making utilization-based scaling reactive rather than predictive. The metric choice is therefore the first scheduling decision for serving: table 6 lists custom inference metrics that predict user-facing degradation before it occurs.

Table 6: Inference Autoscaling Metrics: Custom metrics for GPU inference workloads capture the relationship between load and user-facing performance more accurately than default CPU utilization. Queue depth provides a leading indicator of latency degradation, while pending tokens captures outstanding autoregressive decode work; KV-cache residency and context-length distribution then determine whether that work is also memory-pressure risk.

Metric	Target Range	Considerations
GPU utilization	60 to 80%	Varies by model batch efficiency
Request queue depth	10 to 50 requests	Prevents latency spikes before they manifest in P99 metrics
P99 latency	Below SLO target	Reactive metric that lags demand changes by seconds to minutes
Pending tokens	Model-specific	Tokens queued or currently being decoded across active requests

Beyond the HPA-oriented metrics in table 6, Vertical Pod Autoscaling (VPA) operates on a different axis, adjusting resource requests and limits for individual pods. Where HPA changes how many instances run, VPA changes how much resource each instance receives. For inference, VPA can right-size memory allocations based on observed usage patterns, preventing over-provisioning of CPU memory and host resources that reduces the number of inference pods that fit on each node. GPU resources cannot be vertically scaled without pod restart (the CUDA context must be reinitialized), limiting VPA’s utility for accelerated workloads where model loading takes minutes. However, VPA is valuable for the CPU components of inference pipelines, such as preprocessing, postprocessing, and tokenization, where resource requirements may differ significantly from initial estimates and can be adjusted with a brief pod restart.

Large language model (LLM) inference requires specialized scaling considerations that neither HPA nor VPA fully address, due to the Key-Value (KV) Cache¹⁵ memory growth pattern. A 70B parameter model serving long-context requests may require more than 80 GB of GPU memory for KV cache alone, even when the serving engine stores that cache in paged, noncontiguous blocks. The GPU memory bottleneck for LLM serving is therefore not the model weights (which are fixed) but the KV cache (which grows with request count and context length). Scaling decisions must account for both request rate and context length distribution, not just computational load. A sudden increase in requests with long contexts (for example, a shift from short chatbot queries to document summarization workloads) can exhaust GPU memory even at moderate request rates, requiring scale-out despite low GPU compute utilization. Conversely, a burst of many short-context requests may stress compute without threatening memory limits.

¹⁵ Key-Value (KV) Cache: Stores precomputed attention key and value tensors to avoid recomputing attention over the entire sequence for each new token. KV cache grows linearly with sequence length and batch size; for a 70B model with 128K context, the cache can exceed the model weights in memory consumption. This growth pattern creates a scheduling failure mode invisible to compute-based autoscalers: GPU memory exhaustion at low compute utilization, causing requests to stall even when the GPU appears idle.

This dual resource dimension (compute and memory) suggests that effective LLM autoscaling should monitor both GPU compute utilization and GPU memory pressure, scaling out when either approaches critical thresholds. Some production systems define a composite scaling metric that combines these dimensions, triggering scale-out when the maximum of normalized compute utilization and normalized memory pressure exceeds a threshold. This composite approach prevents the system from being surprised by either type of resource exhaustion.

Autoscaling for inference must also handle Cold Start Latency, which creates a tension between cost efficiency and responsiveness that has no simple resolution. The cold-start problem in serving is quantitatively more severe than in standard web services due to the sheer mass of model weights. A 70B-parameter model in FP16 occupies approximately 140 GB, and all of it must reach GPU memory before the first request can be served, a transfer bound by PCIe or NVLink bandwidth. Even at the per-direction PCIe Gen4 x16 speed of 32 GB/s, the raw transfer takes over 4 seconds, and in practice model initialization, graph compilation, and safety checks extend this to 60–120 seconds. Loading from Non-Volatile Memory Express (NVMe) SSDs at 7 GB/s read bandwidth takes roughly 20 seconds; from network storage, it can exceed 2 minutes.

Aggressive scale-down policies that terminate replicas during brief traffic lulls create painful cold starts when traffic returns minutes later, producing latency spikes that violate SLOs and degrade user experience. The fundamental problem is that inference traffic exhibits burstiness at multiple timescales (seconds, minutes, hours), and short-term lulls do not reliably predict sustained low demand. A five-minute quiet period might be followed by a traffic spike, and the cost of a cold start during that spike (degraded latency for all requests while the model loads) can exceed the cost savings from the few minutes of freed GPU capacity.

Production systems address cold start through three complementary strategies:

• Minimum Replica Count: Keeping at least one warm replica above zero eliminates cold starts for the first burst of traffic but incurs a baseline cost even during truly idle periods.
• Predictive Scaling: Scaling from historical traffic patterns, such as diurnal cycles, day-of-week patterns, and known events, adds capacity before expected peaks rather than reacting after load arrives.
• Prewarming: Loading model replicas into GPU memory on standby GPUs reduces activation from minutes of model loading to seconds of process initialization, but standby replicas consume GPU memory that cannot serve other workloads.

Organizations with multiple models competing for the same GPU pool must balance the prewarming overhead against the cold-start penalty, choosing which models to keep warm based on their traffic patterns and latency requirements.

Predictive autoscaling and SLO management

Reactive autoscaling (like Kubernetes HPA) is inherently backward-looking: it observes a metric breach (for example, GPU utilization exceeding 80 percent), waits for a stabilization window, and then triggers a scale-up event. For a container that starts in milliseconds, this lag is negligible. For a 175B parameter model that takes 3 minutes to load weights from disk to GPU memory, this lag is fatal to SLOs. If traffic doubles in 30 seconds, a realistic scenario during a product launch or viral event, the reactive scaler will provision new replicas only after the surge has already saturated the existing fleet, causing a 3-minute window of degraded latency and dropped requests.

During this cold-start gap, continuous batching (also called in-flight batching) functions as a shock absorber. Serving engines that implement continuous batching, such as vLLM or TensorRT-LLM, do not wait for a fixed batch to fill before beginning generation; instead, they interleave new incoming requests into ongoing decode steps whenever a sequence slot becomes free. When load spikes and new replicas are still loading, the engine responds by increasing the live batch size toward the KV-cache memory limit, accepting more concurrent sequences per iteration. This absorbs additional requests without rejecting them, sustaining acceptable latency at the cost of higher per-request memory pressure. The relief is bounded: once the KV cache is fully saturated (meaning active sequences occupy every available memory page) the engine must begin queuing or shedding load regardless of batch elasticity. The scheduler therefore has a narrow window, typically from the moment the autoscaler fires until KV-cache saturation, in which continuous batching buys time. Tuning minimum replica counts and scale-up thresholds to keep that window open is the principal lever for preventing SLO violations during traffic ramps that exceed cold-start duration.

Reactive autoscaling opens an SLO gap during cold start.

Predictive autoscaling decouples scaling actions from current load. Analyzing historical traffic patterns (diurnal cycles, day-of-week seasonality) and incorporating real-time leading indicators (for example, a surge in login requests often precedes a surge in inference queries) allows the scheduler to preprovision capacity before the demand arrives. Serving our 175B model with a 500 ms P99 SLO requires this anticipation. If the model takes 3 minutes to become ready, the predictive scaler must issue the scale-up command at least 4 minutes before the expected traffic ramp. This transforms the scaling problem from a control theory problem (reacting to error) to a forecasting problem (predicting the future).

Effective scaling policies must be SLO-driven rather than utilization-driven. Targeting a fixed utilization (for example, “keep GPU at 70 percent”) is a proxy that often fails: a model might hit its latency SLO at 60 percent utilization due to memory bandwidth contention, or it might safely run at 90 percent utilization if the requests are compute-bound and uniform. An SLO-driven policy explicitly targets the metric that matters: “Scale up when P99 latency exceeds 400 ms (80 percent of the 500 ms target).” This approach automatically adapts to changes in workload characteristics. If a new model version is less efficient, the latency metric will rise faster, and the scaler will provision more replicas to maintain the SLO, without requiring manual tuning of utilization thresholds.

To prevent Oscillation, where the scaler rapidly adds and removes replicas during noisy traffic, production systems implement Hysteresis and Cooldown Periods. A typical policy scales up aggressively (no stabilization window) to protect the SLO, but scales down conservatively (15-minute cooldown) to avoid thrashing. This asymmetry acknowledges that the cost of an unnecessary scale-up (wasted compute for 15 minutes) is far lower than the cost of a missed scale-up (SLO violation and user churn).

Napkin Math 1.5: The autoscaling lag

Consider a serving fleet with 10 replicas, each capable of 5 QPS while meeting the 500 ms SLO. Total capacity: 50 QPS.

Scenario: Traffic ramps from 40 QPS to 80 QPS linearly over 60 seconds. Model load time: 3 minutes (180 seconds).

Reactive Scaling: HPA detects overload at \(t =\) 15 s (when traffic hits 50 QPS). It requests 10 new replicas. These replicas become ready at \(t =\) 195 s (15 + 180 seconds). From \(t =\) 15 s to \(t =\) 195 s, demand exceeds capacity. During the ramp, excess demand grows linearly from 0 to 30 QPS; after the ramp reaches 80 QPS, the full 30 QPS overload persists until the new replicas are ready. The area above capacity is approximately 4,725 request-equivalents of accumulated excess demand. These requests become backlog or are dropped according to the serving policy; converting this quantity into an SLO-violation count requires a queueing model that tracks request latency and drops.

Predictive scaling: The scaler forecasts the ramp and triggers scale-up at \(t =\) -180 s. The 10 new replicas come online at \(t = 0\) s, pushing capacity to 100 QPS just as the ramp begins. Under these idealized assumptions, the ramp creates no capacity-induced backlog.

Resource isolation

When multiple inference workloads share the same physical hardware, noisy neighbor problems arise when one workload’s resource consumption degrades another’s performance. On GPUs, interference manifests through four shared resource channels:

• Memory bandwidth: One workload’s data movement can saturate the HBM interface.
• L2 cache contention: One workload’s working set can evict another’s cached data.
• PCIe bottlenecks: Concurrent host-device transfers can compete for bus bandwidth.
• Thermal effects: One workload’s heat generation can cause thermal throttling that affects all workloads on the same GPU.

For latency-sensitive inference, even minor interference can push P99 latency above SLO thresholds, turning a seemingly well-provisioned system into an SLO-violating one.

MIG sits at the strong-isolation end of this design space. Table 2 already showed the fixed profiles; the operational implication is that hardware isolation removes most cross-tenant interference but turns partition choice into a placement constraint. MIG is available only on A100 and later GPUs, profiles cannot change without draining all workloads from the GPU, and fixed partition sizes may not match workload requirements. A workload that needs 15 GB of GPU memory wastes 5 GB on a 20 GB MIG instance or cannot fit on a 10 GB instance.

Software approaches to isolation provide more flexibility at the cost of weaker guarantees. Efficiently sharing GPUs for inference requires navigating a hierarchy of isolation mechanisms, each trading performance for security. At the simplest level, CUDA Time-Slicing allows multiple processes to share a GPU by context-switching the compute resources. While flexible, this incurs high latency penalties (often 10–20 ms) and offers no memory isolation. NVIDIA’s Multi-Process Service (MPS)¹⁶ improves this by allowing kernels from different processes to run concurrently on the same GPU, improving throughput for small batch inference while providing weaker isolation than MIG (Volta and newer give each client a separate GPU address space, but compute and memory bandwidth remain shared).

¹⁶ CUDA MPS (Multi-Process Service): Enables concurrent kernel execution from multiple processes on a single GPU, eliminating the 10 to 20 ms context-switch penalty of time-slicing. MPS adds approximately 5 to 10 microseconds of overhead per kernel launch but provides no memory isolation: a process that exhausts GPU memory crashes all colocated workloads. This makes MPS suitable for trusted, same-team inference workloads but dangerous for multi-tenant environments where a single misbehaving model can take down its neighbors.

GPU memory isolation prevents one model from consuming memory needed by another through explicit memory limits enforced by the container runtime. Without such limits, a memory leak, an unexpectedly large batch (triggered by a request with unusually long context), or a misbehaving custom kernel can consume all available GPU memory and crash colocated workloads. MPS improves utilization for small workloads by eliminating the context-switching overhead of time-slicing, but it adds latency overhead of approximately 5 to 10 microseconds per kernel launch and provides limited protection against memory bandwidth interference.

Furthermore, long-running serving instances suffer from dynamic memory fragmentation. The key culprit is the KV cache in transformer inference, which grows and shrinks with request sequence length. Standard allocators struggle with these highly variable lifetimes, leaving holes in GPU memory too small for new requests but collectively wasting gigabytes. Serving engines increasingly use paged memory management to allocate KV cache in noncontiguous blocks, reducing fragmentation-induced out-of-memory failures; Inference at Scale develops the paged-allocation mechanism in full.

At the CPU level, core pinning assigns specific CPU cores to inference pods, preventing the Linux scheduler from migrating processes between cores in ways that invalidate processor caches. For latency-sensitive workloads, isolating cores using isolcpus kernel parameters and taskset affinity removes OS scheduling jitter that manifests as random latency spikes. Combined with non-uniform memory access (NUMA)-aware¹⁷ placement that ensures inference pods access memory through the nearest memory controller (avoiding remote NUMA access that adds 50 to 100 ns per access), this reduces P99 latency by 10 to 30 percent for sub-millisecond inference tasks. The principle is straightforward: inference latency is dominated by memory access patterns, and any source of memory access unpredictability, whether cache eviction from core migration, remote NUMA access, or TLB misses from address space switching, directly degrades tail latency.

¹⁷ NUMA (Non-Uniform Memory Access): Local memory access takes approximately 100 ns vs. 150 to 200 ns for remote-socket access, a 50 to 100 percent penalty. For ML inference preprocessing, placing CPU cores on the wrong NUMA domain relative to the GPU’s PCIe connection adds this penalty to every host-device transfer, inflating P99 latency by 10 to 30 percent for sub-millisecond inference tasks. The fix is straightforward (CPU pinning and NUMA-aware pod scheduling) but often overlooked in Kubernetes deployments where the default scheduler ignores NUMA topology entirely.

The isolation techniques discussed here represent a spectrum from strong (MIG, hardware isolation) to flexible (MPS, software sharing), and the choice depends on the trust boundary between colocated workloads. Multi-tenant inference platforms serving different external customers typically require MIG for security isolation. Single-tenant platforms serving different internal models on the same GPU can use MPS or time-slicing where the isolation requirements are weaker.

GPU sharing economics

The decision to co-locate multiple models on a single GPU defines the efficiency frontier of an inference fleet. This choice operates on a spectrum: at one end, Exclusive Access guarantees isolation but strands capacity; at the other, Aggressive Sharing (via MPS or time-slicing) maximizes utilization but risks latency interference. The physics of this trade-off are dictated by memory fragmentation and compute contention.

Sharing packs two tenants, cutting wasted memory from 67 percent to 35 percent.

Consider an 80 GB A100 GPU serving a 7B parameter model. The model weights in FP16 consume approximately 14 GB. With a typical KV cache and activation overhead, the total runtime footprint is roughly 26 GB. Under an exclusive access policy, this single model leaves 54 GB (67 percent) of the GPU’s high-bandwidth memory dark, a waste of silicon capital. Partitioning the GPU with MIG (for example, a 3g.40gb profile) tightens the container, reducing the waste to roughly 35 percent within the partition, but still leaves the remainder of the GPU strictly segmented. Enabling MPS to pack two such models onto the same 80 GB device consumes 52 GB, dropping the aggregate memory waste to just 35 percent. For fleets running hundreds of small models, shifting from exclusive to shared hosting can reduce the required GPU count by a factor of 2–3\(\times\).

This density comes at the cost of Interference. When two workloads share a GPU via MPS, they compete for SMs and memory bandwidth. The degradation is workload-dependent: two compute-bound models (for example, large batch processing) fighting for ALUs often see 15 to 20 percent throughput degradation each. However, a compute-bound model co-located with a memory-bound model (for example, decoding-heavy generation) often coexist peacefully, seeing only 5 percent degradation because they bottleneck on different hardware resources. The scheduler’s job thus becomes a multi-dimensional bin packing problem: finding complementary workloads that maximize density while respecting latency SLOs.

For a heterogeneous fleet, this logic dictates a bifurcated strategy. Our 175B parameter model requires about 350 GB just for FP16 weights; on 80 GB A100s that means at least five GPUs before KV cache, activation buffers, runtime workspace, and tensor-parallel granularity are included. In practice, an 8-GPU tensor-parallel replica is the cleaner scheduling unit and is a candidate for exclusive access. Conversely, the dozens of 1B to 7B parameter support models, such as toxicity classifiers, query embedding encoders, and auxiliary generation models, are prime candidates for sharing. Packing these smaller models onto shared “utility nodes” liberates high-performance clusters for the heavy lifting of foundation model inference.

Napkin Math 1.6: GPU sharing ROI

Consider a fleet serving 100 distinct 7B models (26 GB footprint each) and 10 distinct 175B models (350 GB footprint each).

Strategy A – exclusive access (one model per GPU): The 7B models require 100 A100 GPUs (80 GB each), achieving only 32.5 percent memory utilization (100 \(\times\) 26 GB used out of 8000 GB total). The 175B models require 80 GPUs (10 models \(\times\) 8 GPUs each for tensor parallelism). Total fleet: 180 GPUs.

Strategy B – mixed sharing (MPS for small, exclusive for large): Pack 2 models per A100 for the 7B models (2 \(\times\) 26 GB = 52 GB, well within 80 GB), requiring only 50 GPUs at 65 percent memory utilization. The 175B models remain unchanged at 80 GPUs. Total fleet: 130 GPUs.

Sharing reduces the fleet size by 27.8 percent (50 fewer GPUs). At $2/GPU-hour, this saves approximately $876,000 annually in compute costs, purely through scheduling policy.

Model routing and traffic management

Once resources are isolated, the orchestrator faces the challenge of directing inference queries to the correct model replicas. For a monolithic web service, simple round-robin load balancing suffices. For a 175B parameter language model served across thousands of GPUs, routing becomes a complex distributed systems problem where the “unit of serving” is no longer a single container but a Tensor Parallel Group, a collection of 8 GPUs that must function as a single logical entity. The load balancer cannot simply route a request to “GPU 12”; it must route to “TP Group 3,” ensuring the request arrives at the exact moment the group is ready to process it.

Request routing strategies determine cluster-wide throughput. Naive Round-Robin distribution fails for generative workloads because request processing times vary by orders of magnitude based on output token length. A round-robin scheduler inevitably sends new requests to replicas backed up with long-generation tasks, causing tail latency to spike. Least-Outstanding-Requests routing improves this by directing traffic to the idlest replicas. However, for large language models, Memory-Aware Routing is superior. Tracking the KV cache occupancy of each replica allows the router to send long-context requests to replicas with ample free memory and short requests to those near capacity, preventing fragmentation-induced out-of-memory errors and increasing effective batch sizes by 30 to 40 percent compared to blind routing.

Traffic shifting mechanisms like canary deployments and A/B testing are essential for safe model updates. When deploying Model v2, the orchestrator does not simply replace Model v1. Instead, it spins up v2 replicas alongside v1, creating a shadow fleet. The traffic manager routes 1 percent of live requests to v2 (often in “shadow mode,” where the response is computed but discarded) to validate latency and error rates against v1. Only after statistical verification does the orchestrator gradually shift weights (5 percent, 25 percent, 100 percent), draining v1 replicas only as v2 proves stable.

Multi-model serving complicates this further. A single cluster often hosts diverse models: a product recommendation model (10 ms P99 SLO, high throughput) alongside a coding assistant model (2,000 ms P99 SLO, bursty usage). Placing these on the same node invites interference; placing them on separate clusters strands capacity. The optimal strategy uses Latency-Aware Bin Packing: the orchestrator co-locates latency-critical models with batch-processing jobs (like embedding generation) that can be throttled instantly, ensuring high utilization without violating strict SLOs.

The training-serving resource boundary

The deepest utilization divide in the ML fleet is the wall between training and serving. Most organizations manage these as separate fiefdoms: a “Production Serving” cluster that must handle peak traffic (and sits 50 percent idle at night) and a “Research Training” cluster that is perpetually backlogged. This Static Partitioning is safe but economically inefficient. Dynamic Sharing unifies these pools, but it introduces a fundamental friction: the mode-switching cost.

Repurposing a GPU from training to serving is not instantaneous. The orchestrator must checkpoint the training job (1 to 5 minutes), kill the container, launch the inference server, load the model weights from storage (2 to 10 minutes), and run warmup queries (1 to 5 minutes) to populate compilation caches. This 5-to-20-minute switching latency means GPUs cannot react to second-by-second traffic bursts. Instead, they must follow diurnal patterns.

Inference traffic typically follows a “human curve”: low at 3 AM, ramping up at 8 AM, peaking at 2 PM, and tapering off at 10 PM. A smart orchestrator forecasts this curve 30 minutes in advance. At 7:30 AM, it preempts low-priority training jobs (like hyperparameter sweeps) to warm up inference replicas. At 10:30 PM, as traffic drops, it drains inference replicas and releases GPUs back to the training scheduler.

Napkin Math 1.7: The diurnal GPU shift

Consider a fleet of 1,024 GPUs managed under two regimes:

Scenario A – static partitioning: The serving pool reserves 400 GPUs to handle peak demand, but average usage is only 150 GPUs (37.5 percent utilization). The training pool reserves 618 GPUs, and backlog ensures 100 percent utilization. Result: 150 GPUs + 618 GPUs = 768 active GPUs. Fleet utilization: 75 percent.

Scenario B – dynamic sharing: Serving claims GPUs strictly as needed (average 150 GPUs). Training reclaims unused serving GPUs (average 250 GPUs), minus a 117-GPU safety buffer. The orchestrator shifts approximately 133 GPUs between roles twice daily based on diurnal forecasts. Result: serving (150 GPUs) + training (618 GPUs + 133 GPUs) = 901 active GPUs (the second training term is reclaimed serving capacity). Fleet utilization: 88 percent.

Dynamic sharing recovers 133 GPUs of effective capacity. At $2/GPU-hour, this creates approximately $2.3M per year in value without purchasing a single additional accelerator.

Optimizing isolation for a single deployment is straightforward, but maintaining strict guarantees becomes volatile when thousands of training and serving workloads coexist on the same infrastructure. The collision of high-priority inference SLAs with resource-hungry training jobs creates a noisy neighbor problem that simple containerization cannot solve. Multi-tenancy and quotas become the governance layer that arbitrates fair access and prevents tragedy-of-the-commons scenarios in shared fleets.

Self-Check: Question

1. Why is standard CPU/GPU utilization (such as reported by Kubernetes Horizontal Pod Autoscaler) an ineffective metric for autoscaling LLM inference replicas, and what metric should be used instead?

   1. GPU utilization is strictly proprietary to NVIDIA and cannot be exported to Prometheus.
   2. GPU utilization measures compute core occupancy rather than request latency or queue saturation; an LLM serving replica can be stalled on KV-cache memory capacity or request concurrency while reporting modest GPU SM utilization, making request concurrency or queue depth the superior scaling metric.
   3. GPU utilization fluctuates faster than the speed of light, violating Nyquist sampling criteria in etcd.
   4. LLM serving replicas do not use GPUs during inference, relying entirely on CPU vector instructions.

2. In a multi-tenant inference environment hosting multiple smaller models on shared GPUs, what is the primary architectural trade-off between NVIDIA Multi-Process Service (MPS) and Multi-Instance GPU (MIG)?

   1. MIG is only supported on consumer GeForce cards, whereas MPS requires enterprise Grace Hopper superchips.
   2. MPS provides sub-nanosecond hardware memory encryption, whereas MIG only supports unencrypted cleartext transfers.
   3. MPS requires dedicated physical PCIe switches per tenant, whereas MIG operates over standard Ethernet cables.
   4. MPS dynamically shares streaming multiprocessors (SMs) and memory bandwidth with high packing density but lacks hard memory fault isolation (an OOM in one client can crash the MPS server), whereas MIG enforces strict hardware isolation (dedicated SMs, memory controllers, L2 cache slices) with full fault isolation but static partition boundaries.

3. How can an organization exploit the diurnal traffic cycle of online serving to improve overall GPU fleet utilization, and what preemption mechanism is necessary to protect serving SLOs?

4. Why does reactive autoscaling alone often fail to maintain P99 latency SLOs during sudden inference traffic spikes, and how does predictive autoscaling mitigate this?

5. True or False: In a multi-replica LLM inference cluster, routing user requests randomly via round-robin DNS achieves the highest overall cluster throughput because it balances raw request counts evenly across GPUs.

6. In LLM inference serving, while ____ measures the responsiveness before the first output token appears, Inter-Token Latency (ITL) measures the generation speed of subsequent tokens.

See Answers →

Multi-Tenancy and Quotas

When the computer vision team hoards 500 GPUs for a week while the natural language processing team sits blocked trying to push a critical bug fix to production, shared clusters devolve into a tragedy of the commons. Teams with the most aggressive job submission rates, the largest jobs, or the most persistent resubmission scripts consume disproportionate resources, while teams with more modest or intermittent needs find the cluster perpetually occupied. This creates organizational friction, political escalation to management, and underinvestment in slower-moving but potentially higher-value projects whose teams lack the engineering effort to compete for resources.

Multi-tenancy and quotas provide the organizational and technical firewalls needed to prevent this degradation. The quota systems discussed here establish formal policies for resource allocation, borrowing, and reclamation, so fair access and high overall utilization become enforceable scheduler properties rather than managerial negotiation.

Hierarchical fair-share

GPU quota allocation typically operates at the namespace or project level, defining how much of the cluster each team is entitled to use. The simplest approach allocates fixed GPU counts per team: Team A gets 500 GPUs, Team B gets 300 GPUs, Team C gets 200 GPUs. This approach is easy to understand and implement but leads to systematic underutilization when teams have variable workloads. If Team A’s 500-GPU allocation sits 50 percent idle during a quiet period while Team B’s queue overflows with urgent work, the cluster wastes 250 GPUs of capacity that rigid quotas forbid Team B from using. The alternative, allocating less quota to each team, creates a different problem: teams are blocked from running legitimate work during peak periods even when the cluster has overall spare capacity.

Hierarchical quotas resolve this tension by enabling departmental limits with sub-team flexibility and cross-team borrowing. Equation 4 bounds the effective quota for any team by both its own allocation and the parent department’s capacity that sibling teams have not already claimed, where \(U_{\text{allocated}}\) is the resources currently committed to each of the other teams in the department:

\[ Q_{\text{effective}} = \min\left(Q_{\text{team}},\ Q_{\text{department}} - \sum_{\text{other teams}} U_{\text{allocated}}\right) \tag{4}\]

When aggregate demand exceeds capacity, each team receives resources proportional to its share allocation, ensuring that no team is systematically disadvantaged. Unused capacity borrows down the hierarchy: if Team A uses only 60 percent of its 500-GPU allocation, the remaining 200 GPUs become available to other teams within the same department. These borrowed resources carry lower preemption priority than owned resources: when Team A later submits jobs that need its full allocation, the scheduler reclaims borrowed capacity by preempting jobs running on those resources, returning capacity to its rightful owner.

This borrowing mechanism is essential for achieving high utilization in multi-tenant clusters. Without borrowing, organizations face a painful choice between allocating enough quota to handle each team’s peak demand (guaranteeing low utilization when demand is uneven, since peaks rarely coincide across all teams) or allocating less (guaranteeing that some team is always quota-blocked during their peak periods). Borrowing resolves this dilemma by allowing peak-level allocations as guarantees while permitting idle capacity to flow to where it is needed. A team that is guaranteed 500 GPUs can always get 500 GPUs when needed (through preemption of borrowed jobs), but it does not waste the cluster’s capacity when its actual demand is lower.

The preemption dynamics of borrowing require careful integration with the checkpoint infrastructure from Fault Tolerance. Jobs running on borrowed capacity must checkpoint frequently enough that preemption (when the owning team reclaims resources) does not waste significant work. Shared clusters often distinguish between “guaranteed” and “opportunistic” scheduling tiers: guaranteed jobs run on the team’s owned quota and face preemption only from higher-priority work, while opportunistic jobs run on borrowed capacity and accept that they may be preempted when the owning team needs its resources back. Training frameworks that support elastic scaling can respond to borrowed-capacity reclamation by shrinking rather than terminating, further reducing the cost of the borrowing mechanism.

Burst capacity and over-subscription

In a shared GPU cluster, static quotas inevitably lead to the “burst capacity” paradox: Team A has a quota of 64 GPUs but needs 256 for a weekend-long training run, while Team B’s allocated 64 GPUs sit idle. The solution requires decoupling “guaranteed quota” from “limit quota.” A team is guaranteed 64 GPUs, which they can access instantly, but can burst up to 512 GPUs if the cluster has slack capacity. This “over-subscription” model relies on preemption: if Team B wakes up and reclaims their guaranteed share, the scheduler must immediately terminate Team A’s burst jobs.

Burst capacity handling enables teams to temporarily exceed their quotas when cluster-wide resources are available, providing a more aggressive sharing mechanism than quota borrowing. Where borrowing reassigns idle capacity from one team to another, burst capacity allows teams to exceed the total allocated capacity by exploiting the gap between requested resources and actual utilization. Over-commitment ratios such as 1.2–1.5\(\times\) are plausible when admission controllers track actual vs. requested resources and intervene when contention materializes. When contention occurs, jobs using burst capacity face preemption first, protecting guaranteed allocations within each team’s owned quota.

To manage this safely, clusters must implement strict priority classes:

• Production Serving: Priority 0 workloads are nonpreemptible.
• Production Training: Priority 1 workloads handle core model updates and are only preemptible by serving outages.
• Interactive/Debug: Priority 2 workloads get high scheduling precedence but short time limits.
• Batch/Research: Priority 3 workloads are fully preemptible and fault-tolerant.

This hierarchy ensures that a massive hyperparameter sweep fills the cluster’s cracks but evaporates instantly when a critical retraining job arrives.

The appropriate over-commitment ratio depends on workload characteristics and requires careful observation of actual resource utilization patterns. If most jobs request 100 percent GPU utilization but actually achieve 70 percent (due to data loading phases, communication overhead, memory allocation gaps, or suboptimal kernel scheduling), a 1.3\(\times\) over-commitment ratio improves cluster-wide utilization without significant contention because the total actual demand (70 percent \(\times\) 1.3 = 91 percent) remains below physical capacity. If jobs are genuinely compute-bound and sustain near-100 percent GPU utilization, over-commitment leads to resource contention and performance degradation for all colocated workloads. The over-commitment ratio should therefore be calibrated empirically based on cluster-wide utilization telemetry, not set based on theoretical assumptions about workload behavior.

Resource accounting and observability

The 64 GPUs that sat idle for a weekend because a researcher forgot to cancel a reservation represent invisible waste in a multi-tenant cloud fleet. In a dedicated cluster, such waste is at least visible: the lights are on, but the fans are quiet. In a shared environment, the cost is hidden and compounds across thousands of jobs. Effective governance requires moving beyond simple “allocation” metrics to a tiered accounting model that distinguishes between what was requested, what was used, and what was useful.

Three tiers of measurement expose this gap:

• Allocated capacity: The resources the scheduler has reserved for a job are unavailable to others.
• Compute utilization: The percentage of time GPU kernels are active measures hardware busyness.
• Productive utilization: The fraction of time the GPU advances model state excludes data loading pauses, communication overhead, and checkpointing.

The distinction is financial, not merely technical. A team might be “allocated” 500 GPUs but only using 350 (70 percent compute utilization) and only “productively using” 280 (56 percent of allocation). If the organization pays for allocation but measures success by training progress, this 44 percent gap represents pure burn.

Allocated, active, and productive utilization drift far apart.

Attribution in a shared fleet presents a forensic challenge. A single “training” job might be a shared experiment between three teams, or a platform test run by a site reliability engineer. Without granular tagging at the job level, costs default to the “infrastructure” bucket, creating a tragedy of the commons where no one owns the bill. Kubernetes labels and Slurm accounts provide the mechanism for attribution, but the organizational discipline to apply them consistently is the harder problem. Mature organizations enforce “no tag, no schedule” policies, rejecting untagged jobs at the admission controller level.

This data feeds the utilization dashboard, the cluster operator’s primary instrument for fleet health. This view must synthesize per-team utilization against quotas, queue depth by priority class, and the Fragmentation Index, a measure of free GPUs that cannot be allocated due to topology constraints. Crucially, it must track the “cost of chaos”: preemption rates, spot instance interruption frequencies, and the recovery overhead they induce. When these metrics are visible, they drive behavior. A 64-GPU fine-tuning run for our 175B model held for 816 h consumes about $130,560 of capacity (64 GPUs \(\times\) 816 h \(\times\) $2.50/GPU-hour). Without proper accounting, this cost is invisible to the team that requested it. With chargeback, the team must justify this expenditure against the model’s expected business value.

Automated anomaly detection turns these metrics into actionable signals. A sudden spike in a team’s consumption to 3\(\times\) their historical norm might indicate a runaway script spawning infinite jobs. A queue that grows faster than it drains signals a capacity shortfall that auto-scaling must address. Conversely, a sudden drop in cluster-wide power consumption typically precedes a mass job failure event, often due to a shared dependency like a storage outage. These alerts allow operators to intervene before the budget is drained or the queue becomes unmanageable.

Systems Perspective 1.7: The three utilization metrics

To debug cluster efficiency, measure utilization at three distinct layers. Allocated Utilization is the percentage of physical GPUs reserved by the scheduler; low values indicate a lack of demand or overly restrictive quotas. Compute Utilization is the percentage of time allocated GPUs are executing kernels (reported by nvidia-smi); low values indicate inefficient code, I/O bottlenecks, or communication stalls. Productive Utilization is the percentage of allocated time spent effectively training (excluding overhead); low values indicate poor distributed scaling, excessive checkpointing, or frequent restarts.

Security and namespace isolation

Multi-tenancy requires not only fair resource allocation but also security isolation that prevents teams from interfering with or observing each other’s workloads. The stakes are real: ML models represent significant intellectual property, training data may contain sensitive information subject to privacy regulations, and the models themselves may encode proprietary business logic. Without proper isolation, a compromised or misconfigured workload in one team’s namespace could access another team’s model weights, training data, or inference traffic.

In Kubernetes environments, namespace separation provides the fundamental isolation boundary. Each team operates within dedicated namespaces with role-based access control limiting visibility to their own resources. Role-based policies control who can submit jobs, view logs, access model artifacts, and modify scheduling policies within each namespace, providing organizational governance over cluster usage.

Network policies extend isolation to the network layer, preventing cross-namespace communication except through explicitly permitted services. Network policies for ML workloads must balance isolation against the communication requirements of distributed training. A practical policy might allow unrestricted all-to-all communication within a namespace (necessary for ring AllReduce as analyzed in Collective Communication, where every worker must communicate with every other worker) while blocking all ingress from other namespaces (preventing external workloads from intercepting gradient traffic or model updates). Egress policies can prevent training jobs from accessing external networks, reducing data exfiltration risk from compromised training code or poisoned dependencies.

Table 7 compares GPU virtualization options across isolation strength, resource efficiency, and workload fit.

Table 7: GPU Virtualization Isolation Spectrum: GPU sharing mechanisms trade isolation against packing efficiency, so the right mechanism depends on both the trust boundary and the workload’s tolerance for interference.

Option	Isolation strength	Efficiency and flexibility	Suitable workload or trust boundary
Time-slicing	Low; workloads share a GPU through software scheduling.	High flexibility and packing efficiency.	Trusted workloads from the same team sharing a GPU for cost efficiency.
MIG	Strong hardware partitioning with fixed GPU slices.	Predictable isolation but less flexible placement.	Multi-tenant inference where different customers share the same physical GPU.
Full device passthrough	Complete isolation through exclusive access to one or more GPUs.	Lowest packing efficiency.	Training jobs that saturate resources or cannot tolerate interference from colocated work.

The choice depends on the trust boundary between colocated workloads and the performance sensitivity of each workload type. Security in multi-tenant environments extends beyond simple resource fairness. Side-channel attacks on shared GPUs are a documented vulnerability; by monitoring contention on shared caches or memory controllers, a malicious tenant can infer the architecture or even data properties of a co-resident model. In highly sensitive environments, hardware isolation mechanisms like MIG or strictly dedicating entire GPU nodes to single tenants become mandatory requirements.

Priority preemption cascades

One eviction ripples to many: the preemption cascade.

When a high-priority job enters a saturated cluster, the scheduler must decide which running workloads to terminate to free up resources. This decision is rarely isolated. In tightly packed clusters, evicting a medium-priority job to accommodate a high-priority request often triggers a Preemption Cascade, where the evicted job immediately attempts to reschedule itself by displacing lower-priority workloads. Without dampening controls, a single urgent inference service deployment can ripple through the queue, destabilizing dozens of training jobs and forcing a storm of checkpoint reloads that saturate storage bandwidth.

The engineering cost of preemption extends far beyond the scheduling latency. The Preemption Tax is the sum of lost computation since the last checkpoint, the overhead of persisting state, and the cold-start penalty upon resumption. For large-scale distributed training, this tax is nonlinear. Consider the 64-GPU fine-tuning run for our 175B parameter model. If preempted, the job loses the work performed since the last snapshot (average 15 minutes). Upon restarting, it requires 20 minutes to reload the massive optimizer state from distributed storage and another 10 minutes of warmup time as data pipelines refill prefetch buffers, just-in-time (JIT) compilers re-optimize kernels, and GPU caches re-populate.

A further hidden cost compounds this tax: reconstructing the state of the distributed data pipeline. Preempting a large-scale training job does not merely interrupt gradient computation: it discards the deterministic state of the data loader, including the random seed used for shuffling, the shuffle order applied to the epoch, and the byte offset into the dataset shards that records exactly which samples have been consumed. Without restoring this state precisely, the restarted job either re-reads samples already processed (poisoning gradient statistics by amplifying certain examples) or uses a different shuffle order (disrupting the statistical independence between epochs). Restoring a petabyte-scale distributed data pipeline to its pre-preemption state typically requires replaying metadata logs across all data loader workers, a process that can extend the effective warmup window well beyond the time consumed by optimizer state reloading alone. Frameworks that store a compact data loader checkpoint—recording the seed, the sampled permutation index, and the current shard offset—reduce this cost significantly, but the majority of production training pipelines require a manual warmup period to approximate the original data ordering before gradient quality returns to baseline.

During this 45-minute recovery window, the GPUs are active but effectively unproductive. At $2/GPU-hour, a single preemption event costs $96 in wasted compute capital. If the scheduler allows unlimited preemption, 12 preemptions per day waste 9 hours per day of that job’s allocation, or 576 GPU-hours per day for this 64-GPU run. To mitigate this, schedulers enforce Preemption Budgets. These rate limits constrain disruption frequency, such as preventing a job from being preempted more than once per hour or capping total cluster preemption churn to 5 percent of capacity in any 10-minute window. This forces the scheduler to wait for natural job completions rather than triggering a cascade, trading slightly higher pending times for significantly higher aggregate throughput.

Quota governance and organizational dynamics

Schedulers manage the second-by-second allocation of silicon, but Quota Governance manages the month-by-month allocation of organizational intent. Quotas are the interface between engineering constraints and business priorities. In mature ML organizations, these are rarely static; they are adjusted through periodic Quota Review Cycles where allocations are recalibrated based on realized utilization rather than forecasted demand. A team that consistently uses only 40 percent of their 128-GPU allocation is effectively blocking other teams from launching experiments, creating a phantom scarcity that drives up queue times despite ample physical capacity.

The central tension in quota management is the Hoarding Problem. Engineering teams, rational in their desire to minimize wait times, often request maximum capacity “just in case” a project accelerates. This leads to low average utilization and high burst demand. Two ML-specific patterns amplify this behavior beyond generic resource anxiety. The first is the RLHF evaluation phase, in which the GPU nodes allocated to the policy model sit largely idle while the team awaits scores from human annotators. The team retains the reservation to avoid queuing again once annotation batches arrive, but the actual GPU occupancy during the annotation window can fall below 10 percent. The second pattern occurs when a team reserves a GPU block for a training run whose preprocessing pipeline has not yet completed. CPU-heavy data tokenization and vocabulary normalization for large corpora routinely takes longer than the team estimated; the GPUs sit reserved and idle while the tokenizer processes the dataset, because releasing the reservation risks losing it to a competing team. Both scenarios are rational from the team’s perspective but represent systematic waste at the cluster level. To counter this, platform teams implement Utilization-Based Reclamation: if a quota pool remains underutilized for a set period, the system automatically reduces the allocation, returning resources to the general pool. This technical enforcement shifts the burden of proof back to the team to justify their reservation.

Financial accountability further aligns incentives. Organizations typically begin with showback, a reporting mechanism that exposes the dollar cost of GPU usage to engineering managers without directly impacting their budgets. This fosters awareness but lacks teeth. As spending scales, organizations transition to chargeback, where compute costs are deducted from team budgets. Chargeback creates immediate pressure to optimize code and release unused quota, though it risks discouraging speculative research if the internal pricing model is too aggressive.

A concrete quota scenario shows how quickly idle reservations become expensive.

Example 1.3: Quota hoarding

Scenario: A research team reserves a static block of 500 GPUs for an upcoming model refresh.

Diagnosis: Upstream data delays postpone training, but the team retains the reservation to ensure future availability. The GPUs sit idle for three weeks, accumulating $504,000 in capital waste while other teams’ queues back up.

Systems lesson: Static quota reservations cause severe capacity hoarding. Fleet orchestrators must implement dynamic “use it or lose it” reclamation policies, automatically demoting idle reserved nodes (<20 percent utilization for 72h) to preemptible spot pools.

The “use it or lose it” reclamation policy addresses the most visible symptom of quota hoarding, but it raises a second-order question: when idle capacity is lent to another team, the system must distinguish borrowed capacity (preemptible on short notice) from guaranteed reservations (protected by SLOs). Defining borrowing, preemption, and guaranteed tiers as first-class policy primitives is the core multi-tenancy design decision.

Checkpoint 1.4: Multi-tenancy design decisions

Consider a 2,000-GPU cluster shared between a research team (60 percent allocation) and a production team (40 percent allocation):

• Idle reclamation: decide whether the production team can use the idle 30 percent of the cluster, and describe the preemption behavior when the research team submits a large job.
• Priority classes: configure priority classes for a production inference workload that needs 100 GPUs with guaranteed latency SLOs and a research training job that can tolerate preemption.
• Over-commitment ratio: determine the appropriate over-commitment ratio when research jobs average 65 percent GPU utilization and production jobs average 85 percent.

Even with sophisticated multi-tenancy and quota policies in place, the cluster can still fall victim to complex, systemic bottlenecks that defy simple monitoring. When utilization drops inexplicably despite full queues, operators must move from dashboards to systematic utilization debugging.

Self-Check: Question

1. A cluster monitoring dashboard reports \(85\%\) GPU utilization, yet researchers complain of slow progress and low experiment output. What distinction between fleet utilization metrics explains this apparent contradiction?

   1. Allocated utilization (GPUs reserved by containers) may be high (\(85\%\)) while compute utilization (SM active time) is low (\(20\%\)) due to data loader stalls, or productive utilization (goodput advancing converged training steps) is near zero due to unhandled AllReduce synchronization deadlocks.
   2. Hardware sensors on GPUs report utilization in hexadecimal rather than decimal percentages.
   3. Grafana dashboards invert metric percentages when cluster temperature exceeds 25 degrees Celsius.
   4. GPU utilization metrics only count CPU core cycles, concealing whether the GPU is plugged into power.

2. What is a ‘preemption cascade’ in a multi-tenant GPU cluster, and how can orchestrators mitigate its destructive impact on cluster throughput?

   1. A preemption cascade occurs when a single GPU catches fire and burns adjacent GPUs in the same server rack; it is mitigated by Halon fire suppression systems.
   2. A preemption cascade occurs when Slurm automatically deletes user source code repositories upon job completion; it is mitigated by Git branch protection.
   3. A preemption cascade occurs when Kubernetes continuously restarts healthy pods due to Docker daemon upgrades; it is mitigated by disabling Linux kernel updates.
   4. A preemption cascade occurs when high-priority jobs evict medium-priority jobs, which in turn evict low-priority jobs across the cluster, triggering simultaneous checkpoint I/O storms and thrashing compute progress; it is mitigated by preemption cooldown periods and holistic preemption limits.

3. In a multi-tenant cluster, how does decoupling ‘guaranteed quota’ from ‘limit (burst) quota’ prevent stranded idle capacity while protecting team allocations?

4. What is ‘quota hoarding’ in shared GPU clusters, and what automated scheduler policies prevent researchers from engaging in it?

5. True or False: Under Dominant Resource Fairness (DRF), if Team A’s workload requires \(4\text{ GPUs}\) and \(8\text{ CPU cores}\) per worker while Team B’s workload requires \(1\text{ GPU}\) and \(64\text{ CPU cores}\) per worker on a cluster with \(100\text{ GPUs}\) and \(1000\text{ CPU cores}\), DRF equalizes each team’s share of their dominant resource (GPUs for Team A, CPUs for Team B).

6. Order the hierarchy of checks performed by a multi-tenant scheduler when Team A in Department X submits a new 16-GPU job request:

1. Evaluate Team A’s internal user priority and historical fair-share usage decay relative to teammate submissions.
2. Determine if cluster-wide unallocated capacity or preemption candidates exist to host the 16-GPU allocation.
3. Verify that Department X has not exceeded its total department-level GPU budget limit.
4. Check whether Team A’s currently active allocations are within its guaranteed quota or if it must request burst/borrowed capacity.
5. Grant the lease, bind the containers to physical hardware, and update accounting meters for Department X and Team A.

See Answers →

Debugging Cluster Utilization

A cluster dashboard shows average GPU utilization stuck at 60 percent, yet developers complain about three-day queue times for small debugging jobs. Hundreds of GPUs sit idle, but no jobs are starting. Bridging this gap between theoretical capacity and production reality requires methodical forensic analysis to untangle the interactions between hardware topology, scheduler policies, and user behavior. Operating above 80 percent utilization causes queue wait times to explode nonlinearly (figure 6).

Figure 6: The Utilization Paradox: As cluster utilization approaches saturation (100 percent), job wait times increase hyperbolically and diverge near saturation, not linearly. Operating above 80 percent utilization (the Danger Zone) causes unpredictable scheduling delays, especially for large jobs, illustrating why 100 percent utilization is an anti-pattern for interactive research clusters. A dashed line shows target utilization at 75 percent.

The debugging method is to compare scheduler state with physical capability, inspect job requests against the actual hardware pools, and then test for a policy-infrastructure mismatch. A full queue and idle accelerators can coexist when the scheduler is honoring constraints that the dashboard has aggregated away. The cluster at the center of example 1.4 runs the heterogeneous hardware in table 8, accumulated across three procurement cycles.

Table 8: Cluster hardware inventory: Three procurement generations create distinct resource pools with different capabilities. A100 nodes support tensor parallelism via NVLink, while V100 nodes are limited to data parallelism.

GPU Type	Count	Memory	Interconnect	Nodes
A100-80 GB	400 GPUs	80 GB HBM2e	NVLink (600 GB/s)	50 nodes \(\times\) 8 GPUs
A100-40 GB	424 GPUs	40 GB HBM2e	NVLink (600 GB/s)	53 nodes \(\times\) 8 GPUs
V100-32 GB	200 GPUs	32 GB HBM2	PCIe Gen3 (15.8 GB/s)	50 nodes \(\times\) 4 GPUs

Example 1.4: Debugging low GPU utilization

Scenario: A 1,024-GPU cluster reports 60 percent average utilization despite over 50 pending jobs in queue.

Diagnosis: Over-specification (users copying A100-80 GB templates for 35 GB memory jobs), pool fragmentation, and legacy V100 avoidance leave A100-40 GB and V100 nodes idle while large jobs wait for A100-80 GB allocations. The scheduler correctly enforced policies written for a homogeneous fleet that no longer matched the heterogeneous infrastructure.

Systems lesson: Surface-level diagnosis often mistakes policy-infrastructure mismatches for scheduling algorithm bugs. Updating distribution layer policies to support automatic capability matching, dynamic memory profiling, and backfilling raises cluster utilization to 84 percent, recovering $7M in annual compute capacity.

Table 9: Utilization recovery results: The distinction between “allocated” and “effective” utilization captures the policy-induced utilization gap: before the fix, A100-80 GB GPUs showed high allocation in Slurm but lower actual compute utilization, while strict job templates stranded other capable hardware pools. The A100-80 GB “effective” values are measured utilization of allocated GPUs; cluster-wide utilization is computed over the full heterogeneous fleet.

Pool	Before	After	Change
A100-80 GB	95% allocated, 72% effective	89% allocated, 94% effective	+22 pp
A100-40 GB	64% allocated	88% allocated	+24 pp
V100	0% allocated	71% allocated	+71 pp
Cluster-wide	60%	84%	+24 pp

This utilization gain adds approximately 246 fully utilized GPUs of productive work (table 9). That is equivalent to buying 410 additional raw GPUs if those GPUs ran at the old 60 percent utilization. At $2/GPU-hour, the equivalent raw capacity represents approximately $7M in annual recovered value, achieved through policy changes requiring zero additional hardware investment.

Common utilization anti-patterns

High-level metrics often mask deep inefficiencies. A cluster dashboard may report 85 percent allocation, but productive utilization, the share of cycles actually advancing model state, can be significantly lower. Debugging these gaps requires identifying specific signatures in the GPU telemetry that reveal the underlying bottleneck.

The zombie job represents the most egregious waste: a training process that has crashed or deadlocked but continues to hold GPU resources. In a distributed run training a 175B parameter model, a single rank failure can leave 128 GPUs allocated but idling if the process cleanup fails. The signature is high memory allocation (greater than 300 GB) paired with near-zero SM utilization (less than 5 percent) persisting for longer than a heartbeat timeout (typically 10 minutes). This state often results from CUDA context corruption or a driver-level deadlock that prevents the container runtime from successfully reaping the process, requiring forceful node-level remediation to reclaim the device.

The data starvation pattern manifests as a high-frequency sawtooth wave in GPU utilization, where the device oscillates between 100 percent load (forward/backward pass) and 0 percent load (waiting for the next batch). While the average utilization might appear acceptable at 60 to 70 percent, the GPU is effectively idling for a third of its life. This occurs when the CPU-bound data pipeline cannot sustain the throughput required by the GPU, a common scenario in computer vision where JPEG decoding and augmentation saturate host CPUs. A utilization variance exceeding 30 percent standard deviation is the primary diagnostic indicator, signaling the need for active prefetching, increased loader parallelism, or local NVMe caching to saturate the accelerator.

The communication bottleneck reveals itself through periodic, synchronized drops in SM utilization across all ranks in a distributed group. For the 175B model, synchronizing gradients for 350 GB of parameters requires massive bandwidth; if the network is undersized or congested, the GPUs spend more time waiting for AllReduce to complete than performing computation. The diagnostic metric is the step time ratio. If total iteration time exceeds 1.5\(\times\) compute-only time (forward plus backward passes in isolation), non-compute overhead exceeds 0.5\(\times\) compute time, which is material but not dominant. Under the simplified decomposition \(T_{\text{total}} = T_{\text{compute}} + T_{\text{comm}}\), communication exceeds computation only when the ratio is greater than 2\(\times\); profiling must also separate communication from input and runtime overhead. Remediation requires topology-aware placement to keep traffic on high-bandwidth switch tiers or algorithmic changes like gradient accumulation to increase the compute-to-communication ratio.

The memory fragmentation trap is an insidious failure mode in long-running inference servers. A 175B model serving diverse request lengths can end up with 80 percent of its HBM allocated to the KV cache, yet be unable to accept a new request because the free memory is scattered in small, noncontiguous blocks. The GPU appears full to the scheduler but underutilized in terms of throughput. This resembles the classic heap fragmentation problem but with higher stakes: a single restart to defragment dumps hundreds of gigabytes of state. Monitoring the ratio of requested token slots to allocated memory bytes often exposes this gap, which paged cache allocators address by decoupling logical sequence contiguity from physical memory layout, recovering usable memory and boosting serving throughput by 2–4\(\times\) without adding hardware.

The debugging example illustrates a broader principle that applies to every system across the fleet stack: scheduling systems are only as effective as the policies they implement, and policies must evolve alongside infrastructure. When infrastructure changes (new hardware generations are added, network topologies are upgraded, workload mixes shift from training-dominant to serving-dominant), the policies encoded in scheduler configuration, job templates, and organizational practices must be re-evaluated and updated. The most expensive scheduling bug is often not a software defect but a policy that was correct for the old infrastructure and was never updated for the new one.

This principle connects scheduling mechanics to fleet governance. Technical policies (GPU type matching, gang scheduling timeouts, preemption grace periods) interact with organizational policies (team quotas, priority hierarchies, cost allocation) and human behavior (job template reuse, resource request habits, queue submission patterns). Effective fleet orchestration requires attending to all three layers simultaneously.

Self-Check: Question

1. An administrator observes that average GPU compute utilization across a 1,000-GPU cluster is only \(55\%\), yet users experience multi-day queue wait times for 64-GPU jobs. Which diagnostic finding points to the primary root cause?

   1. The cluster’s Ethernet cables are physically too long, causing optical signals to decay into radio waves.
   2. Severe resource fragmentation and strict gang scheduling constraints prevent 64-GPU contiguous blocks from forming, while short jobs hold scattered GPUs across nodes without filling them.
   3. The NVIDIA driver automatically caps utilization at 55% when GPU fans are running on low speed.
   4. Python code running in user containers automatically halts execution whenever cluster utilization drops below 60%.

2. During a multi-GPU training job, DCGM reports that GPU SM utilization oscillates rapidly between \(10\%\) and \(95\%\) with low memory copy throughput, while top on the host node shows \(100\%\) CPU utilization across all cores and high disk wait (\(iowait\)). What is the root cause and remedy?

   1. The GPU hardware is defective; the operator should replace the GPUs with CPU-only nodes.
   2. The PyTorch model has too many layers; the model architecture must be pruned to 1 layer.
   3. The training job is bottlenecked on host data loading, decompression, or augmentation; the remedy is optimizing PyTorch num_workers, utilizing shared memory (/dev/shm), caching preprocessed tensors on local NVMe SSDs, or offloading transforms to GPU using DALI.
   4. The cluster InfiniBand switch is operating in half-duplex mode; the administrator must power cycle the spine switch.

3. What is a ‘zombie allocation’ (or ghost job) in a GPU cluster, and how can automated health monitors detect and remediate it?

4. True or False: In a ring AllReduce across 32 nodes, if 31 nodes show \(95\%\) GPU utilization and 1 node shows \(10\%\) utilization, the node with \(10\%\) utilization is the only healthy node while the 31 nodes are encountering network hardware faults.

5. When diagnosing training performance bottlenecks, operators inspect the ____ throughput (measured in GB/s) to ensure that batches transferred from host memory to accelerator HBM are not constrained by host bus contention.

See Answers →

Fallacies and Pitfalls

Those policy-infrastructure mismatches recur as a small set of misconceptions that can make an apparently healthy scheduler waste expensive fleet capacity. It is tempting to look at a cluster running at 98 percent utilization and declare victory, only to discover that researchers have stopped submitting jobs entirely because wait times have stretched into weeks. Cluster orchestration is riddled with counterintuitive traps like this, where optimizing a single metric destroys overall system value.

Fallacy: More sophisticated scheduling algorithms always improve utilization.

Engineers facing low utilization often reach for more advanced schedulers, assuming the algorithm is the bottleneck. As the debugging example in section 1.8 demonstrates, the root cause is frequently policy misconfiguration, not algorithmic limitation. A simple FIFO scheduler with correct policies (capability-based matching, backfill with timeouts, appropriate gang scheduling constraints) often outperforms a sophisticated scheduler with incorrect policies. Before upgrading the scheduler, audit the policies: confirm that users are not requesting resources they do not need, that heterogeneous resources are properly exposed, and that gang-scheduling timeouts are configured.

Pitfall: Treating all GPU-hours as equal when measuring utilization.

Cluster dashboards commonly report “GPU utilization” as a single percentage, averaging across all GPUs and all workloads. This metric hides critical information. A cluster might report 80 percent utilization where 60 percent is productive training, 15 percent is idle GPUs allocated to jobs waiting for gang completion, and 5 percent is GPUs running data loading with no active computation. Effective monitoring distinguishes between allocated utilization (GPUs assigned to jobs), compute utilization (GPUs executing kernels), and productive utilization (GPUs advancing useful training or serving requests). Only the last metric correlates with actual value delivered.

Fallacy: Gang scheduling is always necessary for distributed training.

Gang scheduling prevents deadlock for synchronous training, but not all distributed training is synchronous. Asynchronous training methods tolerate worker arrivals and departures, and elastic training frameworks handle variable worker counts. For workloads that can operate asynchronously or elastically, relaxing the gang scheduling requirement dramatically improves scheduling flexibility and reduces queue wait times. The trade-off is potential convergence degradation from stale gradients or batch size variability, but for many practical workloads (hyperparameter sweeps, fine-tuning, pretraining with adaptive batch size), this trade-off is favorable.

Pitfall: Setting static quotas based on peak demand.

Organizations commonly set team quotas to handle worst-case demand, reasoning that teams need guaranteed access during crunch periods. If Team A’s peak demand is 500 GPUs and Team B’s is 300 GPUs, the cluster needs 800 GPUs with static quotas. In practice, peaks rarely coincide. Hierarchical fair-share with borrowing can serve both teams’ peak demands with a 600-GPU cluster, because when Team A peaks, Team B is typically at moderate demand and can yield borrowed capacity. Static quotas at peak levels waste 25 to 40 percent of cluster capacity on guaranteed-but-unused allocations.

Fallacy: Spot instances are always cheaper for training.

The 60 to 90 percent discount on spot instances creates the impression of automatic savings. The effective cost depends on interruption frequency, checkpoint overhead, and restart time. For a large model with 15-minute checkpoint times and 30-minute restart times, a spot interruption costs 45 minutes of productive time. If interruptions occur every 2 hours, the effective cost rises enough that the nominal 65 percent discount shrinks to about 52 percent savings. For 100B+ parameter models with slow checkpointing on clusters with frequent spot interruptions, on-demand instances can actually be cheaper when accounting for all overhead. The decision requires quantitative analysis of the specific workload and spot market conditions, not blanket assumptions about cost savings.

Pitfall: Ignoring topology when scheduling distributed training.

Schedulers that treat GPUs as interchangeable units create allocations where tensor parallel groups span nodes (forcing NVLink-speed operations over InfiniBand) or AllReduce groups cross spine switches (adding latency and congestion). The 15 to 30 percent throughput penalty from poor topology placement accumulates over multi-week training runs, potentially wasting more resources than would be lost by waiting for a topology-aware allocation. Topology-aware scheduling increases scheduling complexity and may reduce packing efficiency, but for large distributed training jobs, the throughput improvement often justifies the trade-off.

Fallacy: GPU utilization alone is a sufficient autoscaling signal.

GPU utilization is a poor proxy for inference health. A GPU can be 90 percent utilized but still violating latency SLOs because the utilization comes from processing a backlog of queued requests. Conversely, a GPU at 40 percent utilization might be perfectly healthy if it is serving low-latency requests with headroom for bursts. The right metrics are queue depth, P99 latency relative to SLO, and KV cache memory pressure for LLM workloads. Relying on utilization creates a lagging indicator that only triggers scaling after performance has already degraded, whereas queue depth and SLO margin provide leading indicators that allow the fleet to scale before the user experience suffers. Furthermore, for LLMs, memory exhaustion from KV cache growth can occur at low compute utilization, causing requests to stall or fail. An autoscaler watching only compute utilization will miss this failure mode entirely, leaving the service degraded despite apparent capacity.

Pitfall: Using elastic training to avoid gang-scheduling decisions.

Elastic training complements but does not replace gang scheduling. While it allows jobs to resize, the resizing steps themselves often have atomic requirements. Elastic training works for data parallelism but not tensor parallelism (which requires a fixed number of GPUs per group). A 175B model using 8-way tensor parallelism needs exactly 8 GPUs per tensor-parallel group to function; this is a gang constraint that elastic training cannot relax. Elastic training adjusts the number of data-parallel replicas, not the intra-replica parallelism configuration, meaning the scheduler must still enforce gang constraints for the base unit of the model. If a scheduler naively places these 8 GPUs across different racks or fails to allocate them atomically, the tensor parallel group will either fail to initialize or suffer catastrophic performance degradation.

Fallacy: The scheduler needs no workload-specific validation.

Scheduler configuration requires continuous tuning as workloads evolve. Default configurations in Slurm or Kubernetes are optimized for generic workloads, often prioritizing simple fairness over throughput. ML-specific tuning (gang scheduling timeouts, topology-aware placement weights, preemption grace periods, and fair-share decay rates) can improve utilization by 15 to 25 percent. Organizations that deploy a scheduler and never revisit its configuration leave significant value on the table. A default preemption grace period might be too short for a large model to checkpoint, causing wasted work, while an untuned topology weight might fragment the cluster unnecessarily. As the cluster grows and the workload mix shifts (for example, from training-heavy to inference-heavy), the effective scheduling parameters shift with it. Treating the scheduler as a set-and-forget appliance lets the fleet drift into inefficiency.

Pitfall: Treating high utilization as proof of cluster health.

A cluster running at 95 percent utilization sounds efficient but may indicate a problem: insufficient capacity that is choking experimentation velocity. When utilization consistently exceeds 85 to 90 percent, queue times grow rapidly following the well-known M/M/1 queuing result from stochastic process theory. In this model, average wait time is proportional to \(\rho/(1-\rho)\) where \(\rho\) is utilization; at 90 percent utilization, average wait is 9\(\times\) the service time, and at 95 percent, it is 19\(\times\). Researchers waiting hours or days for resources conduct fewer experiments, test fewer hypotheses, and iterate more slowly on model designs. This reduced experimentation velocity has real costs that are harder to quantify than GPU-hours but may be larger in aggregate. The effective utilization target balances resource efficiency against researcher productivity, often landing between 70 and 85 percent for training clusters where queue responsiveness directly affects research output. Inference clusters, where requests are served immediately or not at all, often target even lower utilization (50 to 70 percent) to maintain latency headroom for traffic spikes.

Recognizing these fallacies prevents platform teams from chasing false optimizations that look good on dashboards but degrade developer velocity. The core orchestration principles that successfully navigate these trade-offs share a common thread: they treat scheduling as an engineering discipline, not an administrative function.

Self-Check: Question

1. Why is targeting near \(100\%\) GPU cluster utilization an operational anti-pattern for an interactive AI research and development organization?

   1. GPU hardware components melt if utilization exceeds \(90\%\) for more than 5 consecutive minutes.
   2. Modern deep learning frameworks automatically downgrade model accuracy when cluster utilization exceeds \(85\%\).
   3. Operating above \(95\%\) utilization causes cloud providers to double their hourly billing rates without warning.
   4. As utilization approaches \(100\%\), queuing theory (Pollaczek-Khinchine) dictates that job queue wait times explode hyperbolically, delaying fast feedback for exploratory experiments and severely reducing overall engineering velocity.

2. Explain the fallacy that ‘more sophisticated scheduling algorithms always improve utilization,’ and describe what operators should audit before replacing their cluster scheduler.

3. True or False: Gang scheduling is strictly mandatory for every distributed ML workload, including asynchronous parameter-server training, hyperparameter sweeps, and elastic fine-tuning jobs.

See Answers →

Summary

Fleet orchestration transforms raw data center capacity into productive ML infrastructure by governing how silicon, bandwidth, and state interact across time and space. In terms of the iron law of performance (\(T \approx D_{\text{vol}}/\text{BW} + O/(R_{\text{peak}} \cdot \eta_{\text{hw}}) + L_{\text{lat}}\)), the orchestrator is the system’s central optimizer: placement strategies preserve fabric bandwidth (\(\text{BW}\)), multi-tenancy and quota policies drive realized hardware efficiency (\(\eta_{\text{hw}}\)), and distributed scheduling algorithms bound queue delay and coordination overhead (\(L_{\text{lat}}\)). The difference between a high-performing cluster and thousands of idle accelerators is not hardware capability, but the scheduler’s ability to navigate the trade-offs between atomic consistency, fault tolerance, and placement topology under partial failures and bursty demand.

The chapter’s unifying technical insight is that placement directly dictates physical performance. By aligning model parallelism with the physical network hierarchy, topology-aware scheduling prevents cross-rack bisection bottlenecks, protecting the \(D_{\text{vol}}/\text{BW}\) communication term and increasing throughput by 15 to 30 percent. Domain-specific schedulers—such as Tiresias, Gandiva, Themis, and Pollux—exploit the structure of ML workloads (iterative progress, heavy-tailed runtimes, and predictable convergence scaling) to minimize the coordination and wait-time terms (\(L_{\text{lat}}\)), cutting average job completion time by 40 to 60 percent compared to general-purpose FIFO policies. Elastic training and spot management extend this control by trading worker continuity for economic flexibility, provided jobs can dynamically adapt without incurring unacceptable preemption tax.

At serving time, the scheduling objective shifts from throughput maximization to tail-latency protection. Serving fleets cannot trade queue wait for utilization; autoscaling must respond to leading indicators—queue depth, P99 latency margin, and KV-cache memory pressure—rather than compute utilization alone, which misses memory-bound stalls. Meanwhile, multi-tenant isolation mechanisms (MIG, MPS, time-slicing) and fair-share quota governance prevent noisy-neighbor interference and quota hoarding, ensuring that high aggregate allocation translates directly into high effective hardware efficiency (\(\eta_{\text{hw}}\)).

Ultimately, fleet orchestration is diagnostic and economic as much as algorithmic. Operational pathologies such as zombie jobs, data starvation, topology mismatch, and pool fragmentation are symptoms of policy-infrastructure misalignment rather than deficient hardware. When compute nodes are defined, networks wired, storage attached, fault tolerance integrated, and orchestrators aligned, the fleet transitions from a collection of isolated servers into a coherent, self-healing computing platform.

Key Takeaways: Scheduling is systems engineering

• Distributed scheduling is fundamentally hard: Cluster scheduling faces challenges (partial failures, network partitions, state inconsistency) that single-machine schedulers never encounter. The CAP theorem forces trade-offs between consistency and availability that shape every scheduling system’s design.
• Gang scheduling prevents deadlock but reduces flexibility: Gang scheduling gives distributed training atomic resource allocation to prevent hold-and-wait deadlocks, but rigid gang scheduling wastes resources when combined with backfill timeouts and elastic training alternatives.
• Topology determines performance: Where GPUs are placed within the cluster hierarchy matters as much as how many GPUs a job receives. NVLink vs. InfiniBand placement decisions can create 15 to 30 percent throughput differences for the same job on the same hardware.
• ML-specific scheduling outperforms generic approaches: Exploiting workload characteristics (predictable resource needs, iterative computation, diminishing returns) enables 40 to 60 percent improvements in job completion time compared to general-purpose FIFO or fair-share policies.
• Utilization is a first-order economic driver: Improving cluster utilization from 50 percent to 80 percent on a large cluster effectively adds thousands of GPUs worth of capacity. Closing this gap through policy and algorithm work pays back faster than buying additional accelerators.
• Policies must evolve with infrastructure: Scheduling algorithms are only as effective as the policies they implement. When infrastructure changes (heterogeneous hardware additions, topology upgrades, workload mix shifts), policies must be re-evaluated and updated (principle 13).

The fleet now has compute, communication, storage, and the resilience to keep them running. It still needs a policy that decides what runs where. A scheduler that chases peak utilization can deadlock when large jobs compete for the same nodes, while a scheduler that guarantees progress may leave some capacity deliberately unused. The gap between theoretical and safely sustainable throughput is the price of coordination. At fleet scale, performance therefore depends not only on hardware speed but also on the policy that allocates it.

What’s Next: From orchestration to optimization

With distributed execution established in Distributed Training and fault-tolerant checkpointing in Fault Tolerance, multi-tenant resource management completes the operational foundation of the machine learning fleet. With the orchestrator placing workloads on the right hardware tiers, the next challenge is maximizing single-node and intra-step efficiency. Performance Engineering examines operator fusion, precision engineering, graph compilation, and execution scheduling. Orchestration allocates the right resources; performance engineering extracts peak throughput from them.

Self-Check: Question

1. Which statement best synthesizes the central architectural challenge of fleet orchestration for machine learning?

   1. Orchestration is purely a software compilation task that translates PyTorch code into C++ CUDA binaries without considering cluster hardware.
   2. Orchestration requires balancing physical hardware constraints (NVLink topology, failure rates, power limits) against organizational policies (fairness, priorities, cost) across diverse, long-running, and failure-prone distributed workloads.
   3. Orchestration has been completely solved by standard web application load balancers like NGINX, which handle GPU AllReduce synchronization transparently.
   4. Orchestration only matters for single-GPU laptops and has no relevance to multi-thousand GPU datacenters.

2. How is the historical boundary between HPC batch scheduling (Slurm) and cloud-native service orchestration (Kubernetes) evolving to meet the demands of modern ML lifecycles?

3. True or False: In large-scale ML cluster management, maximizing raw GPU allocated utilization (the percentage of GPUs assigned to active containers) guarantees optimal economic value and research productivity.

See Answers →

Self-Check Answers

Self-Check: Answer

1. What is the fundamental difference in control philosophy between Slurm and Kubernetes when managing distributed ML training workloads?

   1. Slurm relies on continuous reconciliation loops against desired state stored in etcd, whereas Kubernetes executes imperative one-time allocation scripts via a centralized daemon.
   2. Slurm uses peer-to-peer gossip protocols to schedule tasks across nodes, whereas Kubernetes enforces hierarchical master-slave token ring passing.
   3. Slurm employs an imperative allocation model where users request explicit physical resources guaranteed at launch, whereas Kubernetes uses a declarative model where controllers continuously reconcile observed cluster state toward declared desired state.
   4. Slurm schedules containerized microservices with dynamic horizontal auto-scaling, whereas Kubernetes only supports uncontainerized bare-metal MPI processes.

   Answer: The correct answer is C. Slurm originates from HPC batch systems and uses an imperative model (sbatch requests explicit resources like --gres=gpu:8 --nodes=4), providing authoritative upfront allocation. Kubernetes uses a declarative model where controllers continuously observe actual state and execute reconciliation loops to converge toward desired state (e.g., Pods, Deployments). The option swapping Slurm and Kubernetes mechanisms reverses their core architectures. The gossip protocol alternative invents a non-existent scheduling model for Slurm. The containerization claim reverses the platforms’ primary native workload types.

   Learning Objective: Compare the core architectural control mechanisms of Slurm and Kubernetes for ML training workflows.

2. How does NVIDIA Multi-Instance GPU (MIG) partitioning on A100/H100 accelerators differ from software-based GPU time-slicing in multi-tenant serving environments?

   1. MIG uses OS-level software schedulers to alternate kernel execution times, but shares the same L2 cache and memory controllers dynamically.
   2. MIG compiles user PyTorch models into specialized FPGA bitstreams, preventing simultaneous execution of untrusted kernels.
   3. MIG allows dynamic resizing of GPU partitions on a millisecond basis without draining running containers or resetting the device.
   4. MIG partitions the physical accelerator into hardware-isolated instances with dedicated streaming multiprocessors (SMs), memory controllers, and L2 cache slices, eliminating noisy-neighbor interference at the cost of static partition reconfiguration.

   Answer: The correct answer is D. MIG provides true hardware-level partitioning (up to 7 instances on A100/H100) with dedicated SMs, memory controllers, and private L2 cache slices, preventing memory bus contention and noisy-neighbor performance degradation. Its primary trade-off is reconfiguration rigidity: changing MIG profile configurations requires draining all active workloads on the physical GPU. The software-scheduling description applies to standard time-slicing or MPS rather than MIG. The FPGA compilation claim is factually false. The millisecond dynamic resizing contradicts MIG’s static partitioning constraints.

   Learning Objective: Evaluate Multi-Instance GPU (MIG) architectural characteristics and trade-offs for multi-tenant inference.

3. In Slurm’s fair-share priority formula (\(\text{Priority}_{\text{effective}} = \text{Priority}_{\text{base}} \times \frac{F_{\text{target}}}{F_{\text{actual}} + \epsilon}\)), what role does the usage decay half-life play, and what is the trade-off between setting a short versus a long half-life?

   Answer: The half-life determines how quickly the scheduler discounts past GPU consumption in \(F_{\text{actual}}\), allowing heavy users to gradually restore their priority. A short half-life (e.g., 1-2 days) rapidly restores priority but can cause oscillatory usage where teams cycle between hogging and starvation; a long half-life (e.g., 2-4 weeks) stabilizes priority order but penalizes researchers long after a major burst project has completed.

   Learning Objective: Explain the mechanics and purpose of usage half-life decay in Slurm’s fair-share scheduling formula.

4. Why do high-performance distributed training jobs in Kubernetes typically enable hostNetwork: true and leverage the NVIDIA Network Operator instead of relying on default Container Network Interface (CNI) overlay networks?

   Answer: Standard CNI overlay networks (such as VXLAN or Calico) introduce packet encapsulation overhead, CPU switching latency, and lack direct integration with hardware RDMA subsystems. Enabling hostNetwork: true allows worker pods to bypass container network namespaces, enabling direct access to host InfiniBand/RoCE network adapters and GPUDirect RDMA for full-bandwidth, sub-microsecond AllReduce collective communication.

   Learning Objective: Justify why large-scale distributed training jobs in Kubernetes require host networking rather than standard container overlay networks.

5. True or False: Kueue replaces the native Kubernetes kube-scheduler binary entirely to implement custom gang scheduling algorithms, whereas Volcano acts purely as an admission controller layered on top of the default scheduler.

   Answer: False. The roles are reversed: Volcano is an alternative scheduler that replaces kube-scheduler to provide native PodGroup-based gang scheduling, whereas Kueue is an admission controller that manages when jobs enter the cluster (using ClusterQueues and borrowing) while delegating actual pod placement to the default kube-scheduler or specialized placement plugins.

   Learning Objective: Differentiate between scheduler replacement (Volcano) and queue admission control (Kueue) in the Kubernetes batch ecosystem.

6. **Order the following steps in the lifecycle of a distributed training job submitted to a Kubernetes cluster configured with Kubeflow Training Operator and Volcano gang scheduling:

1. The PodGroup is created and Volcano verifies that the cluster can simultaneously satisfy the minimum worker requirement (\(N_{\text{min}}\)).
2. The developer submits a PyTorchJob custom resource manifest defining worker replicas, GPU limits, and rendezvous parameters.
3. Volcano binds all worker pods to specific healthy nodes simultaneously, and kubelets start the training containers with GPUDirect RDMA.
4. Workers establish distributed communication via TorchElastic rendezvous (setting MASTER_ADDR and WORLD_SIZE) and begin synchronized training steps.
5. The Kubeflow Training Operator reconciles the PyTorchJob, creating the individual worker pods and a corresponding Volcano PodGroup.**

Answer: The correct order is (2) -> (5) -> (1) -> (3) -> (4). Explanation: Step (2): The developer submits a PyTorchJob custom resource manifest defining worker replicas, GPU limits, and rendezvous parameters. Step (5): The Kubeflow Training Operator reconciles the PyTorchJob, creating the individual worker pods and a corresponding Volcano PodGroup. Step (1): The PodGroup is created and Volcano verifies that the cluster can simultaneously satisfy the minimum worker requirement (\(N_{\text{min}}\)). Step (3): Volcano binds all worker pods to specific healthy nodes simultaneously, and kubelets start the training containers with GPUDirect RDMA. Step (4): Workers establish distributed communication via TorchElastic rendezvous (setting MASTER_ADDR and WORLD_SIZE) and begin synchronized training steps.

Learning Objective: Order the operational stages of launching a distributed PyTorch training job via the Kubeflow Training Operator and Volcano.

← Back to Questions

Self-Check: Answer

1. When scheduling a 3D-parallel large language model across a multi-rack GPU cluster, how should Tensor Parallelism (TP), Pipeline Parallelism (PP), and Data Parallelism (DP) be mapped across the topology hierarchy to maximize throughput?

   1. Map DP strictly within intra-node NVLink domains, TP across inter-rack spine switches, and PP across rail-aligned network switches.
   2. Map PP strictly within intra-node NVLink domains, DP across Top-of-Rack switches, and TP across multi-hop spine switches.
   3. Map TP strictly within intra-node NVLink domains (highest bandwidth), PP within the same rack or adjacent switches (point-to-point activations), and DP across nodes with rail-aligned network switches.
   4. Map TP, PP, and DP uniformly at random across all available nodes to balance switch thermal loads across the datacenter fabric.

   Answer: The correct answer is C. Tensor Parallelism requires frequent, high-volume AllReduce collectives at every transformer layer, necessitating the extreme bandwidth of intra-node NVLink (450-900 GB/s per direction). Pipeline Parallelism transfers point-to-point activations between adjacent pipeline stages, fitting well within intra-rack or adjacent leaf switches. Data Parallelism performs gradient AllReduce once per step, which can be overlapped with backward compute and optimized across nodes using rail-aligned InfiniBand/RoCE fabrics. Reversing TP and DP places massive layer-by-layer collectives over slow inter-rack links, causing severe throughput collapse. Uniform random mapping ignores bandwidth cliffs and causes massive cross-spine contention.

   Learning Objective: Design an optimal mapping of 3D parallelism dimensions (Tensor, Pipeline, Data) onto a multi-tier cluster topology hierarchy.

2. In a multi-node cluster where binary up/down health probes report all switches healthy but an AllReduce collective experiences a \(30\%\) throughput degradation due to high CRC error rates on a single leaf switch, what is the optimal orchestration response?

   1. Continue running the job indefinitely because a 10-minute checkpoint-and-reschedule migration would cost more compute than a 30% slowdown over a multi-week run.
   2. Rely on the GPU driver to disable NVLink and route all tensor-parallel traffic through host memory over PCIe.
   3. Trigger a performance-aware topology quarantine via taints or drain states, checkpoint the running job, and migrate it to healthy nodes where the break-even time is under an hour.
   4. Terminate the job and permanently de-allocate all 10,000 GPUs in the datacenter until all network transceivers are manually replaced.

   Answer: The correct answer is C. When a ‘limping switch’ introduces high CRC error rates or packet drops, binary health checks pass while collective performance drops sharply. For multi-day or multi-week training runs, taking a 5 to 15 minute pause to checkpoint, drain/taint the degraded switch path, and migrate to healthy topology breaks even in less than an hour against a 20-30% continuous throughput penalty. Ignoring the issue wastes millions of dollars in idle GPU cycles. Disabling NVLink exacerbates the slowdown. De-allocating the entire datacenter is an extreme operational overreaction.

   Learning Objective: Evaluate scheduler diagnostic mechanisms and migration economics for degraded network hardware.

3. In an 8-GPU-per-node cluster with 8 independent network fabric rails, how does rail-aware scheduling place data-parallel ranks, and why does this prevent network switch congestion?

   Answer: Rail-aware scheduling assigns data-parallel rank \(r\) to GPU \(r \bmod 8\) across selected nodes, ensuring that corresponding GPUs communicate strictly over their dedicated, independent network rail switch hierarchy. This isolates each data-parallel collective to a single rail fabric, eliminating cross-rail traffic and switch port contention between parallel communication streams.

   Learning Objective: Explain how rail-optimized scheduling eliminates network congestion during multi-node AllReduce collectives.

4. Why are static topology configuration files (such as Slurm’s topology.conf or static Kubernetes node labels) insufficient on their own for optimal collective placement in large-scale ML clusters?

   Answer: Static files represent nominal design schematics rather than operational reality; they cannot reflect dynamic runtime anomalies such as degraded transceiver link widths, PCIe bus renegotiations, blown spine switch links, or thermal throttling. Collective libraries and orchestrators need dynamic runtime introspection (e.g., inspecting /sys/class/infiniband and NCCL telemetry) to map the ground-truth operational fabric and avoid placing sensitive collectives across degraded links.

   Learning Objective: Compare static topology configuration files with dynamic runtime topology introspection in distributed ML orchestration.

5. **Order the steps executed by a topology-aware scheduler when placing a multi-node distributed training job across a multi-tier datacenter fabric:

1. Filter out candidate nodes that have active taints, hardware ECC alerts, or degraded network link flags.
2. Allocate contiguous GPU ranks within each node to satisfy intra-node NVLink requirements for Tensor Parallel groups.
3. Group nodes into the smallest common network domain (such as a single leaf switch or rack) to minimize switch hops for Pipeline Parallel stages.
4. Receive job submission specifying total GPU count, node granularity, and parallelism dimensions (\(TP, PP, DP\)).
5. Assign Data Parallel ranks to matching GPU rail indices across nodes to ensure non-interfering AllReduce traffic.**

Answer: The correct order is (4) -> (1) -> (3) -> (2) -> (5). Explanation: Step (4): Receive job submission specifying total GPU count, node granularity, and parallelism dimensions (\(TP, PP, DP\)). Step (1): Filter out candidate nodes that have active taints, hardware ECC alerts, or degraded network link flags. Step (3): Group nodes into the smallest common network domain (such as a single leaf switch or rack) to minimize switch hops for Pipeline Parallel stages. Step (2): Allocate contiguous GPU ranks within each node to satisfy intra-node NVLink requirements for Tensor Parallel groups. Step (5): Assign Data Parallel ranks to matching GPU rail indices across nodes to ensure non-interfering AllReduce traffic.

Learning Objective: Order the decision hierarchy used by a topology-aware scheduler to place a multi-node 3D-parallel training job.

6. In cloud orchestration environments such as AWS and GCP, a user specifies a ____ to ensure that allocated virtual machine instances are placed within a single high-bandwidth, low-latency network spine domain rather than scattered across the datacenter.

   Answer: placement group (or Placement Group, cluster placement group). A placement group acts as the topology contract ensuring that distributed instances share maximum bisection bandwidth for collective AllReduce operations.

   Learning Objective: Identify the cloud-native abstraction used to enforce low-latency, high-bandwidth network proximity.

← Back to Questions

Self-Check: Answer

1. When an elastic scheduler resizes a distributed training job from 64 GPUs to 128 GPUs, what trade-off exists between maintaining a constant global batch size versus scaling batch size proportionally with worker count?

   1. Constant global batch size preserves mathematical convergence dynamics but halves per-GPU micro-batch size (potentially lowering hardware compute efficiency or hitting memory limits when shrinking), whereas proportional scaling maintains compute efficiency per GPU but alters optimization trajectory and requires learning rate recalibration.
   2. Constant global batch size requires restarting the model weights from epoch zero, whereas proportional batch size allows continuous gradient descent without learning rate adjustment.
   3. Constant global batch size disables NVLink communication across workers, whereas proportional batch size enables ring AllReduce.
   4. Constant global batch size doubles the learning rate automatically via PyTorch hooks, whereas proportional batch size keeps the learning rate fixed to prevent divergence.

   Answer: The correct answer is A. In elastic training, if global batch size is kept constant (\(B_{\text{global}} = B_{\text{per-worker}} \times N\)), adding workers reduces \(B_{\text{per-worker}}\), which preserves optimization convergence but may reduce GPU compute utilization (or cause OOM when worker count shrinks and \(B_{\text{per-worker}}\) increases). If per-worker batch size is fixed, scaling \(N\) scales \(B_{\text{global}}\), maintaining per-GPU compute intensity but requiring learning rate recalibration (e.g., linear or square-root scaling) to avoid convergence degradation. The other choices present incorrect claims about restarting weights, NVLink, or inverted learning rate rules.

   Learning Objective: Analyze the trade-offs between constant global batch size and adaptive learning rate during elastic worker scaling.

2. An elastic job running on 64 GPUs at 100 steps/minute can scale to 128 GPUs, reaching 180 steps/minute. If resizing triggers a dynamic rendezvous and checkpoint-sync transition pause of 60 seconds (0 steps during pause), what is the minimum duration the job must remain at 128 GPUs to achieve a net gain in total steps completed?

   1. At least 15 seconds.
   2. At least 30 seconds.
   3. At least 45 seconds.
   4. At least 75 seconds.

   Answer: The correct answer is D. In the 60-second (1.0 minute) pause, the 64-GPU baseline would have completed \(100 \text{ steps/min} \times 1.0 \text{ min} = 100 \text{ steps}\). At 128 GPUs, the job produces \(180 \text{ steps/min}\), which is an incremental gain of \(\Delta = 180 - 100 = 80 \text{ steps/min}\). To recover the 100 lost steps, the job must run at 128 GPUs for \(t = 100 / 80 = 1.25 \text{ minutes} = 75 \text{ seconds}\). The choices for 15, 30, and 45 seconds fall well short of recovering the 100 lost steps.

   Learning Objective: Calculate the break-even duration required for an elastic scaling resize to deliver positive net training throughput.

3. What three parameters define the operational contract between an elastic scheduler and an elastic training framework (such as TorchElastic), and why is each necessary?

   Answer: The contract is defined by: (1) Elastic Range \([N_{\text{min}}, N_{\text{max}}]\), specifying the valid worker bounds within which the model can train without OOM or synchronization collapse; (2) Transition Cost, the time required for dynamic rendezvous and state reload, bounding the minimum viable resize interval; and (3) Scaling Efficiency Curve, describing throughput gained per added accelerator to let the scheduler determine whether assigning marginal GPUs is more productive than allocating them elsewhere.

   Learning Objective: Explain the three core parameters of the contract between an elastic scheduler and a training framework.

4. True or False: When high-priority workloads surge on a shared cluster, an elastic scheduler can reclaim capacity from lower-priority distributed training jobs by downsizing them toward their \(N_{\text{min}}\) worker limit rather than terminating and requeuing them, thereby avoiding total progress loss.

   Answer: True. Elastic scheduling transforms the scheduler’s action space from a binary choice of ‘keep all or kill completely’ to continuous resource adjustment. When higher-priority demand arrives, the scheduler signals the elastic job to shrink worker count down toward \(N_{\text{min}}\), allowing it to continue training at reduced throughput while releasing the remaining nodes for the higher-priority job.

   Learning Objective: Compare binary preemption with elastic worker downsizing under sudden cluster contention.

5. **Order the sequence of events that occurs when a worker node fails during an elastic training run managed by TorchElastic:

1. The rendezvous backend (e.g., c10d / etcd) detects the dead worker heartbeat and signals a rendezvous state change to remaining workers.
2. The scheduler detects the unrecoverable node, provisions a replacement node (or authorizes running with reduced workers if within \([N_{\text{min}}, N_{\text{max}}]\)), and joins the new worker.
3. Active workers pause gradient computation at the step boundary and enter the rendezvous barrier.
4. All surviving and new worker ranks establish a new communication world, assign new ranks (\(0 \dots N-1\)), re-initialize process groups, load the latest checkpoint, and resume training.
5. A hardware fault causes an unhandled CUDA kernel crash on worker rank 3, stopping its heartbeats.**

Answer: The correct order is (5) -> (1) -> (3) -> (2) -> (4). Explanation: Step (5): A hardware fault causes an unhandled CUDA kernel crash on worker rank 3, stopping its heartbeats. Step (1): The rendezvous backend (e.g., c10d / etcd) detects the dead worker heartbeat and signals a rendezvous state change to remaining workers. Step (3): Active workers pause gradient computation at the step boundary and enter the rendezvous barrier. Step (2): The scheduler detects the unrecoverable node, provisions a replacement node (or authorizes running with reduced workers if within \([N_{\text{min}}, N_{\text{max}}]\)), and joins the new worker. Step (4): All surviving and new worker ranks establish a new communication world, assign new ranks (\(0 \dots N-1\)), re-initialize process groups, load the latest checkpoint, and resume training.

Learning Objective: Order the operational stages of a dynamic rendezvous event during worker failure and elastic recovery.

← Back to Questions

Self-Check: Answer

1. When training large foundation models on preemptible/spot cloud GPU instances discounted by \(70\%\) compared to on-demand pricing, what operational vulnerability poses the greatest threat to training goodput?

   1. Spot instances limit maximum GPU memory to half of on-demand instances due to hypervisor virtualization overhead.
   2. Spot preemption events within a single Availability Zone or accelerator pool are often highly correlated during market capacity crunches, simultaneously terminating all nodes in a multi-node gang and causing repetitive re-rendezvous overhead unless checkpoint frequency is tuned.
   3. Spot instances cannot communicate over InfiniBand networks and must route all AllReduce collectives through public HTTP REST APIs.
   4. Cloud providers prohibit running PyTorch or TensorFlow frameworks on spot instances under standard acceptable use policies.

   Answer: The correct answer is B. Spot instances offer substantial cost discounts (60-90%) but carry preemption risk. Crucially, spot revocations are not independent random events; during demand surges, cloud providers reclaim entire pools of specific GPU types in a given Availability Zone at once. For multi-node distributed training, losing even a single node halts synchronous training, and frequent correlated preemptions can degrade goodput to zero unless checkpoint intervals and recovery are tightly optimized. Memory limitations, HTTP-only AllReduce, and framework prohibitions are fabricated distractors.

   Learning Objective: Evaluate the cost trade-offs and operational risks of running large-scale distributed training on cloud spot instances.

2. How do Cloud Capacity Blocks (or contiguous reserved GPU instances) differ from standard on-demand and spot instance allocations for large-scale training?

   1. Capacity Blocks guarantee contiguous topological co-location within a single network fabric for a committed time window, eliminating preemption risk and placement fragmentation at the cost of upfront commitment.
   2. Capacity Blocks allow unlimited instantaneous bursting beyond physical cluster limits without financial commitment.
   3. Capacity Blocks dynamically convert idle inference GPUs into spot training nodes with sub-second preemption notices.
   4. Capacity Blocks provide 99% cost discounts by sharing GPU streaming multiprocessors across unrelated cloud tenants.

   Answer: The correct answer is A. Capacity reservations / Capacity Blocks allow organizations to reserve contiguous blocks of GPU instances (e.g., hundreds or thousands of H100s connected via high-bandwidth fabrics) for specific project durations (days to months). This guarantees physical co-location and eliminates both the risk of spot preemption and the risk of on-demand capacity unavailability / fragmentation, in exchange for guaranteed utilization commitment. The bursting, dynamic inference conversion, and multi-tenant SM sharing distractors mischaracterize capacity reservation mechanisms.

   Learning Objective: Compare capacity reservation mechanisms (such as Cloud Capacity Blocks) with spot and on-demand purchasing models.

3. Beyond raw accelerator capital expenditure (CapEx) or hourly cloud VM rates, what major operational and infrastructural factors constitute the Total Cost of Ownership (TCO) of an ML training cluster?

   Answer: Fleet TCO encompasses: (1) Datacenter power and cooling infrastructure (often exceeding \(1-2 \text{ kW}\) per node); (2) High-radix network fabric switches and optical cabling (InfiniBand/RoCE); (3) High-throughput storage arrays and metadata servers for dataset caching and checkpointing; (4) Network egress and cross-AZ data transfer fees; (5) Control-plane management overhead; and (6) Idle capacity costs resulting from scheduling fragmentation and queue wait times.

   Learning Objective: Analyze the non-hardware cost factors that contribute to the Total Cost of Ownership of GPU training fleets.

4. In a distributed training architecture with parameter servers or a central coordinator/master node alongside data-parallel workers, how should roles be partitioned between on-demand and spot instances to optimize cost while maintaining system stability?

   Answer: Place the central coordinator/master node, checkpoint metadata store, and primary parameter server on reliable on-demand instances to ensure the control plane never loses state or crashes unexpectedly; allocate elastic, stateless data-parallel workers on discounted spot instances, enabling workers to be preempted and dynamically re-rendezvoused without tearing down the entire training job session.

   Learning Objective: Design a fault-tolerant hybrid spot/on-demand placement strategy for distributed training architectures.

5. True or False: Because each VM instance in a cloud provider has an independent lifecycle, placing worker nodes of a 64-node distributed training job across different instance types within the same Availability Zone guarantees that preemption events will remain uncorrelated and statistically independent.

   Answer: False. Cloud capacity pressures and spot revocations are driven by macroeconomic cluster demand spikes and provider-level resource reclamation within a datacenter or Availability Zone. Preemption events for high-demand GPU types (e.g., A100/H100) are heavily correlated within an AZ; when capacity becomes scarce, providers reclaim multiple instances simultaneously, invalidating assumptions of independent failure.

   Learning Objective: Evaluate the assumption of statistical independence in cloud spot instance preemption events.

← Back to Questions

Self-Check: Answer

1. How does the Tiresias scheduler solve the problem of inaccurate and inflated user runtime estimates in GPU cluster scheduling?

   1. It uses a Two-Dimensional Gittins Index (Least Attained Service) based on observed spatial (GPU count) and temporal (elapsed execution time) resource attainment, prioritizing jobs that have consumed less cumulative compute without asking users for duration estimates.
   2. It requires users to pay monetary deposits that are forfeited if their job runtime deviates by more than \(10\%\) from their initial estimate.
   3. It parses Python ASTs at submission time to statically calculate the exact loop iteration count and FLOP requirements of every training script.
   4. It forces all jobs to run for exactly 60 minutes before automatically killing and requeueing them.

   Answer: The correct answer is A. Tiresias eliminates the game-theoretic distortion of user runtime estimates by adopting duration-agnostic scheduling. It implements a 2D Gittins Index (Multi-Level Feedback Queue / Least Attained Service) that tracks both spatial (GPU count) and temporal (time elapsed) resources consumed. Jobs with low attained service are given higher priority, naturally prioritizing short exploratory jobs over long pretraining runs without requiring users to predict runtimes. Financial penalties, AST-based static runtime calculation, and fixed 60-minute time limits are incorrect.

   Learning Objective: Explain how Tiresias eliminates reliance on user runtime estimates using Least Attained Service (LAS) scheduling.

2. In the Pollux scheduling architecture, how is the Goodput** metric (\(G_j\)) defined to co-optimize cluster resource allocation and job-level training dynamics?**

   1. Goodput is the total number of raw GPU clock cycles executed divided by the cluster’s electricity consumption in kilowatt-hours.
   2. Goodput is the ratio of network packets transmitted over InfiniBand to the number of retransmissions caused by packet drops.
   3. Goodput is the product of job training speed (step throughput \(T_j\), steps per second) and statistical efficiency (\(E_j\), progress made per step toward convergence given the current batch size and worker allocation).
   4. Goodput is the percentage of time GPUs spend running CUDA kernels compared to CPU data loading time.

   Answer: The correct answer is C. Pollux introduces Goodput \(G_j(a, M) = T_j(a, M) \times E_j(M)\), where \(T_j(a, M)\) is the system throughput (execution speed in steps/second given resource allocation \(a\) and batch size \(M\)) and \(E_j(M)\) is the statistical efficiency (how effectively each gradient step reduces validation loss relative to a baseline batch size). By maximizing total cluster goodput, Pollux dynamically adjusts both resource allocations (GPUs) and training hyperparameters (batch size, learning rate). The other choices define energy efficiency, network reliability, and GPU utilization metrics.

   Learning Objective: Apply the Pollux Goodput optimization metric to balance cluster-wide resource allocation and job-level batch size.

3. What mechanisms does Gandiva use to achieve fine-grained, low-overhead time-slicing and job migration across GPUs?

   Answer: Gandiva operates at the granularity of mini-batch iterations. It monitors cyclic execution patterns and executes time-slicing, job suspension, and GPU migration strictly at iteration boundaries when GPU memory contains only model weights (and intermediate activations are deallocated), minimizing memory serialization overhead and eliminating wasted progress between checkpoints.

   Learning Objective: Describe the core mechanisms used by Gandiva to achieve low-latency scheduling without wasting training progress.

4. The custom ML scheduler Themis optimizes for ____ fairness, ensuring that each job completes in a time comparable to what it would achieve in a dedicated cluster share, thereby preventing gaming and multi-resource allocation monopolies.

   Answer: finish-time (or finish-time fairness, finish time). Themis designs an auction-based, finish-time fairness metric that balances resource allocation based on how much a scheduler’s decision affects the actual completion time of competing jobs.

   Learning Objective: Identify the fairness concept implemented by Themis to prevent users from gaming cluster resource allocators.

5. **Order these custom ML scheduler milestones chronologically by their introduction of specialized ML scheduling primitives:

1. Pollux: Co-optimizing cluster resource allocation with dynamic batch size tuning via statistical Goodput modeling.
2. Gandiva: Iteration-aware scheduling enabling sub-second time-slicing and migration at mini-batch boundaries.
3. Traditional HPC schedulers (Slurm / PBS): Static batch queue allocation with priority and fair-share decay.
4. Tiresias: Duration-agnostic 2D Least Attained Service (LAS) scheduling eliminating reliance on user runtime estimates.**

Answer: The correct order is (3) -> (2) -> (4) -> (1). Explanation: Step (3): Traditional HPC schedulers (Slurm / PBS): Static batch queue allocation with priority and fair-share decay. Step (2): Gandiva (OSDI 2018): Iteration-aware scheduling enabling sub-second time-slicing and migration at mini-batch boundaries. Step (4): Tiresias (NSDI 2019): Duration-agnostic 2D Least Attained Service (LAS) scheduling eliminating reliance on user runtime estimates. Step (1): Pollux (OSDI 2021): Co-optimizing cluster resource allocation with dynamic batch size tuning via statistical Goodput modeling.

Learning Objective: Classify and sequence the primary design innovations introduced by research ML schedulers into modern orchestration.

← Back to Questions

Self-Check: Answer

1. Why is standard CPU/GPU utilization (such as reported by Kubernetes Horizontal Pod Autoscaler) an ineffective metric for autoscaling LLM inference replicas, and what metric should be used instead?

   1. GPU utilization is strictly proprietary to NVIDIA and cannot be exported to Prometheus.
   2. GPU utilization measures compute core occupancy rather than request latency or queue saturation; an LLM serving replica can be stalled on KV-cache memory capacity or request concurrency while reporting modest GPU SM utilization, making request concurrency or queue depth the superior scaling metric.
   3. GPU utilization fluctuates faster than the speed of light, violating Nyquist sampling criteria in etcd.
   4. LLM serving replicas do not use GPUs during inference, relying entirely on CPU vector instructions.

   Answer: The correct answer is B. Standard GPU utilization metrics (SM activity) poorly reflect LLM serving load because models frequently become memory-bound (KV-cache exhaustion) or queue-bound (incoming requests waiting for iteration slots in continuous batching) while SM utilization remains moderate or spiky. Using request concurrency, pending queue depth (e.g., from vLLM/TGI), or TTFT/ITL latency metrics provides a direct, responsive signal for autoscaling. The proprietary claim, Nyquist violation, and CPU-only claim are fictitious distractors.

   Learning Objective: Evaluate autoscaling metrics and cold-start latency challenges in LLM serving infrastructure.

2. In a multi-tenant inference environment hosting multiple smaller models on shared GPUs, what is the primary architectural trade-off between NVIDIA Multi-Process Service (MPS) and Multi-Instance GPU (MIG)?

   1. MIG is only supported on consumer GeForce cards, whereas MPS requires enterprise Grace Hopper superchips.
   2. MPS provides sub-nanosecond hardware memory encryption, whereas MIG only supports unencrypted cleartext transfers.
   3. MPS requires dedicated physical PCIe switches per tenant, whereas MIG operates over standard Ethernet cables.
   4. MPS dynamically shares streaming multiprocessors (SMs) and memory bandwidth with high packing density but lacks hard memory fault isolation (an OOM in one client can crash the MPS server), whereas MIG enforces strict hardware isolation (dedicated SMs, memory controllers, L2 cache slices) with full fault isolation but static partition boundaries.

   Answer: The correct answer is D. MPS allows multiple processes to concurrently share GPU compute resources (SMs) and memory bandwidth with low kernel launch latency and high utilization, but a memory fault (such as an unhandled CUDA error or OOM) in one process can bring down the entire shared MPS context. MIG provides strict hardware partitioning (dedicated SMs, memory controllers, L2 cache slices) preventing fault propagation and noisy-neighbor effects, but requires static partition profiles that cannot dynamically adapt to shifting request sizes without draining. The other options present incorrect hardware and feature claims.

   Learning Objective: Compare GPU resource sharing mechanisms (MIG, MPS, Time-slicing) along isolation, memory protection, and efficiency dimensions.

3. How can an organization exploit the diurnal traffic cycle of online serving to improve overall GPU fleet utilization, and what preemption mechanism is necessary to protect serving SLOs?

   Answer: Online inference demand follows predictable diurnal curves (peaking during business hours and dropping by \(50-70\%\) at night). Organizations can run opportunistic batch training, fine-tuning, or offline evaluation jobs on idle serving GPUs during off-peak hours. To protect latency SLOs during sudden morning traffic surges, the orchestrator must enforce priority preemption with rapid checkpointing or elastic downsizing so training jobs immediately yield capacity back to serving deployments.

   Learning Objective: Design an operational policy for harvesting diurnal idle inference GPU capacity for batch training.

4. Why does reactive autoscaling alone often fail to maintain P99 latency SLOs during sudden inference traffic spikes, and how does predictive autoscaling mitigate this?

   Answer: Reactive autoscaling triggers only after metrics (e.g., request queue depth) exceed thresholds, after which provisioning a new GPU container entails pulling multi-gigabyte container images, loading tens of gigabytes of model weights into HBM, and warming up CUDA kernels (a cold start of 30-120 seconds). During this lag, incoming traffic overflows buffers and violates latency SLOs. Predictive autoscaling uses time-series forecasting to pre-provision GPU replicas in advance of anticipated traffic ramps.

   Learning Objective: Analyze why predictive autoscaling is necessary to prevent SLO violations during inference traffic spikes.

5. True or False: In a multi-replica LLM inference cluster, routing user requests randomly via round-robin DNS achieves the highest overall cluster throughput because it balances raw request counts evenly across GPUs.

   Answer: False. LLM inference performance depends heavily on KV-cache reuse. Routing requests with shared system prompts or document contexts to the specific GPU replicas that already host the cached prefix in memory (prefix-aware / KV-cache-aware routing) drastically reduces time-to-first-token (TTFT) and eliminates redundant prefill compute, outperforming naive round-robin routing.

   Learning Objective: Evaluate the interaction between intelligent request routing and GPU memory locality in LLM inference fleets.

6. In LLM inference serving, while ____ measures the responsiveness before the first output token appears, Inter-Token Latency (ITL) measures the generation speed of subsequent tokens.

   Answer: Time to First Token (or TTFT, Time-To-First-Token). TTFT captures prompt processing / prefill latency, which along with ITL defines the end-to-end user experience and service level objectives.

   Learning Objective: Identify key performance indicators used in LLM serving autoscaling.

← Back to Questions

Self-Check: Answer

1. A cluster monitoring dashboard reports \(85\%\) GPU utilization, yet researchers complain of slow progress and low experiment output. What distinction between fleet utilization metrics explains this apparent contradiction?

   1. Allocated utilization (GPUs reserved by containers) may be high (\(85\%\)) while compute utilization (SM active time) is low (\(20\%\)) due to data loader stalls, or productive utilization (goodput advancing converged training steps) is near zero due to unhandled AllReduce synchronization deadlocks.
   2. Hardware sensors on GPUs report utilization in hexadecimal rather than decimal percentages.
   3. Grafana dashboards invert metric percentages when cluster temperature exceeds 25 degrees Celsius.
   4. GPU utilization metrics only count CPU core cycles, concealing whether the GPU is plugged into power.

   Answer: The correct answer is A. Cluster observability requires distinguishing between three metrics: (1) Allocated Utilization (fraction of GPUs assigned to jobs, which can be 85%), (2) Compute Utilization (fraction of time GPU SMs are actively executing instructions rather than waiting on CPU data loading, host memory, or I/O), and (3) Productive Utilization / Goodput (fraction of compute advancing valid training steps rather than stalled in communication barriers, deadlocks, or throwaway restarts). The hexadecimal, temperature inversion, and CPU-only options are absurd distractors.

   Learning Objective: Differentiate between Allocated Utilization, Compute Utilization, and Productive Goodput in cluster observability.

2. What is a ‘preemption cascade’ in a multi-tenant GPU cluster, and how can orchestrators mitigate its destructive impact on cluster throughput?

   1. A preemption cascade occurs when a single GPU catches fire and burns adjacent GPUs in the same server rack; it is mitigated by Halon fire suppression systems.
   2. A preemption cascade occurs when Slurm automatically deletes user source code repositories upon job completion; it is mitigated by Git branch protection.
   3. A preemption cascade occurs when Kubernetes continuously restarts healthy pods due to Docker daemon upgrades; it is mitigated by disabling Linux kernel updates.
   4. A preemption cascade occurs when high-priority jobs evict medium-priority jobs, which in turn evict low-priority jobs across the cluster, triggering simultaneous checkpoint I/O storms and thrashing compute progress; it is mitigated by preemption cooldown periods and holistic preemption limits.

   Answer: The correct answer is D. In multi-priority environments, an influx of high-priority jobs can evict medium-priority jobs, which then preempt low-priority jobs to find room. If dozens of jobs simultaneously start saving checkpoints before termination, the resulting I/O storm saturates shared storage, prolongs eviction delays, and causes cluster thrashing. Schedulers mitigate this using preemption cooldown timers (preventing a recently started job from immediate re-preemption), holistic bandwidth reservation for checkpointing, and non-preempting priority tiers (preemptionPolicy: Never). The physical fire, code deletion, and Docker upgrade options are incorrect.

   Learning Objective: Evaluate the risks of cascading preemption storms and design mechanisms to prevent them.

3. In a multi-tenant cluster, how does decoupling ‘guaranteed quota’ from ‘limit (burst) quota’ prevent stranded idle capacity while protecting team allocations?

   Answer: A guaranteed quota represents a team’s baseline dedicated capacity that is immediately available and immune to eviction by peers. A limit (burst) quota allows a team to opportunistically utilize idle cluster capacity above their guarantee when overall demand is low. If the owning team reclaims their guaranteed quota, the scheduler preempts the bursting team’s opportunistic jobs, maximizing aggregate utilization without violating baseline access rights.

   Learning Objective: Explain the architectural difference between guaranteed quota and limit quota in shared GPU clusters.

4. What is ‘quota hoarding’ in shared GPU clusters, and what automated scheduler policies prevent researchers from engaging in it?

   Answer: Quota hoarding occurs when users submit placeholder or dummy sleeping jobs (e.g., interactive shells or infinite loops) to keep GPUs continuously reserved under their quota even when not running useful computation. Schedulers counter this via: (1) Inactivity timeouts that detect zero SM/memory activity and auto-terminate idle sessions; (2) Dynamic lease-based allocations that require active renewal; and (3) Fair-share decay accounting that charges users for holding time regardless of compute intensity.

   Learning Objective: Analyze the organizational anti-pattern of quota hoarding and describe governance mechanisms to eliminate it.

5. True or False: Under Dominant Resource Fairness (DRF), if Team A’s workload requires \(4\text{ GPUs}\) and \(8\text{ CPU cores}\) per worker while Team B’s workload requires \(1\text{ GPU}\) and \(64\text{ CPU cores}\) per worker on a cluster with \(100\text{ GPUs}\) and \(1000\text{ CPU cores}\), DRF equalizes each team’s share of their dominant resource (GPUs for Team A, CPUs for Team B).

   Answer: True. Dominant Resource Fairness (DRF) multi-resource allocation identifies each tenant’s dominant resource (the resource type for which the tenant requests the largest fraction of total cluster capacity) and equalizes the dominant shares across all active tenants, ensuring multi-resource equity without allowing single-resource monopolies.

   Learning Objective: Apply Dominant Resource Fairness (DRF) principles to multi-resource ML allocations.

6. **Order the hierarchy of checks performed by a multi-tenant scheduler when Team A in Department X submits a new 16-GPU job request:

1. Evaluate Team A’s internal user priority and historical fair-share usage decay relative to teammate submissions.
2. Determine if cluster-wide unallocated capacity or preemption candidates exist to host the 16-GPU allocation.
3. Verify that Department X has not exceeded its total department-level GPU budget limit.
4. Check whether Team A’s currently active allocations are within its guaranteed quota or if it must request burst/borrowed capacity.
5. Grant the lease, bind the containers to physical hardware, and update accounting meters for Department X and Team A.**

Answer: The correct order is (3) -> (4) -> (1) -> (2) -> (5). Explanation: Step (3): Verify that Department X has not exceeded its total department-level GPU budget limit. Step (4): Check whether Team A’s currently active allocations are within its guaranteed quota or if it must request burst/borrowed capacity. Step (1): Evaluate Team A’s internal user priority and historical fair-share usage decay relative to teammate submissions. Step (2): Determine if cluster-wide unallocated capacity or preemption candidates exist to host the 16-GPU allocation. Step (5): Grant the lease, bind the containers to physical hardware, and update accounting meters for Department X and Team A.

Learning Objective: Order the decision hierarchy in a hierarchical multi-tenant fair-share scheduling engine.

← Back to Questions

Self-Check: Answer

1. An administrator observes that average GPU compute utilization across a 1,000-GPU cluster is only \(55\%\), yet users experience multi-day queue wait times for 64-GPU jobs. Which diagnostic finding points to the primary root cause?

   1. The cluster’s Ethernet cables are physically too long, causing optical signals to decay into radio waves.
   2. Severe resource fragmentation and strict gang scheduling constraints prevent 64-GPU contiguous blocks from forming, while short jobs hold scattered GPUs across nodes without filling them.
   3. The NVIDIA driver automatically caps utilization at 55% when GPU fans are running on low speed.
   4. Python code running in user containers automatically halts execution whenever cluster utilization drops below 60%.

   Answer: The correct answer is B. In multi-tenant GPU clusters, moderate average utilization (e.g., 55%) frequently coexists with long queue times due to fragmentation and gang scheduling requirements: uncoordinated, non-gang small jobs occupy 1-2 GPUs per node across many nodes, leaving hundreds of GPUs idle in aggregate but preventing any single 64-GPU contiguous gang allocation from assembling. Cable length, fan speed capping, and Python auto-halting are fictional distractors.

   Learning Objective: Analyze the root causes of high queue wait times coexisting with moderate average cluster utilization.

2. During a multi-GPU training job, DCGM reports that GPU SM utilization oscillates rapidly between \(10\%\) and \(95\%\) with low memory copy throughput, while top on the host node shows \(100\%\) CPU utilization across all cores and high disk wait (\(iowait\)). What is the root cause and remedy?

   1. The GPU hardware is defective; the operator should replace the GPUs with CPU-only nodes.
   2. The PyTorch model has too many layers; the model architecture must be pruned to 1 layer.
   3. The training job is bottlenecked on host data loading, decompression, or augmentation; the remedy is optimizing PyTorch num_workers, utilizing shared memory (/dev/shm), caching preprocessed tensors on local NVMe SSDs, or offloading transforms to GPU using DALI.
   4. The cluster InfiniBand switch is operating in half-duplex mode; the administrator must power cycle the spine switch.

   Answer: The correct answer is C. An oscillating GPU utilization pattern accompanied by pinned host CPU cores and high \(iowait\) is the classic signature of an input data pipeline bottleneck: GPUs finish backpropagation rapidly, then stall waiting for the host CPU DataLoader workers to fetch, decompress, and augment the next batch of training images or tokens. Remedies include increasing/tuning num_workers, storing datasets on local NVMe SSDs, pinning host memory (pin_memory=True), using /dev/shm, or leveraging GPU-accelerated pipelines like NVIDIA DALI. The other options are incorrect diagnoses and destructive actions.

   Learning Objective: Identify hardware telemetry patterns indicating host CPU and I/O bottlenecks in GPU training runs.

3. What is a ‘zombie allocation’ (or ghost job) in a GPU cluster, and how can automated health monitors detect and remediate it?

   Answer: A zombie allocation occurs when the scheduler records a GPU as allocated to a running job, but the underlying container or user training process has crashed or entered an unrecoverable CUDA error state, leaving the GPU idle and inaccessible to other users. Automated monitors detect zombies by correlating scheduler allocation tables with DCGM metrics (zero SM and memory activity over a prolonged window) or process tree inspections (no active PID attached to /dev/nvidia*), automatically draining the allocation, killing orphaned processes, and resetting the GPU context.

   Learning Objective: Formulate a diagnostic procedure to identify and clean up zombie GPU allocations.

4. True or False: In a ring AllReduce across 32 nodes, if 31 nodes show \(95\%\) GPU utilization and 1 node shows \(10\%\) utilization, the node with \(10\%\) utilization is the only healthy node while the 31 nodes are encountering network hardware faults.

   Answer: False. The reverse is true: in synchronous ring AllReduce, a single slow worker or degraded network link (the straggler) slows down the entire collective; while the straggler takes longer to compute or transfer its chunk, all other 31 workers finish their compute early and spin in the synchronization barrier (often registering high kernel active time in spin-locks or waiting for the straggler to complete). Profiling communication timelines reveals the straggler as the source of the delay.

   Learning Objective: Evaluate telemetry signals to pinpoint straggler nodes in multi-node AllReduce rings.

5. When diagnosing training performance bottlenecks, operators inspect the ____ throughput (measured in GB/s) to ensure that batches transferred from host memory to accelerator HBM are not constrained by host bus contention.

   Answer: PCIe (or PCIe bandwidth, PCIe throughput, PCIe transfer). PCIe transfer bandwidth (Host-to-Device / H2D) is a key diagnostic metric for detecting unpinned memory copies and bus saturation during data loading.

   Learning Objective: Identify the hardware metric used to diagnose host-to-device data transfer bottlenecks.

← Back to Questions

Self-Check: Answer

1. Why is targeting near \(100\%\) GPU cluster utilization an operational anti-pattern for an interactive AI research and development organization?

   1. GPU hardware components melt if utilization exceeds \(90\%\) for more than 5 consecutive minutes.
   2. Modern deep learning frameworks automatically downgrade model accuracy when cluster utilization exceeds \(85\%\).
   3. Operating above \(95\%\) utilization causes cloud providers to double their hourly billing rates without warning.
   4. As utilization approaches \(100\%\), queuing theory (Pollaczek-Khinchine) dictates that job queue wait times explode hyperbolically, delaying fast feedback for exploratory experiments and severely reducing overall engineering velocity.

   Answer: The correct answer is D. According to queuing theory, as cluster utilization (\(\rho\)) approaches \(1.0\), the waiting time in the queue diverges toward infinity (\(W_q \propto \frac{\rho}{1-\rho}\)). In research organizations with heavy-tailed job sizes (\(C_s \approx 3\)), operating above \(80-85\%\) utilization causes small debugging and exploratory jobs to wait days in queues. This destroys developer velocity and feedback loops, proving that maximizing raw utilization at the expense of queue latency is an anti-pattern. Hardware melting, accuracy downgrading, and stealth billing rate doubling are fabricated.

   Learning Objective: Evaluate why operating a research GPU cluster at near 100% utilization is counterproductive to organizational velocity.

2. Explain the fallacy that ‘more sophisticated scheduling algorithms always improve utilization,’ and describe what operators should audit before replacing their cluster scheduler.

   Answer: Low cluster utilization and long wait times are more frequently caused by policy misconfigurations than algorithmic limitations. Before replacing or upgrading a scheduler, operators should audit: (1) Resource request inflation (users requesting 8 GPUs when 1 is needed); (2) Unconfigured gang scheduling timeouts; (3) Mismatched hardware partitions (e.g., jobs requiring A100s when A10s suffice); and (4) Stale fair-share decay parameters that fail to balance contention.

   Learning Objective: Explain why policy misconfigurations rather than scheduling algorithm limitations are often the root cause of poor cluster efficiency.

3. True or False: Gang scheduling is strictly mandatory for every distributed ML workload, including asynchronous parameter-server training, hyperparameter sweeps, and elastic fine-tuning jobs.

   Answer: False. Gang scheduling is mandatory only for bulk-synchronous parallel training where every worker must reach every AllReduce barrier. Workloads using asynchronous gradient updates (e.g., parameter servers), independent hyperparameter trials, or elastic training frameworks with dynamic worker membership can make useful progress with variable or partial allocations, and enforcing rigid gang scheduling on them unnecessarily increases queue wait times.

   Learning Objective: Evaluate when gang scheduling constraints can be relaxed in distributed machine learning workloads.

← Back to Questions

Self-Check: Answer

1. Which statement best synthesizes the central architectural challenge of fleet orchestration for machine learning?

   1. Orchestration is purely a software compilation task that translates PyTorch code into C++ CUDA binaries without considering cluster hardware.
   2. Orchestration requires balancing physical hardware constraints (NVLink topology, failure rates, power limits) against organizational policies (fairness, priorities, cost) across diverse, long-running, and failure-prone distributed workloads.
   3. Orchestration has been completely solved by standard web application load balancers like NGINX, which handle GPU AllReduce synchronization transparently.
   4. Orchestration only matters for single-GPU laptops and has no relevance to multi-thousand GPU datacenters.

   Answer: The correct answer is B. Fleet orchestration operates at the intersection of physical infrastructure (GPU heterogeneity, NVLink topologies, power boundaries, component failure rates) and organizational objectives (fair-share equity, priority preemption, budget optimization, latency vs. throughput). It translates multi-dimensional, distributed resource demands into physical allocations while ensuring high goodput and fault resilience. Software compilation, NGINX load balancing, and laptop-only claims misrepresent the role of orchestration.

   Learning Objective: Synthesize the primary architectural trade-offs across the fleet orchestration stack.

2. How is the historical boundary between HPC batch scheduling (Slurm) and cloud-native service orchestration (Kubernetes) evolving to meet the demands of modern ML lifecycles?

   Answer: The boundary is converging: Kubernetes is integrating HPC capabilities (gang scheduling via Volcano/Kueue, topology-aware placement, RDMA bypass, and fair-share queues) to support large-scale distributed training, while HPC environments are incorporating containerization, declarative APIs, and dynamic service management. Organizations increasingly deploy hybrid architectures where outer Kubernetes control planes manage end-to-end pipelines and serving, orchestrating high-performance batch training clusters beneath.

   Learning Objective: Explain how the convergence of cloud-native and HPC orchestration paradigms addresses modern AI training and serving needs.

3. True or False: In large-scale ML cluster management, maximizing raw GPU allocated utilization (the percentage of GPUs assigned to active containers) guarantees optimal economic value and research productivity.

   Answer: False. Raw allocated utilization measures only resource reservation, not productive work. A cluster with \(95\%\) allocated utilization may deliver near-zero research value if GPUs are stalled in AllReduce deadlocks, starved by slow CPU data loaders, thrashing in preemption cascades, or executing diverged training runs. Productive Goodput—the rate of valid, converged training steps completed per dollar spent—is the definitive measure of fleet efficiency.

   Learning Objective: Justify why Productive Goodput rather than raw GPU utilization is the definitive metric for fleet efficiency.

← Back to Questions