The Economics and Architecture of Inference

Inference at Scale

Global inference blueprint where requests route through regional gateways, batching systems, cache pages, model shards, and fleet serving workers.

Purpose

Why does inference cost eventually dwarf training cost, and what does this mean for system design?

Training is a one-time expense; serving continues for the model’s operational lifetime. A model trained once for millions of dollars may serve billions of requests, each consuming compute, memory, network, and energy. For a successful production model, inference operating expense can exceed the original training cost by orders of magnitude. That cost structure shapes the architecture. Milliseconds of latency and percentage points of accelerator utilization compound across billions of requests. Performance engineering supplied the local toolkit—fusion, precision, compilation, and algorithmic changes such as speculative decoding—and this chapter applies those tools at serving scale. Batching, sharding, routing, autoscaling, and isolation must preserve strict latency budgets under live traffic. Reliability requirements also intensify because downtime means lost revenue and broken user experiences rather than delayed experiments. In C³ terms, serving batches compute for throughput while holding coordination overhead within tail-latency budgets.

Learning Objectives
  • Quantify lifetime serving cost from request volume, token mix, accelerator price, utilization, and latency targets
  • Select batching and scheduling policies for vision, LLM, recommender, and streaming workloads
  • Analyze attention-cache capacity, fragmentation, and decode-time policies for memory-bound language-model serving
  • Compare sharding, disaggregation, and quantized serving designs using communication, memory, and quality constraints
  • Evaluate routing, load balancing, and isolation controls against tail latency and noisy-neighbor risks
  • Design autoscaling and multi-region failover policies that balance cold starts, cost, and SLO compliance
  • Synthesize serving architectures that manage C³ trade-offs across request, replica, service, and platform layers

Imagine a language model that costs $5 million to train over two months. Once deployed to a global user base serving thousands of queries per second, that same model can consume $5 million in inference compute every week. Inference is no longer a temporary batch job. It is a continuous, latency-sensitive service.

Operator fusion, precision engineering, and graph compilation can maximize throughput on individual forward passes. The next challenge emerges when a single optimized model must serve thousands of concurrent users across a globally distributed fleet. Single-machine inference optimization, including batching, caching, model optimization, and hardware acceleration, provides the building blocks. Distributed approaches become necessary when those techniques reach their limits.

Distributed inference systems must solve problems that do not exist at single-machine scale. Load balancing1 becomes critical when requests must be distributed across hundreds of GPU instances while maintaining latency guarantees. Request routing must account for model-specific characteristics: recommendation systems with trillion-parameter embedding tables require different placement strategies than large language models that generate responses token by token. Autoscaling must anticipate demand fluctuations that can change request volume by orders of magnitude within minutes while maintaining latency bounds users expect.

1 Load Balancing (Inference): GPU inference requests range from 10 ms to 30+ seconds with high variance, unlike web requests that complete in uniform milliseconds. This variance invalidates round-robin and random assignment, forcing queue-depth-aware routing strategies that add monitoring overhead but prevent tail-latency blowups across heterogeneous GPU fleets.

The economics of inference at scale differ fundamentally from training economics. Training costs are dominated by compute time and can be amortized over the lifetime of the resulting model. Inference costs are directly tied to user traffic and revenue. An e-commerce recommendation system might serve millions of requests per second during peak shopping periods, with each request contributing directly to potential revenue. The cost of overprovisioning during quiet periods or underprovisioning during peaks translates immediately to business impact. Inference efficiency becomes a first-order concern in ways that training efficiency rarely achieves. Building such a service requires three linked decisions: when distribution becomes necessary, how the architecture preserves latency bounds under varying load, and how resource utilization stays high enough that serving cost does not erode the value the model creates.

When single-machine serving is insufficient

Three distinct signals indicate when distributed inference becomes necessary rather than merely optional. Table 1 categorizes these triggers by constraint type and corresponding strategy.

The first signal is memory exhaustion, which occurs when model parameters, key-value caches, or embedding tables exceed single-device capacity. A single NVIDIA H100 GPU provides 80 GB of HBM32 memory; Assumption Provenance records the provenance of this capacity figure and the other canonical hardware constants used throughout the analysis that follows. GPT-4-class models dwarf that capacity: the public ~1.8T-parameter mixture-of-experts estimate implies ~3.5 TB for weights alone in FP16 precision, forcing distribution across multiple GPUs regardless of throughput requirements. Recommendation systems with trillion-parameter embedding tables face similar constraints: Meta’s DLRM architecture3 stores embedding tables that require multiple terabytes of memory.

2 HBM3 (High Bandwidth Memory 3): 3D-stacked DRAM delivering 3.35 TB/s on the H100 vs. 2.04 TB/s for HBM2e on the A100. Because large language model (LLM) decode is memory-bandwidth-bound, this improvement translates directly into higher tokens-per-second and larger feasible batch sizes, making high-bandwidth memory (HBM) generation one major variable in inference cost-per-token.

3 DLRM (Deep Learning Recommendation Model): Meta’s 2019 reference architecture separates dense features – GPU-bound multilayer perceptrons (MLPs) – from sparse features (CPU-bound embedding lookups), creating a hybrid serving topology (Naumov et al. 2019). Production recommendation characterizations show why this structure matters for serving: embedding-heavy recommendation models create memory-access and capacity constraints that differ from dense convolutional neural network (CNN), recurrent neural network (RNN), or LLM inference (Gupta et al. 2020).

Naumov, Maxim, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, et al. 2019. “Deep Learning Recommendation Model for Personalization and Recommendation Systems.” arXiv Preprint arXiv:1906.00091.
Gupta, Udit, Carole-Jean Wu, Xiaodong Wang, Maxim Naumov, Brandon Reagen, David Brooks, Bradford Cottel, et al. 2020. “The Architectural Implications of Facebook’s DNN-Based Personalized Recommendation.” 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), 488–501. https://doi.org/10.1109/HPCA47549.2020.00047.

Beyond memory constraints, throughput limitations emerge when request volume exceeds single-machine capacity even with optimal batching. Consider a recommendation system serving 100,000 queries per second with a 10 ms latency budget. If single-machine throughput peaks at 10,000 QPS, no amount of optimization on that machine can satisfy demand. Horizontal scaling across multiple replicas becomes mandatory.

Finally, strict latency requirements drive distribution when model execution time exceeds latency budgets even at batch size one. Large language models generating responses token by token face this constraint acutely. A 70-billion parameter model requires approximately 140 GB of memory in FP16, exceeding a single 80 GB H100-class GPU before KV cache is considered. Even when quantization or larger-memory hardware makes a single-replica configuration possible, decode throughput is often limited by memory bandwidth. Sharding the model across multiple GPUs enables parallel computation that reduces time-to-first-token below acceptable thresholds.

Table 1: Triggers for Distributed Inference: Each constraint type indicates different distribution strategies. Memory constraints require model sharding; throughput constraints require replication; latency constraints may require either depending on whether the bottleneck is compute or memory bandwidth.
Constraint Single-Machine Limit Example Workload Distribution Strategy
Memory 80 GB (H100) GPT-4 (~3.5 TB FP16) Tensor/pipeline parallelism
Throughput ~10K QPS (vision) 100K QPS RecSys Horizontal replication
Latency Forward pass \(> \text{SLO}\) 500 ms LLM TTFT Model sharding

Memory, throughput, and latency triggers all become user-visible once an interactive endpoint starts streaming. Time to First Token (TTFT)4 determines when the user sees a response begin, while Time Per Output Token (TPOT) determines whether the rest of the response arrives at a readable pace; this is why inference reverses many of the optimization priorities from training.

4 TTFT (Time to First Token) vs. TPOT (Time per Output Token): Time to first token (prefill) determines the perceived responsiveness; time per output token (decode) determines the perceived reading speed. Humans read at roughly 5–10 tokens/sec; TPOT above roughly 100–200 ms can begin to feel slower than natural reading, while many products target lower TPOT for smooth streaming. This dual service level agreement (SLA) requirement forces schedulers to prioritize decode tokens over new prefill prompts.

Checkpoint 1.1: Distribution strategy selection

Verify your understanding of when to move from single-machine to distributed inference:

The fundamental inversion: Training vs. inference

The contrast between training and inference optimization extends beyond the basic throughput-vs.-latency distinction. Training optimizes for samples processed per hour and tolerates latency variations. Inference optimizes for response time and must meet strict latency bounds. At scale, this inversion manifests in system architecture, resource allocation, and operational priorities. Table 2 details six key system aspects where these differences emerge.

Table 2: Training vs. Inference System Requirements: The fundamental inversion from throughput to latency optimization ripples through system design, state management, batch formation, and failure recovery.
Aspect Distributed Training Distributed Inference
Primary metric Throughput (samples/hour) Latency (P99 ms)
Acceptable variance Hours Milliseconds
State management Checkpoints (periodic) Session state (continuous)
Batch formation Large, controlled Request-driven, variable
Failure tolerance Restart from checkpoint Redirect without user impact
Cost structure Fixed duration, variable rate Variable duration, fixed SLO

Training tolerates substantial latency variance because the optimization target is aggregate progress over hours or days. A training iteration that takes 2 seconds instead of the usual 1 second represents acceptable variation. An inference request that takes 2 seconds instead of 100 milliseconds represents catastrophic failure, potentially causing user abandonment or cascading timeouts in dependent services.

State management differs fundamentally. Training maintains model state (parameters, optimizer states) that evolves gradually and can be captured in periodic checkpoints. Inference often maintains session state (conversation history, key-value caches, user context) that must be preserved across requests and cannot tolerate the staleness that checkpoint-based recovery would introduce.

Failure handling diverges correspondingly. Training failures trigger checkpoint restoration and continuation, with minutes of lost progress being acceptable. Inference failures must be invisible to users. Requests redirect to healthy replicas, degraded results substitute for unavailable models, and service-level objectives (SLOs) must be maintained despite infrastructure instability.

The serving tax: Overhead of distribution

Distributing inference across multiple machines introduces overhead absent from single-machine serving. This “serving tax” must be understood and budgeted within latency constraints.

Network communication adds latency for every cross-machine interaction. Within a data center, network round-trip times range from 50–500 microseconds depending on topology and congestion. For model sharding that requires synchronization between GPUs on different machines, each synchronization point adds this overhead. A model sharded across 8 machines with 4 synchronization points per inference adds 200 microseconds to 2 milliseconds of network latency. The α-β Communication Model develops the \(\alpha\)-\(\beta\) model that decomposes each such transfer into a fixed startup latency plus a bandwidth-dependent term, giving the quantitative framework for budgeting these synchronization costs against a latency SLO.

Serialization overhead compounds the problem by converting in-memory tensors to network-transmittable formats. Traditional serializers like Protocol Buffers, JSON, and Python’s pickle perform a parse-allocate-copy cycle on every transfer, and a 1 GB activation tensor takes approximately 100 milliseconds to serialize and deserialize through such formats. Zero-copy alternatives like FlatBuffers and Cap’n Proto5 sidestep most of this cost by accessing wire-format data in place, but they impose stricter schema layout requirements that not every production stack adopts.

5 Zero-Copy Serialization: FlatBuffers and Cap’n Proto access wire-format data in place, eliminating the parse-allocate-copy cycle of Protocol Buffers or JavaScript Object Notation (JSON). For distributed inference with large activation tensors, this drops per-hop serialization overhead from milliseconds to microseconds, reclaiming latency budget that would otherwise consume 5–10 percent of a tight SLO.

Load balancer latency adds another layer. Requests must be routed to appropriate replicas, which requires examining request metadata, consulting routing tables, and forwarding to selected backends. Well-optimized load balancers add 100–500 microseconds; poorly configured ones can add milliseconds.

Coordination overhead emerges when requests require fan-out to multiple services. A recommendation system that queries a user model, item model, and ranking model in parallel must coordinate these queries and aggregate results. The coordination logic itself consumes CPU cycles and introduces latency variation. Finally, queuing delay (\(T_{\text{queuing}}\)) accumulates whenever request arrival rates temporarily exceed available replica processing capacity.

The total serving tax often consumes 10–30 percent of the latency budget in distributed systems, as equation 1 shows:

\[T_{\text{total}} = T_{\text{compute}} + T_{\text{network}} + T_{\text{serialization}} + T_{\text{coordination}} + T_{\text{queuing}} \tag{1}\]

Minimizing this tax requires co-locating communicating components, using high-bandwidth interconnects, and designing communication patterns that minimize round trips.

Serving cost can dominate training cost

Two horizontal bars comparing lifetime cost terms. The serving OpEx bar is wide and shaded orange, dominating; the training CapEx bar is a narrow gray sliver.

Lifetime serving cost dwarfs one-time training.

The serving tax quantified in equation 1 consumes a fraction of the latency budget per request. The true economic impact of inference emerges when evaluating cost over a model’s entire operational lifetime. The Serving Cost Dominance Law (principle 15) states that serving cost can dominate training cost by orders of magnitude because training is a one-time capital expenditure (CapEx) while serving is a continuous operational expenditure (OpEx) that scales with user growth. A quick cost calculation makes the multiplier concrete.

Napkin Math 1.1: The serving cost multiplier
Problem: A team has spent $2,000,000 training a 70B parameter model and now serves it to 1,000,000 daily active users, each making 50 requests/day. Is training or serving the dominant cost over 1 year?

Math:

  1. Training Cost: $2,000,000 (One-time).
  2. Metric: Annual inference volume is \(10^{6}\,\text{users} \times 50\,\text{reqs} \times 365\,\text{days} \approx\) 18.25B requests/year.
  3. Cost per Request: Assume a 70B model on H100 costs \(\approx\) $0.001/request (input + output).
  4. Annual Serving Cost: 18.25B requests \(\times\) $0.001/request = $18,250,000.

Systems insight: Serving costs 9.1× more than training in the first year alone. A 10 percent serving-cost or latency-linked efficiency improvement saves about $1.8M in the first year, nearly paying for the original training run.

The total cost of operating a model comprises training cost (a one-time expense) and serving cost (an ongoing expense), as equation 2 shows:

\[C_{\text{total}} = C_{\text{training}} + C_{\text{serving}} \times T_{\text{deployment}} \times Q_{\text{rate}} \tag{2}\]

where \(C_{\text{training}}\) is the one-time cost to train the model, \(C_{\text{serving}}\) is the cost per query served, \(T_{\text{deployment}}\) is the deployment duration in appropriate time units, and \(Q_{\text{rate}}\) is the query rate.

The same leverage holds across application categories with different cost structures. A recommendation model (DLRM) trained for $12,000 ($3000 hardware for 1,000 GPU-hours at $3/GPU-hour, plus 3× additional engineering cost for data and experimentation) and served at 10,000 QPS for 730 days accumulates \(6.31 \times 10^{11}\) lifetime queries, which at $10/million queries costs $6,307,200 to serve, a 525.6× leverage on serving optimization over training optimization. The cost dominance ratio varies by application. Table 3 quantifies this disparity:

Table 3: Training vs. Serving Cost Ratios: High-QPS applications like recommendation systems show extreme serving cost dominance over training (up to 1000\(\times\)), whereas lower-volume LLM APIs and analytics exhibit more moderate ratios (10–100\(\times\)).
Application Training Cost Annual Serving Cost Ratio
Recommendation (high QPS) $10K–$100K $1M–$10M 100–1000\(\times\)
Search ranking $100K–$1M $10M–$100M 100–1000\(\times\)
LLM API $1M–$100M $10M–$1B 10–100\(\times\)
Internal analytics $1K–$10K $10K–$100K 10–100\(\times\)

The cost structure motivates serving optimization: every percentage point of efficiency improvement yields ongoing cost reduction over the model’s operational lifetime. Figure 1 illustrates why optimization matters by showing how the gap between naive and optimized serving determines whether an inference service is profitable or hemorrhaging money.

Figure 1: The Serving Cost Curve: Cost per 1000 tokens decreases with scale and optimization. Single-GPU serving reaches break-even only at high volume. Multi-GPU batching breaks even at moderate volume. Fully optimized serving (INT4, continuous batching, speculative decoding) achieves profitability at much lower volumes, fundamentally changing the economics of inference.

The cost ratios in table 3 tell a static story, but the dynamics of cost accumulation reveal an even more striking pattern. Figure 2 plots cumulative total deployment cost over a 36-month window for three representative scenarios: a small model with low traffic, a 70B LLM serving one million daily active users, and a recommendation system serving 100 million users. The vertical crossover markers show when cumulative serving spend equals the original training cost; because the plotted curves include both training and serving, the total-cost curve is at roughly twice the training baseline at that marker. For recommendation systems, serving dominates within days; for high-traffic LLMs, within roughly six weeks. Only small models with expensive training and low traffic see training cost dominate for extended periods. This temporal view reinforces why inference optimization deserves early engineering attention at production scale. For generative LLM services, serving-specific designs can translate directly into throughput, cost, and power gains (Patel et al. 2024).

Patel, Pratyush, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. “Splitwise: Efficient Generative LLM Inference Using Phase Splitting.” 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 118–32. https://doi.org/10.1109/isca59077.2024.00019.

Figure 2: The Serving Cost Crossover: Cumulative total deployment cost (initial training plus ongoing serving) for three scenarios. Vertical markers indicate when the serving portion equals the one-time training cost. For high-traffic LLMs, serving cost matches training cost within weeks. For recommendation systems at 100M daily active users, serving dominates within days. Only low-traffic, expensive-to-train models see training cost dominate for extended periods.

Inference beyond LLMs

A common misconception equates inference at scale with LLM serving. Large language models present distinctive challenges and attract attention, but they are only one part of production inference. In large consumer platforms, recommendation and ranking workloads can dominate AI inference cycles and capacity (Gupta et al. 2020; Hazelwood et al. 2018). Vision and image processing, NLP and LLM workloads, fraud detection, ads, and other classification tasks fill out the remainder. Table 4 compares these model types by serving pressure and optimization challenge. Recommendation binds on embedding lookup, vision on batch efficiency, LLMs on memory bandwidth, and speech on sequential decode. No single optimization serves all four.

Recommendation systems often dominate high-volume consumer inference because they serve predictions for many user interactions. Every page load, scroll, or click can trigger ranking or retrieval inference. A user browsing an e-commerce site might generate many recommendation requests in a single session. In contrast, LLM queries typically require explicit user action and occur less frequently.

The distribution has direct implications for technique selection. Recommendation systems have driven important production inference innovations: dynamic batching, embedding sharding, feature store architectures, and low-latency serving were all developed primarily for recommendation workloads (Naumov et al. 2019; Gupta et al. 2020). LLM-specific techniques like continuous batching and KV cache management address a different slice of production inference. Text-to-image systems such as DALL-E provide one multimodal model example (Ramesh et al. 2021), but multimodal serving volume and latency targets remain product-specific.

Ramesh, A., M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. 2021. “Zero-Shot Text-to-Image Generation.” In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, edited by Marina Meila and Tong Zhang, vol. 139. Proceedings of Machine Learning Research. PMLR.
Table 4: Production Inference Landscape: Different model types create different serving pressure, latency requirements, and optimization challenges. The table is a qualitative workload map rather than a universal traffic-share distribution. Technique selection must match the specific workload.
Model Type Serving Pressure Latency Target Key Challenge
Recommendation/ranking High volume in consumer platforms \(<10\text{ ms}\) P99 Embedding lookup
Vision (CNN) Workload-dependent 20–100 ms Batch efficiency
LLM High cost per request 100 ms–10s Memory bandwidth
Speech/Audio Stream-driven Real-time Sequential decode
Multimodal/text-to-image Emerging and product-specific Varies Cross-modal coordination

The serving hierarchy

The optimization techniques organize into a Serving Hierarchy, analogous to the memory hierarchy in computer architecture. Each level owns a different bottleneck, so an optimization that helps one level can leave another unchanged. The request level follows one request through preprocessing, batching, caching, and model execution, where the target is the latency a user sees. The replica level looks inside one model instance, where GPU utilization, memory management, kernel efficiency, and model optimization determine how much useful throughput that replica can deliver before it saturates. The service level then works across many replicas of the same model, using load balancing, request routing, and autoscaling to turn individual replicas into aggregate capacity. The platform level works across services and tenants, where resource allocation, multi-tenancy, scheduling, and placement decide whether a shared serving fleet remains efficient without allowing one workload to degrade another.

The hierarchy matters because each level changes a different metric and fails at a different boundary. A related deployment stack appears in figure 3, showing how requests pass through edge, routing, and model-serving infrastructure in production.

Figure 3: Serving Deployment Stack: A three-tier serving stack with cumulative latency budget from top to bottom. Tier 1 (CDN/Edge Cache) handles geographic distribution and cached static responses (SLA <10 ms). Tier 2 (Gateway/Router) handles request routing, rate limiting, and authentication (SLA <50 ms). Tier 3 (Model Serving Cluster) runs GPU workers with autoscaling and continuous batching (SLA <2 s).

Each level has distinct optimization levers, and table 5 shows why the lever is level-specific: request-level changes reduce per-request latency, replica-level changes raise utilization, service-level changes add aggregate capacity, and platform-level changes improve fleet efficiency across tenants. Optimizing the wrong level moves the wrong metric.

Table 5: Serving Hierarchy Optimization Targets: Each level of the serving hierarchy targets distinct performance metrics—from per-request latency at the request tier to aggregate resource efficiency at the platform tier.
Level Optimization Target Key Techniques
Request Per-request latency Dynamic batching, caching, prefetching
Replica Throughput, utilization Memory optimization, kernel fusion
Service Aggregate capacity Load balancing, routing, autoscaling
Platform Resource efficiency Multi-tenancy, scheduling, placement
Checkpoint 1.2: The serving hierarchy

Verify your understanding of where specific optimizations sit within the serving hierarchy:

The hierarchy guides the rest of the design while allowing a few techniques to cross levels. Batching, KV-cache layout, and decode-time optimizations start at the request level. Quantization, adapter state, and model sharding change the replica’s memory and compute budget. Disaggregated serving, load balancing, and autoscaling coordinate replicas into a service, while multi-tenancy and resource isolation govern the platform. Quantization appears across the hierarchy because it is a representation-level lever that changes memory, bandwidth, and cost budgets wherever model state or KV state resides.

Serving Architecture Dimensions

A recommendation system that processes 100,000 embedding lookups per second across a sharded feature store requires a different serving architecture from a 70-billion-parameter language model generating tokens one at a time. Framework choice alone does not explain the difference. The workloads impose distinct constraints on batching, memory management, scheduling, and deployment topology. This section identifies those architectural dimensions, using specific frameworks as examples rather than as the subject of study.

Batching strategy: The throughput-latency trade-off

The most consequential architectural decision in a serving system is how it forms batches from incoming requests. This choice determines the fundamental throughput-latency operating point.

Static Batching collects a fixed number of requests before dispatching them to the accelerator. For vision models processing fixed-size inputs, this approach maximizes GPU utilization because all requests in the batch execute identical computation graphs with predictable memory access patterns. A ResNet-50 inference server can batch 32 or 64 images with near-linear throughput scaling because the batch amortizes fixed launch overhead and reuses resident weights across many inputs.

Autoregressive language models break this assumption. Each request generates a different number of output tokens, so static batching forces all requests to wait for the longest generation in the batch and can leave accelerator capacity idle after shorter requests finish. Continuous batching solves this by allowing requests to enter and exit the batch at each decoding step, an iteration-level scheduling approach introduced by Orca (Yu et al. 2022). When one request finishes generation, a new request immediately fills its slot. Systems like vLLM6 combine this scheduling style with PagedAttention-based KV-cache management, achieving 2–4\(\times\) throughput gains over prior LLM serving systems by reducing memory waste (Kwon et al. 2023).

6 vLLM (Virtual Large Language Model): The name signals its core design: applying OS-style virtual memory to KV cache management, decoupling logical sequence addresses from physical GPU memory. Developed at UC Berkeley (2023), vLLM achieved 2–4\(\times\) throughput over FasterTransformer and up to 24\(\times\) over HuggingFace Transformers by eliminating the 60–80 percent memory fragmentation that capped batch sizes in prior serving systems (Kwon et al. 2023).

Kwon, Woosuk, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” Proceedings of the 29th Symposium on Operating Systems Principles, 611–26. https://doi.org/10.1145/3600006.3613165.

Recommendation systems introduce a third pattern: Feature-parallel Batching. Because the bottleneck is distributed embedding lookup rather than dense matrix multiplication, these systems batch requests by feature type and shard the batch across embedding servers. The dense MLP computation that follows can then operate on prefetched, prebatched feature vectors. The batching strategies section (section 1.2) develops these approaches in full quantitative detail.

Memory management: From preallocation to paging

Serving systems differ fundamentally in how they manage accelerator memory, and this difference determines the maximum concurrent request capacity.

Definition 1.1: KV cache

KV Cache is a per-request LLM inference memory buffer that stores the Key and Value attention tensors of all previously generated tokens so the next token can be produced without recomputing the full attention over the prefix.

  1. Significance: It reduces per-token attention compute from \(O(n^2)\) to \(O(n)\) in the sequence length, but its footprint grows linearly with both context length and batch size, often consuming more HBM than the model weights themselves and becoming the binding constraint on serving capacity. KV cache fundamentals derives the fundamental sizing math.
  2. Distinction: Unlike model weights (which are shared across all requests and statically allocated), the KV cache is request-private and dynamically sized; each concurrent user pays a separate, sequence-length-proportional memory tax that the scheduler must budget for.
  3. Common pitfall: A frequent misconception is that reducing model size proportionally reduces serving memory. In practice, the KV cache dominates HBM at long contexts, so a quantized 70B model can still be capacity-bound at batch sizes the weights alone would easily admit.

Preallocated Memory Management reserves a fixed memory budget per request at admission time based on the maximum possible output length. For a model supporting 4,096-token outputs, every request reserves memory for 4,096 tokens regardless of whether it generates 50 or 4,000. This approach is simple and predictable but wastes memory proportional to the gap between maximum and actual output lengths. In practice, 60–80 percent of reserved KV cache memory goes unused.

Paged Memory Management, inspired by operating system virtual memory, allocates memory in fixed-size blocks (pages) and maps logical sequence positions to physical memory locations through a block table. As a request generates tokens, new physical blocks are allocated on demand. When generation completes, blocks return to the free pool immediately. This approach, exemplified by PagedAttention (covered in section 1.3.3), achieves near-100 percent memory utilization by eliminating both internal fragmentation (partially filled preallocations) and external fragmentation (unusable gaps between allocations).

The memory management strategy directly determines batch capacity: paged systems can serve 2–4\(\times\) more concurrent requests than preallocated systems on the same hardware, because they reclaim memory that preallocation wastes. For a service with a fixed GPU budget, this translates directly to 2–4\(\times\) lower cost per request.

Scheduling policy: FCFS, preemptive, and priority-aware

The scheduling policy determines which requests receive GPU time and in what order. This decision becomes critical when request mix is heterogeneous, with some requests requiring 50 tokens of output and others requiring 4,000. First-Come-First-Served (FCFS) Scheduling processes requests in arrival order. FCFS is fair and simple but suffers from head-of-line blocking: a single long-generation request delays all subsequent requests. For workloads with high output-length variance, FCFS produces poor tail latency.

Preemptive Scheduling allows the system to pause a long-running request and swap its KV cache to CPU memory (or discard it for later recomputation) to make room for shorter, higher-priority requests. The cost of preemption is the swap overhead (transferring KV cache between GPU and CPU memory) or the recomputation cost (re-running the prefill phase when the preempted request resumes). Production systems typically preempt when a request has consumed more than 2\(\times\) the median generation length.

Priority-Aware Scheduling assigns different service classes to requests and guarantees that high-priority requests receive GPU slots before lower-priority ones. A production API might classify revenue-generating customer requests as critical, internal batch processing as standard, and free-tier traffic as best-effort. The scheduling policy then ensures that critical requests never wait behind best-effort traffic, even during load spikes.

Deployment topology: Single-GPU to disaggregated

The deployment topology determines how model computation maps to physical hardware, and this mapping is driven by the ratio of model size to single-device memory capacity. Single-GPU deployment is the simplest topology: the entire model fits in one device’s memory, and all inference computation occurs locally. For models under 15–20 billion parameters (30–40 GB in FP16), this topology provides the lowest latency because it eliminates all inter-device communication.

Multi-GPU tensor parallelism shards each layer’s weight matrices across multiple GPUs connected by high-bandwidth interconnect (NVLink at 900 GB/s bidirectional [450 GB/s per direction] on H100). Each GPU computes a partial result for every layer, and an all-reduce operation synchronizes the partial results before the next layer. This topology is necessary when model weights exceed single-device memory and provides latency reduction proportional to the parallelism degree, at the cost of communication overhead per layer. The model sharding section (section 1.4) analyzes the communication overhead quantitatively.

Disaggregated serving separates the prefill phase (processing the input prompt, which is compute-bound) from the decode phase (generating tokens one at a time, which is memory-bandwidth-bound) onto different hardware pools optimized for each workload profile. Prefill nodes use high-FLOP/s accelerators; decode nodes use high-bandwidth-memory configurations. This separation allows each phase to operate at its hardware’s optimal operating point.

Stateful vs. stateless: The scaling divide

A serving system’s statefulness determines whether horizontal scaling is trivial or requires careful engineering. Vision models and embedding lookups are typically stateless: any replica can serve any request because no per-session state persists between requests. Horizontal scaling is straightforward – add replicas behind a load balancer – and failure recovery is instant because the load balancer routes to a healthy replica.

LLM serving with KV cache is inherently stateful: the cache accumulated during a conversation creates replica-specific state that cannot be reconstructed without re-running the entire conversation history through prefill. This statefulness has cascading implications for system design. Load balancing requires sticky routing to direct subsequent requests in a conversation to the same replica. Autoscaling down requires draining active sessions, which can take minutes for long conversations. Failure recovery is expensive: when a stateful replica crashes, users either experience the latency of regenerating context (seconds) or lose conversation state entirely.

The choice between stateless and stateful serving is not a framework feature but a consequence of the model architecture. Systems that serve autoregressive models must engineer for statefulness; systems that serve fixed-computation models can treat scaling as a simpler capacity-planning exercise.

Architectural comparison

Table 6 summarizes how the batching, memory, scheduling, topology, and state dimensions interact across the major workload types. The table reveals that no single serving architecture is optimal for all workloads; the constraint profile of each workload type determines the appropriate design point along each dimension.

Table 6: Serving Architecture Dimensions by Workload Type: Each workload’s constraint profile determines the optimal design point along five architectural dimensions. Selecting a serving system amounts to choosing the combination that matches the workload’s constraints.
Dimension Vision/Embedding LLM (Autoregressive) Recommendation
Batching Static/dynamic (uniform inputs) Continuous (variable outputs) Feature-parallel (sharded embeddings)
Memory Preallocated (predictable) Paged (variable KV cache) Distributed (embedding tables)
Scheduling FCFS (uniform cost) Preemptive (high variance) Priority-aware (SLO tiers)
Topology Single-GPU replicas Tensor/pipeline parallel Hybrid CPU-GPU sharding
State Stateless Stateful (KV cache) Stateless (feature store)
Checkpoint 1.3: Serving dimensions

Verify your understanding of how workload constraints drive architectural choices:

The architectural dimensions determine which serving system to select for a given workload. Rather than comparing frameworks feature by feature, an engineer identifies the workload’s position along each dimension (batching pattern, memory profile, scheduling requirements, deployment topology, and statefulness) and selects the system that matches. The optimization techniques in this chapter operate within this architectural framework, each addressing a specific dimension.

Self-Check: Question

  1. Which combination of architectural design choices correctly reflects the requirements of autoregressive large language model (LLM) serving compared to vision and recommendation workloads?

    1. Continuous batching, paged memory management, preemptive scheduling, and stateful routing
    2. Static batching, preallocated memory management, first-come-first-served (FCFS) scheduling, and stateless routing
    3. Feature-parallel batching, distributed memory tables, priority-aware scheduling, and stateless routing
    4. Streaming frame batching, pinned buffer allocation, non-preemptive scheduling, and edge-only topology

  2. Why does preallocated memory management in LLM serving waste 60% to 80% of GPU KV cache memory, and how does paged memory management solve this problem?

  3. True or False: The statefulness of an inference serving system is an optional configuration feature of the serving framework rather than an inherent property of the model architecture.

  4. Why is pure First-Come-First-Served (FCFS) scheduling problematic for multi-tenant LLM serving, and what mechanism does preemptive scheduling use to resolve it?

    1. FCFS requires excessive GPU memory for block tables; preemptive scheduling replaces block tables with static pointers.
    2. FCFS causes extreme AllReduce communication overhead; preemptive scheduling converts tensor parallelism to pipeline parallelism.
    3. FCFS forces all requests to run at INT4 precision; preemptive scheduling dynamically promotes high-priority requests to FP16.
    4. FCFS suffers from head-of-line blocking when long generations delay short queries; preemptive scheduling swaps long-running KV caches to CPU DRAM to free GPU slots.

  5. The scheduling technique that decouples batch membership from request boundaries and evaluates sequence entry and termination at every decode iteration is called ____ batching (or iteration-level scheduling).

See Answers →

Batching Strategies at Scale

A stream of hundreds of disparate chat requests arrives at an inference server every second. Processing them one by one leaves much of the GPU idle, starved for work between small decode steps. Waiting to gather a large static batch forces the first request in line to endure unacceptable latency. Batching strategies at scale demand dynamic algorithms that fuse requests on the fly without violating strict tail-latency deadlines.

Processing multiple requests together amortizes fixed costs (model loading, kernel launch overhead, and memory transfer latency) across more useful work, trading higher per-request latency for dramatically improved throughput. Single-machine serving applies this insight through Dynamic Batching, which collects requests within a time window before processing them together.

At scale, batching becomes more complex because different model architectures have distinct batching requirements. A strategy optimal for vision models may be catastrophic for LLMs, and techniques developed for recommendation systems may not apply to either.

In large consumer platforms, recommendation systems can constitute the majority of AI inference cycles or capacity pressure, with vision, language, ranking, fraud, ads, and other classification workloads sharing the remainder (Gupta et al. 2020; Hazelwood et al. 2018). Despite this distribution, batching strategies are presented in order of conceptual complexity: vision (straightforward batching), LLMs (continuous batching with KV cache), and recommendation (feature-parallel batching with distributed embedding). This ordering builds understanding progressively, even though production systems frequently encounter recommendation workloads first. The taxonomy that follows matches batching strategies to model characteristics, providing quantitative analysis of when each approach applies and what performance to expect.

Why batching differs across model types

Batching efficiency depends on how computation scales with batch size relative to how memory and communication scale. Different model architectures exhibit different scaling relationships, requiring different batching strategies, as figure 4 summarizes.

Figure 4: Batching Strategies Compared: Three timelines show how batching policies trade wait time for GPU utilization. (a) No batching: requests processed sequentially, GPU idle between them (around 40 percent utilization). (b) Static batching: requests wait until a full batch is assembled before dispatch, with padding for short requests (around 70 percent utilization). (c) Dynamic batching: assembles a batch within a short time window with adaptive size, reducing wait and lifting utilization (around 85 percent).

For Vision Models, including convolutional neural networks (CNNs) and vision transformers (ViTs) processing fixed-size images, computation scales linearly with batch size while memory scales sub-linearly due to weight sharing. Larger batches improve GPU utilization with minimal overhead, making static or dynamic batching with large batch sizes optimal.

For LLMs in the decode phase, computation per token is small relative to memory bandwidth requirements for loading model weights. The bottleneck is memory bandwidth, not compute. Larger batches amortize weight loading across more tokens, dramatically improving throughput but with diminishing returns as batch size grows.

For recommendation systems, the bottleneck is often embedding lookup rather than dense computation. Batching strategies must optimize for parallel embedding access patterns rather than matrix multiplication throughput.

The physics of batching: The efficiency curve

Batching is not merely a heuristic; it is a trade-off governed by the physics of hardware utilization. Modeling the relationship between batch size \((B)\), request latency \((T_{\text{lat}})\), and throughput \((X)\) identifies the optimal operating point for an inference system.

Here \(T_{\text{lat}}\) follows standard queuing notation for request time in system. Elsewhere in the book, \(L_{\text{lat}}\) denotes fixed latency overheads or latency components; this section keeps \(T_{\text{lat}}\) for the end-to-end request-time variable used in Little’s Law and batching equations.

The latency equation decomposes per-request latency into fixed overheads (kernel launch, memory loading) and variable costs (compute per sample):

\[T_{\text{lat}}(B) = T_{\text{fixed}} + B \times T_{\text{variable}}\]

  • \(T_{\text{fixed}}\): Costs paid once per batch (for example, loading weights from HBM, kernel launch latency).
  • \(T_{\text{variable}}\): Marginal cost of adding one request (for example, compute time for that sample).

The throughput equation describes the system’s capacity:

\[X(B) = \frac{B}{T_{\text{lat}}(B)} = \frac{B}{T_{\text{fixed}} + B \times T_{\text{variable}}}\]

The resulting Batching Efficiency Curve shows three distinct regimes. For small batch sizes (\(B\)), throughput is dominated by \(T_{\text{fixed}}\), making the system latency-bound (or overhead-bound) where increasing \(B\) yields large but diminishing throughput gains. As \(B\) becomes large, the \(T_{\text{fixed}}\) term becomes negligible, and throughput asymptotically approaches the hardware limit \(1/T_{\text{variable}}\), leaving the system compute-bound (or bandwidth-bound for LLMs). The optimal batch size sits at the knee of the curve, the point where throughput gains diminish while latency continues to grow linearly.

A throughput curve that rises steeply then plateaus, with a knee dot marking the optimal batch size. A red wash shades the region past the knee, where larger batches add latency without gaining throughput.

Throughput saturates at the batch size where latency hits the SLO.

The engineering goal is to find the maximum \(B\) such that \(T_{\text{lat}}(B) \le \text{SLO}\). This formulation explains why vision models (high \(T_{\text{variable}}\)) saturate at smaller batches than LLMs (high \(T_{\text{fixed}}\) due to weight loading), requiring different tuning strategies.

Napkin Math 1.2: The batching efficiency curve
Problem: An engineer optimizes two services: a vision model (ResNet) and an LLM (70B). At what batch size does each hit the “knee” of its efficiency curve?

Math: The knee occurs where the variable compute cost \((B \times T_{\text{var}})\) starts to exceed the fixed overhead \((T_{\text{fixed}})\).

  1. Vision Model \((T_{\text{fixed}} = 2\text{ ms}, T_{\text{var}} = 1\text{ ms})\):
    • Knee: \(B \approx 2\text{ ms}/1\text{ ms} =\) 2.
    • Result: This simplified latency model reaches its first overhead-amortization knee at a small batch. Production CNNs often continue gaining throughput until batches of 32–64+ before hardware saturation.
  2. LLM Decode \((T_{\text{fixed}} = 40\text{ ms}, T_{\text{var}} = 0.5\text{ ms})\):
    • Knee: \(B \approx 40\text{ ms}/0.5\text{ ms} =\) 80.
    • Result: LLMs require massive batches to amortize the expensive weight-loading overhead.

Systems insight: Because the LLM’s “fixed overhead” (loading 140 GB of weights from HBM) is so large, the system has not reached the efficiency knee until batch size 80. LLM serving therefore requires continuous batching and paged memory: the system must pack hundreds of concurrent users into a single batch just to overcome the memory bandwidth bottleneck.

Translating these batch-level equations into system-wide capacity planning requires a classical result from queuing theory: Little’s Law,7 which relates concurrency, throughput, and latency in any stable system. Across model architectures, table 7 shows that batching choice follows the binding bottleneck: compute for vision models, memory capacity or bandwidth for LLM prefill and decode, embedding lookup for recommendation systems, and real-time latency for speech.

7 Little’s Law: From queuing theory, \(Q_{\text{req}} = \lambda_{\text{arr}} T_{\text{lat}}\), so concurrency equals arrival rate times latency. For a serving fleet, this defines the capacity envelope: if a GPU can handle 32 concurrent requests and each takes 100 ms, the maximum arrival rate is 320 req/s. Beyond this, queues grow rapidly as utilization approaches 1, and tail latency \((L_{\text{lat}})\) explodes.

Table 7: Batching Strategy by Model Type: Each model class requires a distinct batching policy (from static/dynamic in vision to continuous in LLM decode and feature-parallel in recommendation) matched to its dominant hardware bottleneck.
Model Type Batching Strategy Typical Batch Size Key Constraint Throughput Scaling
Vision (CNN) Static/Dynamic 32–256 GPU compute Near-linear to 64+
LLM (prefill) Dynamic 1–64 Memory capacity Sub-linear
LLM (decode) Continuous 100–1000s Memory bandwidth Log-linear
RecSys Feature-parallel 1000–10000s Embedding lookup Depends on sharding
Speech Streaming 1 Real-time N/A (latency-bound)

Theorem 1.1: Little's Law for inference
Concept: In any stable queuing system, the average number of requests in the system \((Q_{\text{req}})\) equals the arrival rate \((\lambda_{\text{arr}})\) multiplied by the average time a request spends in the system \((T_{\text{lat}})\).

\[ Q_{\text{req}} = \lambda_{\text{arr}} \cdot T_{\text{lat}} \]

Application: Concurrency Planning

  • Target Throughput \((\lambda_{\text{arr}})\): 1,000 requests/sec
  • Latency SLO \((T_{\text{lat}})\): 100 ms (0.1 s)

Required Concurrency \((Q_{\text{req}})\): \(Q_{\text{req}} = 1000 \times 0.1 =\) 100 concurrent requests

Capacity Planning: If a single GPU replica handles batch size 8 with 80 ms latency:

  1. Replica Throughput \(=\) \(8/0.08 =\) 100 req/s
  2. Throughput-only lower bound \(=\) \(\lceil 1000/100\rceil =\) 10 replicas
  3. SLO-sized replica count \(=\) \(\lceil 100/8\rceil =\) 13 replicas

Verification: Total system concurrency \(=\) \(13\text{ replicas} \times 8\text{ batch} =\) 104, which leaves 4 request slots above the 100 required by Little’s Law. The throughput-only lower bound would run at the stability boundary; the SLO-sized fleet runs at about 76.9 percent service utilization, leaving finite headroom for queueing variance and tail latency.

The distinction between throughput capacity and latency capacity is exactly what queuing theory formalizes; Queuing theory for batched inference derives the M/D/1 wait-time distribution that determines how much headroom a serving SLO needs.

Static and dynamic batching for vision models

Vision models represent the simplest batching case because inputs have uniform size (after preprocessing) and computation follows a predictable pattern. Single-machine batching principles apply directly, with scale introducing considerations of batch formation across multiple replicas.

Static batching collects exactly \(B\) requests before processing. This maximizes GPU utilization when request arrival is predictable but causes unbounded latency during low-traffic periods.

Dynamic batching collects requests for a maximum time window \(T_{\text{window}}\) or until reaching maximum batch size \(B_{\text{max}}\), whichever occurs first. The expected latency under Poisson arrivals with rate \(\lambda_{\text{arr}}\) follows equation 3:

\[E[T_{\text{total}}] = E[T_{\text{queue}}] + T_{\text{batch}} + T_{\text{inference}}(B) \tag{3}\]

where \(E[T_{\text{queue}}]\) is the expected queuing delay, \(T_{\text{batch}}\) is the batch formation delay (up to \(T_{\text{window}}\)), and \(T_{\text{inference}}(B)\) is the inference time for batch size \(B\). The arrival rate enters through the queuing and formation terms: under Poisson arrivals the mean inter-arrival gap is \(1/\lambda_{\text{arr}}\), so a higher \(\lambda_{\text{arr}}\) fills the batch faster and shrinks both \(E[T_{\text{queue}}]\) and \(T_{\text{batch}}\), while a lower rate pushes them toward the \(T_{\text{window}}\) ceiling. The worked example that follows makes these interactions concrete.

Example 1.1: Dynamic batching for ResNet-50
Scenario: A vision classification service receives requests at 5,000 QPS with a 50 ms P99 latency SLO across 10 replicas.

Diagnosis: Serving requests individually (no batching) overloads the system (\(\rho_{\text{serv}} = \lambda_{\text{arr}} \times T_{\text{svc}} = 500 \times 0.005 = 2.5\) utilization). Implementing dynamic batching windows (Option B: 10 ms, Option C: 20 ms) packs requests to lower per-request compute time, achieving up to 33 percent higher per-replica capacity within the latency SLO.

Systems lesson: Dynamic batching is effective when the latency budget has unused slack. The serving policy converts latency headroom into throughput, but requires careful tuning to prevent SLO violations.

At scale with multiple replicas, batch formation can occur either at individual replicas or at a centralized batching layer. Replica-local batching has each replica independently form batches from its assigned traffic. This approach is simpler to implement but may result in uneven batch sizes across replicas when load is imbalanced. Centralized batching uses a batching service to collect requests and dispatch formed batches to replicas. This achieves more uniform batch sizes but adds a centralization bottleneck and additional network hop. Production systems typically use replica-local batching with load balancing that ensures roughly equal traffic distribution, achieving the benefits of centralized batching without the complexity.

Continuous batching for LLM inference

The autoregressive bottleneck (principle 14) governs this regime: in generative models, the decode phase is strictly memory-bandwidth bound because the entire model weight set must be loaded for every single token generated. Throughput scales with batch size, sharing weight loads across multiple requests, not compute power.

Definition 1.2: Continuous batching

Continuous Batching is a serving strategy that decouples batch membership from iteration boundaries, allowing new requests to enter and completed ones to exit at every decode step.

  1. Significance: It maximizes system throughput \((X)\) by eliminating the padding waste and head-of-line blocking inherent in static batching. It ensures the GPU remains saturated even when requests have widely varying sequence lengths.
  2. Distinction: Unlike static or dynamic batching (which group requests at the request level), continuous batching operates at the Iteration Level, dynamically reshaping the compute tensor at each clock cycle.
  3. Common pitfall: A frequent misconception is that continuous batching is “purely a scheduler change.” In reality, it requires a Dynamic Memory Manager (like PagedAttention) because the KV caches for different requests grow and shrink at different rates, preventing static memory preallocation.

Decoupling batch membership from iteration boundaries is slot reuse: when one request reaches its stop token, the scheduler hands its KV-cache slot to a waiting request on the next decode step instead of leaving capacity idle until the longest sequence finishes.

Autoregressive language models present a unique batching challenge that static and dynamic approaches handle poorly. The Orca system8 (Yu et al. 2022) exposes the limitation. Traditional batching forces all sequences in a batch to complete before any new sequences can join, wasting compute when sequences finish at different times.

8 Orca (Iteration-Level Scheduling): Introduced by Yu et al. (2022), Orca demonstrated that scheduling at iteration granularity rather than request granularity allows sequences to enter and exit batches at each decode step. This eliminated the head-of-line blocking that wasted 50 percent+ of GPU compute in static batching and established the scheduling paradigm adopted by LLM serving systems such as vLLM.

Yu, Gyeong-In, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. “Orca: A Distributed Serving System for Transformer-Based Generative Models.” 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 521–38.

Consider a batch of 8 sequences. If one sequence completes after 10 tokens while others require 100 tokens, the completed sequence’s GPU resources sit idle for 90 iterations. With traditional batching:

\[\text{Wasted compute} = \frac{(100 - 10) \times 1}{100 \times 8} = 11.25\%\]

For realistic output length distributions with high variance, wasted compute can exceed 50 percent. Continuous batching (also called iteration-level batching) decouples batch membership from iteration boundaries. Algorithm 1 states the scheduler as a decode loop: completed requests release KV-cache pages, waiting requests enter freed slots, and the next batched kernel runs over the reorganized active set. This technique is central to the throughput-latency trade-off that defines large-scale LLM serving—Archetype A workloads (GPT-4/Llama-3, Three systems archetypes) would be economically unviable without it, because their memory-bandwidth-bound decode phase leaves compute cores idle waiting for weights to load, and only by interleaving unrelated requests can the bandwidth be saturated.

\begin{algorithm} \caption{Continuous batching decode scheduler} \begin{algorithmic} \Require waiting queue $Q$; active set $A$; KV-cache page allocator; active-token capacity $C_{\text{tok}}$; batched decode kernel \Ensure generated tokens for completed requests; a continually refreshed active batch \While{the service is running} \State admit requests from $Q$ while capacity and free KV-cache pages remain; allocate state, append to $A$ \State run one batched decode kernel over all sequences in $A$ \State append each sampled token; extend its KV-cache pages \State remove completed or length-limited sequences from $A$; return their pages to the allocator \If{HBM pressure exceeds the policy threshold} \State preempt low-priority sequences by paging their KV-cache to CPU DRAM \EndIf \State fill freed capacity from $Q$, or resume paused sequences whose pages now fit \EndWhile \end{algorithmic} \end{algorithm}

Admitting, completing, and refilling slots on every iteration keeps active-token capacity from idling behind the longest sequence in the batch, while the preemption path adds HBM-to-CPU-DRAM transfer latency under memory pressure. The throughput gain therefore scales with output-length variance and must be balanced against KV-cache movement. Static batching leaves the GPU idle while waiting for the longest request in a batch, whereas continuous batching keeps accelerator execution saturated (figure 5).

Figure 5: Static vs. Continuous Batching: In static batching (A), all requests in a batch must wait for the longest request to complete before the GPU can begin the next batch, leading to significant idle compute time (shaded gray). Continuous batching (B) allows new requests to enter the batch as soon as any request finishes, keeping the GPU saturated and dramatically improving throughput.

Continuous batching throughput analysis

The contrast in figure 5 motivates the throughput analysis: static batching leaves slots idle until the longest sequence in a batch finishes, while continuous batching can refill those slots at iteration boundaries. Output-length variance creates the opportunity, but it does not determine the throughput gain by itself. The realized gain also depends on batch capacity, the full length distribution, request arrivals, KV-cache admission, prefill scheduling, and scheduler overhead. Consequently, there is no workload-independent gain formula based only on the coefficient of variation (CV), \(\text{CV} = \sigma / \mu\); serving teams must compare policies under the same arrival trace, latency objective, and memory budget.

The implementation study in example 1.2 examines how vLLM realizes continuous batching through iteration-level scheduling, paged memory management, preemption, throughput, and GPU utilization.

Example 1.2: Continuous batching in vLLM
Scenario: An LLM serving engine (vLLM) manages autoregressive generation for incoming requests with highly variable output sequence lengths (Kwon et al. 2023).

Diagnosis: Static batching leaves compute cores idle for over 50 percent of decode steps waiting for the longest request in a batch to finish. Continuous batching evaluates sequence termination at every iteration step, immediately releasing completed KV-cache pages and admitting waiting requests. Under memory pressure, low-priority requests are swapped to CPU DRAM.

Systems lesson: Continuous batching decouples batch membership from request-level boundaries. Combining iteration-level scheduling with dynamic paged memory (PagedAttention) yields 2–4\(\times\) throughput gains over static batching.

The throughput numbers establish that continuous batching wins when output lengths vary, but they do not show where the gain comes from or how large it can grow. Quantifying the waste that traditional batching leaves on the table turns that intuition into a number a scheduler can act on.

Quantitative analysis: Traditional vs. continuous batching

The first baseline is traditional LLM batching, where unequal output lengths turn request heterogeneity into wasted decode cycles. The mathematics of batching waste reveals exactly how much throughput is lost and under what conditions continuous batching delivers the greatest improvement.

The waste function for traditional batching

Traditional batching (also called static batching) processes all requests in a batch through all decode iterations until the longest sequence completes. For batch size \(B\) with output lengths \(\{S_1, S_2, ..., S_B\}\), the total compute performed is:

\[O_{\text{traditional}} = B \times S_{\text{max}} \times c_{\text{decode}}\]

where \(S_{\text{max}} = \max_i(S_i)\) and \(c_{\text{decode}}\) is the compute cost per decode iteration per sequence. However, the useful compute is only:

\[O_{\text{useful}} = \sum_{i=1}^{B} S_i \times c_{\text{decode}}\]

Equation 4 defines the Waste Ratio that quantifies the inefficiency:

\[W = 1 - \frac{O_{\text{useful}}}{O_{\text{traditional}}} = 1 - \frac{\sum_{i=1}^{B} S_i}{B \times S_{\text{max}}} = 1 - \frac{\bar{S}}{S_{\text{max}}} \tag{4}\]

where \(\bar{S}\) is the mean output sequence length. This reveals that waste depends entirely on the ratio of mean to maximum output length within the batch. For uniform output lengths \((\bar{S} = S_{\text{max}})\), waste is zero. For highly variable lengths, waste can exceed 50 percent.

Worked example: LLM serving with variable-length outputs

Consider a GPT-class model serving four concurrent requests with the generation lengths shown in table 8 (in tokens):

Table 8: Concurrent Request Generation Lengths: Four concurrent requests on a GPT-class model with varying prompt and output lengths. The 4\(\times\) spread in output length (50 to 200 tokens) drives the batching waste analyzed in this section: a static batch of four pads every request to the longest, wasting compute on the three shorter requests until the longest finishes generating.
Request Prompt Length Output Length Total Tokens
R1 100 50 150
R2 80 200 280
R3 120 100 220
R4 90 150 240

Napkin Math 1.3: Continuous batching waste calculation
Problem: Given four LLM requests with unequal output lengths, quantify how much average latency traditional batching wastes and how continuous batching recovers that idle slot time.

Variables:

  • Decode time per iteration (batch of 4): 20 ms
  • Maximum output length in batch: 200 tokens (R2)
  • Mean output length: \((50 + 200 + 100 + 150) / 4 =\) 125 tokens

Traditional batching: All four requests must wait for R2 to complete its 200 tokens.

  • Total decode iterations: 200
  • Total batch time: \(200 \times 20\text{ ms} =\) 4,000 ms
  • Request completion times:
    • R1 completes useful work at iteration 50, but waits until iteration 200 → latency = 4,000 ms
    • R2 completes at iteration 200 → latency = 4,000 ms
    • R3 completes useful work at iteration 100, but waits until iteration 200 → latency = 4,000 ms
    • R4 completes useful work at iteration 150, but waits until iteration 200 → latency = 4,000 ms

Waste calculation using equation 4:

\(W = 1 - \frac{125}{200} = 1 - 0.625 = 37.5\%\)

The GPU performs \(4 \times 200 =\) 800 “sequence-iterations” but only 500 are useful.

Continuous batching: Sequences depart the batch upon completion, and new requests can join.

  • Iteration 50: R1 completes → slot freed, new request R5 can join
  • Iteration 100: R3 completes → slot freed, new request R6 can join
  • Iteration 150: R4 completes → slot freed, new request R7 can join
  • Iteration 200: R2 completes

Request latencies with continuous batching (assuming no queuing delay):

  • R1: \(50 \times 20\text{ ms} =\) 1,000 ms (4× improvement over traditional)
  • R3: \(100 \times 20\text{ ms} =\) 2,000 (2× improvement)
  • R4: \(150 \times 20\text{ ms} =\) 3,000 (1.33× improvement)
  • R2: \(200 \times 20\text{ ms} =\) 4,000 (no improvement for longest request)

Average latency comparison

  • Traditional: 4,000 ms (all requests)
  • Continuous: (1,000 ms + 2,000 + 3,000 + 4,000) / 4 = 2,500

Result: Continuous batching reduces average latency by 37.5 percent, exactly matching the waste ratio.

Systems insight: Continuous batching does not make the longest request faster; it prevents shorter requests from occupying finished slots while they wait for that longest request. The gain comes from reusing slots at iteration boundaries, which is the slot-reuse pattern shown in figure 5.

When continuous batching provides maximum benefit

The continuous-batching analysis reveals that benefit scales with output length variance. A useful upper-bound intuition comes from comparing the longest sequence in a traditional batch with the average sequence length. Let \(\text{CV} = \sigma / \mu\) be the coefficient of variation of output lengths. If the longest request in a batch is roughly \(k\) standard deviations above the mean, then:

\[\text{Improvement} \approx \frac{S_{\text{max}}}{\bar{S}} = \frac{\mu + k\sigma}{\mu} = 1 + k \cdot \text{CV}\]

where \(k\) is the number of standard deviations the maximum output exceeds the mean. Real systems realize less than this upper bound because scheduler overhead, KV-cache pressure, and refill gaps consume part of the theoretical gain. The table therefore reports effective waste and speedup, where \(\text{speedup} \approx 1/(1 - W_{\text{effective}})\).

Table 9 quantifies this relationship across different workload types, including retrieval-augmented generation (RAG) (Lewis et al. 2020):

Lewis, Patrick, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, et al. 2020. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Advances in Neural Information Processing Systems 33: 9459–74.
Table 9: Continuous Batching Benefit by Workload Type: Higher output length variance (CV) yields greater improvement from continuous batching. Workloads with predictable output lengths (code completion) see modest gains, while highly variable workloads (RAG with documents of varying length) see dramatic improvement.
Workload Type CV k Waste (Trad.) Speedup (Continuous)
Code completion 0.3 2.5 16.7% 1.2×
Chat (short responses) 0.6 2 33.3% 1.5×
General text generation 1 2.5 50% 2×
Creative writing 1.5 3 64.3% 2.8×
RAG with variable docs 2 2.5 71.4% 3.5×

The systems pattern is straightforward: continuous batching is most valuable when output lengths are unpredictable, mixed workload types share the same cluster, request volume is high enough to refill vacated slots, and latency SLOs make early completion valuable. It provides minimal benefit when output lengths are uniform, as in classification or embedding generation, when batch sizes are too small for slot reuse to matter, or when request volume is too low to refill vacated slots.

Implementation complexity trade-offs

The same variability that makes continuous batching valuable also makes it harder to operate. Its performance benefits come with implementation complexity that systems engineers must weigh.

Memory management complexity increases substantially. Traditional batching allocates a fixed KV cache region per sequence at batch formation, deallocating only when the entire batch completes. Continuous batching requires dynamic allocation as sequences grow and immediate deallocation upon completion, necessitating sophisticated memory management akin to operating system virtual memory.

Scheduler complexity rises correspondingly. Traditional batching uses simple FIFO scheduling: collect requests until the batch is full or the timeout expires, then execute. Continuous batching requires per-iteration decision-making about which sequences to admit, which to preempt if memory pressure exists, and how to handle priority classes. This increases scheduler overhead from \(\mathcal{O}(1)\) per batch to \(\mathcal{O}(B)\) per iteration.

Kernel design must also adapt. Batched GPU kernels traditionally assume fixed batch composition. Continuous batching requires kernels that handle variable-length sequences efficiently, often through techniques like packing multiple short sequences into shared attention masks or using specialized memory layouts that support dynamic batch membership.

Table 10 summarizes these trade-offs:

Table 10: Traditional vs. Continuous Batching Trade-Offs: Continuous batching provides significant throughput gains for variable-length workloads at the cost of implementation complexity. For uniform-length workloads, the complexity overhead may not justify adoption.
Dimension Traditional Batching Continuous Batching
Implementation effort Low (standard frameworks) High (custom scheduler, kernels)
Memory overhead Fixed allocation Dynamic + fragmentation mgmt
Scheduler latency ~0.1 ms per batch ~0.5-1 ms per iteration
Debugging complexity Deterministic behavior State-dependent, harder to trace
Throughput (variable) Baseline 1.5–3.5\(\times\) improvement
Throughput (uniform) Baseline ~1.0\(\times\) (no improvement)

For new LLM serving deployments, continuous batching frameworks like vLLM, TensorRT-LLM, or Text Generation Inference provide mature implementation paths. The decision becomes whether to adopt these frameworks or build custom serving infrastructure. For organizations with existing traditional batching systems, the migration cost must be weighed against the workload’s output length variance using table 9.

Even with mature serving frameworks, diagnosing tail latency anomalies requires systematic investigation across the system stack. A representative P99 latency regression shows how the fleet stack methodology applies across infrastructure, execution, and serving layers.

Example 1.3: Debugging high P99 latency
Scenario: An LLM serving system exhibits high tail latency: P50 = 100 ms and P95 = 180 ms maintain SLO, but P99 spikes to \(L_{\text{lat}} =\) 500 ms against a 200 ms target despite GPU utilization \(\eta_{\text{hw}} =\) 85 percent.

Diagnosis: The system runs across 4 A100 GPUs linked via PCIe Gen4 (32 GB/s per GPU) rather than NVLink. For 100 MB tensor-parallel transfers, inter-GPU communication latency is \(T_{\text{comm}} \approx\) 3.1 ms over PCIe versus 0.17 ms over NVLink. The scheduling policy uses max batch size 32 and timeout 50 ms, causing 5 percent of batches to experience head-of-line queuing delay \(T_{\text{queue}}\): short prompt requests wait behind long-sequence generations.

Systems lesson: Root-cause tail latency stems from scheduling policy and interconnect bottlenecks. Implementing priority-aware scheduling with differentiated batch size limits by request type:

  1. Request classification: Classify requests by expected output length (short: under 50 tokens, medium: 50 to 200 tokens, long: over 200 tokens)
  2. Differentiated batching: Limit short batches to 8, medium to 16, long to 32

This dropped P99 latency to \(L_{\text{lat}} =\) 185 ms.

The batching strategies examined so far divide along a fundamental constraint boundary. Vision workloads are compute-bound with fixed-shape tensors, so every image in a batch undergoes identical arithmetic, reducing the batch formation problem to packing a GPU’s compute pipeline as tightly as possible. LLM workloads are memory-bound with variable-length sequences, so the KV cache grows per-request and per-token, shifting the batch formation problem to memory accounting, eviction policies, and iteration-level scheduling that continuous batching provides. Each regime produced a different dominant strategy because the scarce resource differs: compute throughput for vision, memory capacity for language.

Recommendation systems present a third constraint regime that resembles neither. Sparse embedding lookups, not dense matrix multiplications, dominate both compute time and memory traffic. A single recommendation request may touch millions of embedding table entries scattered across shards, while the dense ranking head that follows is comparatively small. This access pattern demands a batching strategy organized around feature types and embedding locality rather than around request shape or sequence length.

Feature-parallel batching for recommendation systems

Recommendation batching starts by grouping work around embedding locality rather than request shape. Figure 6 shows how the request is split into feature-specific paths before the dense ranking head recombines the retrieved representations.

Figure 6: Feature-Parallel Batching Pipeline: Requests are batched by feature type (Users, Items, Context) and dispatched to specialized embedding servers in parallel. The retrieved embeddings are then concatenated and processed by the dense ranking head. This architecture enables scaling to trillions of parameters by decoupling embedding storage from dense compute.

Recommendation systems expose the same scheduling principle under a different bottleneck. Their computation pattern involves four stages:

  1. Sparse Feature Lookup: Retrieve embeddings for user, item, and context features
  2. Dense Feature Processing: Transform and normalize dense features
  3. Feature Interaction: Compute interactions between features (often via attention or factorization)
  4. Ranking Head: Produce final scores

The sparse embedding lookup often dominates latency and determines batching strategy. Feature-parallel batching processes different feature types in parallel rather than batching entire requests:

Request 1: [user_id_1, item_ids_1, context_1]
Request 2: [user_id_2, item_ids_2, context_2]
Request 3: [user_id_3, item_ids_3, context_3]

Feature-parallel view:
User embeddings:  [lookup(user_1), lookup(user_2), lookup(user_3)]  → parallel
Item embeddings:  [lookup(items_1), lookup(items_2), lookup(items_3)]  → parallel
Context features: [process(ctx_1), process(ctx_2), process(ctx_3)]  → parallel

Then: Combine features per request for ranking

Feature-parallel batching is natural when embeddings are sharded across servers: each embedding server handles lookups for its shard across all requests in the batch. At Meta-scale request volumes, this turns feature sharding into a serving strategy rather than a storage detail.

Example 1.4: RecSys batching at Meta scale
Scenario: Meta-scale recommendation infrastructure serves 10 million QPS, querying 5,000 embedding lookups per request across 1,000 shards.

Diagnosis: Single-threaded per-request embedding lookups fail at 50 million lookups/sec per shard. Accumulating requests over a 1 ms window forms batches of 50K lookups per shard, converting random memory reads into sequential accesses.

Systems lesson: Recommendation batching is organized around sparse feature locality rather than dense tensor shapes. Batching lookups at the embedding shard level maximizes memory bandwidth utilization within a 5.2 ms end-to-end latency budget.

Streaming inference for real-time applications

Streaming workloads define the boundary where batching itself becomes the wrong response. Real-time speech recognition, video analysis, and robotics require processing inputs as they arrive with minimal latency.

Streaming inference processes inputs incrementally without waiting for batch formation. In speech recognition, it processes audio frames (10–20 ms chunks) as they arrive from the microphone. In video analysis, it processes frames at the capture rate (30–60 FPS) without buffering. In robotics, it processes sensor readings at the control loop frequency (100–1000 Hz).

For streaming applications, the relevant metric is not throughput but time to process each input:

\[T_{\text{streaming}} = T_{\text{capture}} + T_{\text{transfer}} + T_{\text{inference}} + T_{\text{action}}\]

where all components must complete within the inter-frame interval. A streaming speech-to-text pipeline shows how these latency components compose under tight real-time constraints.

Example 1.5: Streaming speech recognition pipeline
Scenario: A real-time streaming speech-to-text pipeline processes 20 ms audio frames under a strict 100 ms end-to-end latency budget (5 frames max delay).

Diagnosis: Traditional batching introduces intolerable waiting latency. The pipeline must process frames incrementally (\(T_{\text{streaming}} = T_{\text{capture}} + T_{\text{transfer}} + T_{\text{inference}} + T_{\text{action}}\)), overlapping audio capture with acoustic model evaluation.

Systems lesson: Streaming inference prioritizes per-frame latency over batch throughput. When inter-frame intervals are tight, batching is removed and execution is optimized for continuous incremental pipelining.

Adaptive batching strategies

Once batching is workload-specific, fixed parameters become fragile. Production systems adapt batching behavior based on current conditions:

Traffic-adaptive batching adjusts the batch window based on arrival rate:

\[T_{\text{window}} = \min\left(T_{\text{max}}, \frac{B_{\text{target}}}{\lambda_{\text{current}}}\right)\]

When traffic is high, the window shrinks because the target batch size fills quickly. When traffic is low, the window extends but is capped to bound maximum latency.

SLO-adaptive batching takes a complementary approach, monitoring latency percentiles and adjusting parameters aggressively:

if P99_latency > 0.9 * SLO:
    reduce B_max by 20%
    reduce T_window by 20%
elif P99_latency < 0.5 * SLO:
    increase B_max by 10%
    increase T_window by 10%

The feedback loop maintains latency headroom while maximizing throughput during normal operation. Request-aware batching adds a third dimension by considering request characteristics when forming batches. For LLMs, this means grouping requests by expected output length (inferred from prompt type), grouping them by prompt length to minimize padding, and prioritizing latency-sensitive requests in smaller batches.

Production serving infrastructure embodies these adaptive principles. The NVIDIA Triton Inference Server implements a configurable adaptive batching system that illustrates how SLO-aware and request-aware strategies operate in practice. Triton exposes three knobs: Max_batch_size (the upper bound on batch size), Batching_timeout_ms (the maximum time to wait for batch formation), and Preferred_batch_size (target batch sizes that align with kernel efficiency). Internally, the scheduler maintains separate queues for each preferred batch size and routes requests to minimize total latency:

\[\text{Queue selection} = \operatorname{arg\,min}_{q} \left( \text{wait}_q + \text{exec}(|q| + 1) \right)\]

This optimization weighs both the current queue length and the kernel-efficiency of the resulting batch size. Table 11 shows the result for ResNet-50 on V100: the scheduler automatically increases batch size to maintain throughput as traffic grows, with measured throughput tracking offered load until saturation near 2,000 QPS.

Table 11: Triton Adaptive Batching: ResNet-50 on V100: Observed batch size, average latency, and sustained throughput for the Triton inference server’s adaptive batcher under increasing traffic on ResNet-50 (V100).
Traffic Level Avg Batch Size Avg Latency Throughput
100 QPS 2.1 8 ms 100 QPS
500 QPS 6.3 12 ms 500 QPS
1000 QPS 12.4 18 ms 1000 QPS
2000 QPS 24.1 28 ms 1980 QPS

Every batching strategy to this point assumes the work per request is fixed once a request is admitted: continuous batching and adaptive windows tune when requests join a batch and how long the system waits, but the number of decode steps each request needs is treated as given. Reasoning-heavy inference breaks that assumption. When a model can spend more tokens thinking before it answers, the work per request becomes a variable the scheduler must set, not a property it can only observe.

A two-rung magnitude ladder on a log scale, in slate. The reasoning response rung is about 12.8 seconds; the fast pattern-match rung is about 0.1 seconds. The gap is roughly 128 times.

Reasoning expands latency about 128 times over fast pattern-matching.

The logic wall: Test-time compute scaling

Test-time scaling turns reasoning into a serving resource. A request may issue more generated tokens, internal chain-of-thought (CoT) tokens, or search steps before producing the answer. Large-model capability work describes emergent behaviors (Wei et al. 2022), while later analysis cautions that some apparent emergence can be an artifact of metric choice (Schaeffer et al. 2023).

Wei, Jason, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, et al. 2022. “Emergent Abilities of Large Language Models.” Transactions on Machine Learning Research 4. https://doi.org/10.48550/arXiv.2206.07682.
Schaeffer, Rylan, Brando Miranda, and Sanmi Koyejo. 2023. “Are Emergent Abilities of Large Language Models a Mirage?” Advances in Neural Information Processing Systems (NeurIPS), 55565–81. https://doi.org/10.52202/075280-2425.

From a serving-systems perspective, the relevant shift is the work issued per request. Extra reasoning consumes latency budget, KV-cache residency, and accelerator time. Models that move from “fast thinking” (instant pattern matching) to “slow thinking” (deliberative reasoning) push pressure from HBM bandwidth toward Test-Time Compute: the scheduler must decide how many search steps or CoT tokens a request may consume. The Logic Wall is the resulting serving constraint: for complex problems, compute per request grows with task difficulty, and a fleet optimized for tokens per second also needs a policy for allocating thinking time. This is a different control than the batch-window tuning of the previous section: adaptive batching decides when requests run together, while test-time compute decides how much work each request is allowed to do.

Napkin Math 1.4: Scaling reasoning depth
Problem: Calculate the latency impact of a model that uses 128 “Thinking Tokens” to solve a complex math proof vs. a standard answer.

  1. Standard response: 1 token answer = 100 ms.
  2. Reasoning response: 128 of internal search/CoT before the answer.
  3. The latency: 128 \(\times\) 100 ms = 12.8 seconds.

Systems insight: Test-time scaling transforms the serving architecture from a throughput factory to a search engine. While standard serving optimizes for tokens per second, reasoning-heavy models are constrained by steps per second. This creates a “reasoning SLO”: users may tolerate 12 seconds for a correct proof, but not for a simple greeting. In the Machine Learning Fleet, this motivates dynamic compute allocation, where the scheduler grants more thinking time to harder prompts.

Variable work per request closes the batching story: continuous and adaptive batching handle variation the system observes, while test-time compute is variation the system chooses. Dynamic Compute Allocation assigns per-request compute budgets based on task difficulty, latency tolerance, and fleet load rather than applying a fixed decode budget to every prompt. With both the batching mechanism and the per-request compute budget now on the table, the remaining task is to decide which combination a given workload should run.

Quantitative summary: Batching strategy selection

The selection problem is to match the batching mechanism to model shape, traffic variance, and latency budget. That match depends on where in the request path the latency budget is spent.

Checkpoint 1.4: Batching strategy trade-offs

Verify your understanding of different batching mechanics:

Before selecting a batching strategy, it is essential to understand where latency accumulates across the full request lifecycle. Figure 7 maps each stage from client to response, revealing the “serving tax” that serialization, routing, and coordination impose outside of GPU compute.

Figure 7: End-to-End Inference Pipeline: A detailed view of the request lifecycle: client requests arrive at a load balancer, then undergo preprocessing (tokenization) on the CPU. The data moves to the GPU for the compute-bound prefill stage and the memory-bound decode stage, which uses the KV cache. Finally, the response is postprocessed on the CPU (detokenization) and sent back. This visualization highlights the critical “Serving Tax” components (serialization, routing, coordination) that consume latency budget outside of the actual GPU compute time.

Noncompute stages (serialization, queuing, routing) consume a substantial fraction of the total latency budget (figure 7), requiring batching strategy selection to account for the full end-to-end pipeline, not GPU execution time alone. A decision tree guides strategy selection.

Is it autoregressive text generation?
|- Yes → Continuous batching with chunked prefill
`- No → Is it real-time streaming speech/video/robotics?
        |- Yes → Streaming pipeline with minimal or no batching
        `- No → Do embedding lookups dominate latency?
                |- Yes → Feature-parallel batching (RecSys)
                `- No → Dynamic batching with adaptive parameters

Table 12 shows that each strategy tunes toward a different objective, so the parameter worth tuning follows from the binding goal: throughput for static, the latency-throughput balance for dynamic, decode-variance for continuous, shard capacity for feature-parallel, and the real-time deadline for streaming.

Table 12: Batching Strategy Parameters: Key configurable parameters (max batch size, queue timeout, iteration limits) tuned across static, dynamic, continuous, feature-parallel, and streaming strategies to optimize throughput vs. latency.
Strategy Key Parameters Tuning Goal
Static Batch size Maximize throughput
Dynamic Window, max batch Balance latency vs. throughput
Continuous Chunk size, max batch Minimize decode latency variance
Feature-parallel Accumulation window Match embedding shard capacity
Streaming Pipeline depth Meet real-time deadline

Even a well-tuned continuous batching strategy cannot eliminate a deeper, architecture-specific bottleneck for large language models. The context window of every active request must be stored in GPU memory, making KV cache management the next binding constraint on serving throughput.

Self-Check: Question

  1. A vision model has fixed latency overhead \(T_{\text{fixed}} = 2\text{ ms}\) and variable compute time \(T_{\text{var}} = 1\text{ ms/item}\). An LLM decode step has \(T_{\text{fixed}} = 40\text{ ms}\) (loading 140 GB weights from HBM) and \(T_{\text{var}} = 0.5\text{ ms/token}\). At what approximate batch size \(B\) does each workload reach the ‘knee’ of its batching efficiency curve (\(B \approx T_{\text{fixed}} / T_{\text{var}}\))?

    1. Vision knee at \(B \approx 10\); LLM decode knee at \(B \approx 20\)
    2. Vision knee at \(B \approx 80\); LLM decode knee at \(B \approx 2\)
    3. Vision knee at \(B \approx 4\); LLM decode knee at \(B \approx 40\)
    4. Vision knee at \(B \approx 2\); LLM decode knee at \(B \approx 80\)

  2. Define the Waste Ratio (\(W = 1 - \frac{\bar{S}}{S_{\text{max}}}\)) in traditional static LLM batching, and explain why continuous batching reduces this waste ratio to near zero.

  3. Order the four stages of a production recommendation system pipeline under feature-parallel batching:

  1. Feature Interaction (computing cross-feature embeddings via attention/factorization)
  2. Dense Feature Processing (transforming and normalizing continuous dense features)
  3. Sparse Feature Lookup (retrieving embeddings from sharded embedding tables across servers)
  4. Ranking Head (evaluating MLP/transformer layers to produce final candidate scores)

  1. A production inference service targets an aggregate arrival rate \(\lambda_{\text{arr}} = 1{,}000\text{ requests/sec}\) with a strict P99 latency SLO of \(T_{\text{lat}} = 100\text{ ms}\) (\(0.1\text{ s}\)). Each GPU replica executes batches of size \(B = 8\) with an observed replica latency of \(80\text{ ms}\) (\(0.08\text{ s}\)). According to Little’s Law (\(Q_{\text{req}} = \lambda_{\text{arr}} \cdot T_{\text{lat}}\)), what is the required concurrency and the minimum number of SLO-sized replicas needed?

    1. Required concurrency \(Q_{\text{req}} = 100\); minimum SLO-sized fleet \(= 13\) replicas
    2. Required concurrency \(Q_{\text{req}} = 1{,}000\); minimum SLO-sized fleet \(= 125\) replicas
    3. Required concurrency \(Q_{\text{req}} = 80\); minimum SLO-sized fleet \(= 10\) replicas
    4. Required concurrency \(Q_{\text{req}} = 100\); minimum SLO-sized fleet \(= 10\) replicas

  2. What is the ‘Logic Wall’ in test-time compute scaling, and how does it fundamentally shift the serving optimization objective from tokens-per-second to steps-per-second?

  3. True or False: In adaptive batching systems like NVIDIA Triton, increasing the batching timeout window (\(T_{\text{window}}\)) during sudden high-traffic load spikes is the optimal policy to reduce tail latency.

See Answers →

Memory and Decode-Time Management

As an LLM generates a 2,000-word essay, it must constantly recall every word it has previously written. It does this by storing intermediate attention states in a rapidly expanding memory buffer known as the KV Cache. Unmanaged, this cache will aggressively fragment GPU memory, causing out-of-memory crashes even when 40 percent of the VRAM is technically free.

Two classes of techniques meet at this boundary. KV-state capacity techniques, such as PagedAttention and prefix caching, decide where attention state lives and how much fragmentation the memory manager tolerates. Decode-time latency techniques, such as speculative decoding, change how much target-model work is needed per emitted token. Speculative decoding is not KV-cache compression; it trades extra draft-model state, target-model verification, and scheduler complexity for lower time per output token under the same memory budget.

The KV cache wall: Memory-bound capacity

Increasing the batch size to maximize throughput in LLM serving is bounded by the KV cache wall. Model weights represent a fixed “static tax” on GPU memory, while the KV cache grows linearly with both batch size and sequence length (figure 8).

Figure 8: The KV Cache Wall: Total GPU memory usage for a 4-bit quantized 70B model (35 GB weights) on an 80 GB H100. While the model fits easily at small context lengths, the KV cache (sloped lines) eventually consumes all remaining HBM. At 128K context, a batch size of 2 is physically impossible on a single GPU (35 GB + 84 GB > 80 GB), forcing the system to either reduce batch size (killing throughput) or shard the model.

The visualization reveals why the replica level must sometimes shard models that would otherwise fit on a single GPU. Sharding provides the memory headroom needed to maintain high batch sizes for long-context requests. Without sharding, a 128K context request effectively “evicts” all other users from the GPU.

The same formula can be turned into an explicit batch-size limit for production hardware.

Napkin Math 1.5: KV-cache capacity estimator
Problem: A deployment serves Llama-3-70B (FP16 weights \(\approx\) 141.2 GB) on an 8\(\times\) H100 node (640 GB total HBM). The goal is to determine the maximum batch size for a context length of 128K tokens.

Formula: \[ M_{\text{KV}} = 2 \times N_L \times H_{\text{KV}} \times d_{\text{head}} \times s_{\text{elem}} \] \[ M_{\text{total}} = M_{\text{weights}} + (B \times S \times M_{\text{KV}}) \]

Parameters:

  • \(N_L = 80\), \(H_{\text{KV}} = 8\) for Llama-3-70B GQA, \(d_{\text{head}} = 128\).
  • \(s_{\text{elem}} = 2\) bytes (FP16).
  • Context \(= 131,072\) tokens.

Step 1: Calculate memory per token. \[ M_{\text{KV}} = 2 \times 80 \times 8 \times 128 \times 2 = 327,680 \text{ bytes} \approx 0.33 \text{ MB/token} \]

Step 2: Calculate cache per request. \[ 131,072 \text{ tokens} \times 0.33 \text{ MB/token} \approx 42.9 \text{ GB/request} \]

Step 3: Determine max batch size. Available Memory for KV = 687.2 GB (Total) - 141.2 GB (Weights) - 20 GB (System) = 526.0 GB. \[ \text{Max Batch} = \left\lfloor \frac{526.0}{42.9} \right\rfloor = 12 \]

Systems insight: Even with grouped query attention (GQA), the shared-KV-head attention variant from Weight-only vs. weight-activation quantization that the 8-head count in this calculation already reflects, 128K-context requests consume tens of gigabytes of KV cache, so the system serves only a small number of concurrent long-context requests before memory becomes the binding constraint. Addressing this requires PagedAttention (to reduce fragmentation) and KV-cache quantization or sharding.

The fragmentation problem

Traditional memory allocation for KV cache preallocates contiguous memory for each sequence based on maximum expected length. This creates two forms of waste.

Internal fragmentation wastes memory within each allocation. Sequences shorter than the maximum allocation leave the unused portion idle. If maximum length is 4,096 but average output is 100 tokens, 97.5 percent of allocated memory is wasted.

External fragmentation compounds this problem across allocations. As sequences complete and new ones start, memory becomes fragmented into noncontiguous free blocks. Even with sufficient total free memory, no single block may be large enough for a new maximum-length allocation.

Consider a simplified example with 8 memory slots and maximum sequence length of 4. Fixed-size reservations first create internal fragmentation:

Time 0: Allocate Seq A (slots 0-3), Seq B (slots 4-7)
        [A][A][a][a][B][B][B][b]
        lowercase = reserved but unused

External fragmentation appears after sequences complete and new sequences start:

Time 1: Active Seq C and Seq D leave two 2-slot gaps
        [ ][ ][C][C][ ][ ][D][D]

Time 2: Try to allocate Seq E (needs 4 contiguous slots)
        [ ][ ][C][C][ ][ ][D][D]  <- Total free slots = 4, largest block = 2

Result: enough total free capacity exists, but no contiguous block is large enough.

Production systems report 60–80 percent memory waste from fragmentation under realistic workloads, severely limiting batch sizes and throughput.

PagedAttention

The fragmentation example exposes the allocator failure: enough KV-cache capacity can exist in aggregate while no contiguous block is available for the next sequence. PagedAttention fixes that serving problem by treating KV-cache storage like virtual memory rather than a single preallocated slab.

Definition 1.3: PagedAttention

PagedAttention is an LLM serving memory management technique that applies virtual memory principles to KV cache allocation, storing attention states in noncontiguous, fixed-size physical blocks.

  1. Significance: It eliminates internal and external fragmentation, which can waste 60–80 percent of the memory allocated to the KV cache under contiguous preallocation. Allowing sequences to grow dynamically enables 2–4\(\times\) higher concurrent throughput \((X)\) on the same hardware.
  2. Distinction: Unlike contiguous allocation (which requires prereserving the maximum context length), PagedAttention uses a block table to map logical sequence indices to physical memory pages, allocating only what is currently used.
  3. Common pitfall: A frequent misconception is that PagedAttention speeds up individual token math. In reality, it is a capacity optimization: it improves system-level efficiency by allowing larger batch sizes, though it adds a small indirection overhead \((L_{\text{lat}})\) for the pointer lookups.

The virtual-memory shift changes the allocation problem: capacity no longer has to be reserved as one contiguous block before the request starts.

Systems Perspective 1.1: Analogy: The inefficient hotel
Imagine a hotel where every guest might stay anywhere from 1 to 10 days, but they do not know in advance.

Under contiguous allocation, the hotel manager blocks out a 10-day suite for every guest in advance. A 100-room hotel becomes “fully booked” with only 10 guests, wasting 90 percent of its capacity (Internal Fragmentation).

Under PagedAttention (Virtual Memory), the manager assigns a guest 1 room at a time. If the guest stays another day, they are given whatever room is available next, even if it is on a different floor. The front desk maintains a “Block Table” to keep track of which rooms belong to which guest. The hotel can now accommodate 100 guests simultaneously, recovering the 90 percent wasted capacity.

PagedAttention (Kwon et al. 2023), introduced in vLLM, applies virtual memory concepts to KV cache management. Instead of contiguous allocation, the KV cache is divided into fixed-size pages, using 16-token blocks in the default vLLM configuration, and sequences are allocated pages on demand. Figure 9 illustrates the key concepts including page tables that map logical sequence positions to physical memory pages, block size that defines the number of tokens per page (typically 16 tokens), and physical blocks that provide fixed-size memory allocations assignable to any sequence.

Figure 9: KV Cache Fragmentation and the PagedAttention Solution: Top left: internal fragmentation wastes memory by preallocating for the maximum sequence length. Top right: external fragmentation leaves unusable memory gaps. Bottom: PagedAttention solves this by mapping contiguous logical pages of a sequence to noncontiguous physical memory blocks via a block table, allowing the system to fill fragmentation gaps with small blocks from any sequence.

PagedAttention provides four capacity benefits:

  • Internal fragmentation: It allocates only the pages needed for actual tokens.
  • External fragmentation: Any free page can be used by any sequence.
  • Dynamic growth: Sequences can grow without preallocation.
  • Prefix sharing: Common prefixes can share physical pages.

Together, these properties turn KV cache capacity into a schedulable resource rather than a fixed per-request reservation.

Example 1.6: PagedAttention implementation details
Scenario: An LLM inference server allocates KV cache memory contiguously for incoming sequences based on maximum sequence length (e.g., 4,096 tokens).

Diagnosis: Contiguous pre-allocation suffers from internal and external memory fragmentation, wasting 60 to 70 percent of GPU HBM (30 to 40 percent pool utilization). PagedAttention maps virtual sequence pages to non-contiguous physical HBM blocks (e.g., 16 tokens/block) via per-sequence page tables, executing non-contiguous gather operations in custom attention kernels.

Systems lesson: PagedAttention eliminates internal and external memory fragmentation. Increasing KV cache pool utilization from 30 to 40 percent to over 95 percent allows serving engines to pack significantly more concurrent sequences, achieving 2.5–4\(\times\) higher throughput.

Prefix caching

Many LLM workloads share common prefixes across requests. System prompts like “You are a helpful assistant…” are prepended to every request. Few-shot examples use the same examples for many queries. Document context involves multiple questions about the same document. Recomputing these shared prefixes wastes both compute (prefill) and memory (duplicate KV cache entries).

Prefix caching shares KV cache entries across requests with common prefixes. Figure 10 demonstrates how shared system prompts avoid redundant computation.

Figure 10: Prefix Caching via Block Sharing: PagedAttention enables efficient prefix caching by allowing multiple sequences’ block tables to point to the same physical blocks for shared content. In this example, the System Prompt is stored in blocks 0-5. Request A and Request B map their first 6 logical pages to these same physical blocks, storing only their unique suffixes in new blocks. This dramatically reduces memory usage and prefill computation for workloads with shared context.

With PagedAttention, prefix caching integrates naturally through copy-on-write semantics:

System prompt → Physical blocks [0, 1, 2, 3, 4, 5]

Request A page table: [0, 1, 2, 3, 4, 5, 10, 11]  <- shares prefix blocks
Request B page table: [0, 1, 2, 3, 4, 5, 12, 13, 14]  <- shares prefix blocks
Request C page table: [0, 1, 2, 3, 4, 5, 15]  <- shares prefix blocks

All three requests reference the same physical blocks for the system prompt. Only when generating unique tokens do they allocate new blocks. The savings can be substantial when many concurrent requests share the same system prompt; notebook 1.6 quantifies them for a typical chatbot deployment.

Napkin Math 1.6: Prefix caching at scale

Scenario: A chatbot service uses a 2,000-token system prompt and serves 1,000 concurrent users.

Without prefix caching:

  • KV cache per user: \(2000 + 500\text{ (avg response)} =\) 2,500 tokens
  • Total KV cache: 2,500 tokens \(\times 1000 \times 2 \times 80 \times 8192 \times 2 =\) 6.6 TB

With prefix caching:

  • Shared prefix: 2,000 tokens (once)
  • Unique per user: 500 tokens
  • Total: \((2000 \times 1) + (500 \times 1000) =\) 502,000
  • Memory: 502,000 \(\times 2 \times 80 \times 8192 \times 2 =\) 1.3 TB

Result: Prefix caching yields a 79.9 percent memory reduction in KV cache, enabling 5× more concurrent users.

Systems insight: Prefix caching pays off when the prefix hit rate is high because many requests share a long prefix. Table 13 contrasts hit rates and memory savings across workloads, showing where the technique changes serving capacity and where it does not.

Table 13: Prefix Caching Effectiveness by Workload: Typical prefix hit rates and resulting KV-cache memory savings for chatbot, document-QA, and general API workloads, showing where prefix caching pays off and where it does not.
Workload Prefix Hit Rate Memory Savings
Chatbot (same system prompt) 95%+ 70-80%
Document QA (same doc) 80-90% 50-70%
General API (diverse) 20–40% 10–30%

KV cache compression and architectural optimization

The capacity-management techniques in section 1.3.3 and section 1.3.4 tolerate the cache; compression shrinks it. The calculation here uses a 64-KV-head multi-head attention (MHA) baseline so the grouped query attention subsection can isolate the head-count reduction as a separate architectural lever. For a 70B parameter model with 80 layers, 64 heads, head dimension 128, and sequence length 4096 in FP16, the KV cache requires:

\[\text{KV cache} = 2 \times 80 \times 64 \times 128 \times 4096 \times 2 \approx 10.7\text{ GB (FP16)}\]

A single request’s KV cache alone consumes a substantial fraction of the H100’s 80 GB of HBM. For a batch of 8 concurrent requests, the KV cache would require 85.9 GB, exceeding single-GPU capacity. Reducing this footprint requires two complementary strategies: quantization and architectural optimization.

KV cache quantization

Reducing the size of cached values through lower precision provides direct memory savings. Weight-only quantization reduces weight precision while keeping activations in high precision. While effective for storage, this does not reduce the KV cache. KV Cache Quantization explicitly targets the activations, storing cached keys and values in INT8, FP8, or even INT4.

Compressing the KV cache to INT4 reduces it to approximately 2.7 GB per request, a 4× reduction. This freed memory directly translates into larger batch sizes and higher serving throughput. KV cache values exhibit different distributions than model weights: KIVI observes channel-wise outliers in keys and uses per-channel key quantization, while values lack the same channel-wise pattern and are quantized per-token.

Grouped query attention (GQA)

Grouped query attention (Ainslie et al. 2023), the shared-KV-head architecture established in Weight-only vs. weight-activation quantization, is the second lever on cache size, complementary to quantization: where quantization shrinks the bytes per cached element, GQA shrinks how many key-value heads are cached per layer. The serving consequence is the head-count arithmetic. For the representative 70B model, switching from the MHA baseline of 64 KV heads to GQA with 8 KV heads shrinks the KV cache by 8\(\times\):

Ainslie, Joshua, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.” Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 4895–901. https://doi.org/10.18653/v1/2023.emnlp-main.298.

\[\text{KV cache (GQA)} = 2 \times 80 \times 8 \times 128 \times 4096 \times 2 \approx 1.3\text{ GB (FP16)}\]

At 1.3 GB, the GQA cache is dramatically more manageable than the 10.7 GB required by MHA. Combining GQA with INT8 KV cache quantization yields sub-gigabyte per-request cache sizes, often enabling substantially larger batches on a single GPU. GQA has become a common architectural choice for inference-optimized LLMs because it reduces KV-cache bandwidth and capacity pressure with comparatively small quality impact in many deployments.

Napkin Math 1.7: The precision dividend
Problem: A team serves a 70B model on 4\(\times\) H100 GPUs. Weights in FP16 consume 140 GB (35 GB/GPU). KV cache at FP16 consumes 10.7 GB per request. How does quantizing weights to INT4 and KV cache to INT8 change the maximum batch size?

Before optimization (all FP16):

  • Weights: 35 GB/GPU
  • Available for KV cache: 80 GB \(-\) 35 GB = 45 GB/GPU
  • KV cache per request: 10.7 GB \(\div\) 4 \(\approx\) 2.7 GB/GPU
  • Maximum batch size: \(\lfloor\) 45 GB divided by 2.7 GB \(\rfloor\) \(\approx\) 16 requests

After optimization (INT4 weights, INT8 KV cache):

  • Weights: 35 GB \(\times (4/16) =\) 8.75 GB/GPU (INT4)
  • Available for KV cache: 80 GB \(-\) 8.75 GB = 71.25 GB/GPU
  • KV cache per request (INT8): 5.4 GB \(\div\) 4 \(\approx\) 1.3 GB/GPU
  • Maximum batch size: \(\lfloor\) 71.25 GB divided by 1.3 GB \(\rfloor\) \(\approx\) 53 requests

Systems insight: Precision engineering fundamentally changes serving economics by enabling larger batch sizes. Larger batches amortize the fixed cost of weight loading, shifting operations from memory-bound toward compute-bound.

Speculative decoding as a serving policy

Speculative decoding was established as an algorithmic optimization in Speculative decoding: a small draft model proposes \(K\) tokens, the target model verifies them in one parallel forward pass, and rejection sampling guarantees the emitted tokens follow the target model’s distribution exactly, so the technique is lossless and bounded by the draft acceptance rate \(\alpha_{\text{acc}}\). The serving question is different. Speculation becomes a resource-admission policy under SLA pressure: the scheduler must decide when to enable it, how its memory cost interacts with KV-cache admission, and what it does to the SLA the endpoint is bound by.

A roofline silhouette: a blue memory-bound slope rising to a dashed ridge line, then an orange compute-bound ceiling. A workload dot sits deep on the memory-bound slope, the regime batch-1 decode occupies before parallel verification raises its arithmetic intensity toward the ridge.

Decode is memory-bound; parallel verification moves work toward the ridge.

The latency win is real but conditional, and the condition is the acceptance rate. When the draft model proposes \(K\) tokens, high acceptance rates allow the target model to verify and emit multiple tokens in a single forward pass. When acceptance drops, rejected tokens are discarded, reducing the effective generation rate back toward single-token decoding. Mathematically, the expected number of tokens emitted per round, including the correction token, is \(\frac{1 - \alpha_{\text{acc}}^{K+1}}{1 - \alpha_{\text{acc}}}\), which collapses toward 1 as acceptance falls. For \(K = 5\) draft tokens and a draft model 20\(\times\) faster than the target, that sensitivity is steep:

  • High acceptance \((\alpha_{\text{acc}} = 0.9)\): Expected 4.69 tokens per round, speedup of 3.7×.
  • Medium acceptance \((\alpha_{\text{acc}} = 0.7)\): Expected 2.94 tokens per round, speedup of 2.4×.
  • Low acceptance \((\alpha_{\text{acc}} = 0.5)\): Expected 1.97 tokens per round, speedup of 1.6×.

Predictable text achieves \(\alpha_{\text{acc}} > 0.9\), while creative reasoning or code generation may fall below 0.5; well-aligned model pairs that share training data and architecture commonly land at 0.6–0.8. Because acceptance is a property of the workload, not of the hardware, a single global setting is wrong for a mixed fleet: the same speculation that halves time per token on a chat endpoint can slow a code-generation endpoint that sits below the break-even acceptance rate. The serving system must therefore decide per endpoint, and three serving concerns govern that decision.

The speculation tax and KV-cache admission

The first concern is memory. The draft model is not free state: it requires its own GPU allocation for weights, and each in-flight request that uses speculation reserves a small additional KV cache for the draft model alongside the target’s. Pairing a 7B draft model with a 70B target adds roughly 14 GB of weights per replica plus per-request draft cache. This speculation tax competes directly with the KV-cache admission budget developed in section 1.3.1: every gigabyte the draft model holds is a gigabyte the admission controller cannot allocate to a concurrent request. Speculation therefore shrinks the maximum batch size the node can admit, trading aggregate throughput for the latency of individual requests. The admission controller and the speculation policy cannot be tuned independently; enabling speculation lowers the concurrency ceiling that the KV-cache wall already imposes.

The latency-throughput tension under an SLA

The second concern is which service-level objective the endpoint is bound by. Speculation improves time per output token but, by lowering the concurrency ceiling, reduces the requests per second a replica can sustain. An endpoint with a tight time-to-first-token or per-token latency SLA, such as interactive chat, benefits: the speculation tax buys headroom against the latency target that matters. An endpoint bound by a throughput SLA, such as bulk document processing or batch summarization, is harmed: speculation spends batch capacity to improve a latency metric no one is measuring, and the lost concurrency directly threatens the throughput target. The decision is therefore an SLA question. The scheduler enables speculation on latency-bound endpoints and disables it on throughput-bound ones, and on a shared replica serving both it must account for the draft model’s resident cost against the stricter of the two budgets.

Per-endpoint scheduling

The third concern is that these decisions are dynamic, not static configuration. Acceptance rate drifts with the traffic mix, and the value of speculation collapses at high load: when a replica is already near its concurrency ceiling, the decode phase is closer to compute-bound and the marginal latency benefit of speculation shrinks even as its memory tax stays fixed. Production schedulers therefore treat speculation as a runtime knob gated on observed load and acceptance. They enable it for latency-sensitive endpoints during normal operation, disable it under traffic spikes to reclaim KV-cache pages for admission, and may switch draft strategies per endpoint. Self-speculative Decoding uses early exit from the target model itself as the draft, eliminating the separate-model memory tax at the cost of lower acceptance; Medusa (Cai et al. 2024) adds lightweight prediction heads to the target backbone; Lookahead Decoding uses Jacobi iteration to avoid a draft model entirely. Each variant trades a different slice of the speculation tax against acceptance, and the scheduler chooses among them per endpoint based on which SLA binds and how much KV-cache budget remains.

Cai, Tianle, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads.” Proceedings of the 41st International Conference on Machine Learning (ICML).

KV cache in distributed settings

When a conversation is moved, rebalanced, or split across devices, its KV state must either move with it or be rebuilt token by token. Section 1.4 develops the tensor and pipeline parallelism strategies that distribute a model across devices; each strategy carries a different consequence for KV-cache management:

Under tensor parallelism, the KV cache is sharded across devices along with attention heads. Each device stores cache for its subset of heads.

8-way tensor parallelism:
Device 0: KV cache for heads 0-7
Device 1: KV cache for heads 8-15
...
Device 7: KV cache for heads 56-63

Cross-device sharing adds a constraint: prefix caching across tensor-parallel devices requires cache to be sharded identically on all devices. This is automatic when prefixes are processed with the same tensor-parallel configuration.

KV cache migration presents a further challenge. When the router moves a conversation to a different replica because of failure or rebalancing (routing keyed on a hash of the session ID, the consistent-hashing scheme developed in section 1.5.6), the KV cache must be migrated:

Migration options:
1. Rebuild: Re-run prefill on new replica (500 ms+ for long context)
2. Transfer: Send KV cache over network (100 MB at 100Gbps = 8 ms)
3. Hybrid: Transfer if small, rebuild if large

Decision threshold:
if cache_size_bytes/network_bandwidth < prefill_time:
    transfer()
else:
    rebuild()

For Llama-70B GQA with 4K context, KV cache is about 1.3 GB total per sequence, or about 162.5 MB per device under 8-way tensor parallelism. At 100 Gbps (12.5 GB/s), transferring one shard takes roughly 13 ms, while transferring the full cache would take about 100 ms. Both are often preferable to a ~500 ms prefill rebuild, so transfer remains the better choice when bandwidth is available.

Memory management best practices

Effective KV-cache management begins with capacity accounting rather than an eviction policy. The serving process first reserves memory for weights, activations, runtime overhead, and safety headroom, then treats the remaining HBM as a managed pool for active sequences:

Available GPU memory = Total - Weights - Activations - Overhead
KV pool size = 0.9 * Available  # Leave 10% headroom

Max concurrent sequences = KV pool size / (avg_seq_length * per_token_cache)

That budget determines how many sequences can coexist before the scheduler must choose between rejection, preemption, or paging. When the cache is full, LRU eviction removes the sequence with the oldest access, size-based eviction frees the longest sequence first, and priority-based eviction protects paid-tier or latency-critical requests. Continuous batching then turns eviction into a scheduling decision: a high-priority request that cannot fit triggers victim selection, swaps the victim’s KV cache to CPU memory, allocates GPU memory to the new request, and restores the victim later if its service-level objective still permits the delay.

Example 1.7: KV cache memory hierarchy
Scenario: A multi-tenant LLM serving system experiences KV-cache HBM exhaustion (80 GB limit) during peak concurrency.

Diagnosis: Pure HBM allocation limits node concurrency to 50 active sequences. Evicting or dropping requests degrades user experience. Implementing a 3-tier memory hierarchy (HBM \(\to\) CPU DRAM \(\to\) Non-Volatile Memory Express (NVMe) SSD) allows swapping inactive sequence pages to CPU DRAM (1–5 ms swap latency) or NVMe (10–50 ms).

Systems lesson: Multi-tier KV-cache paging increases concurrent sequence capacity by 10\(\times\) (50 to 500 sequences). Paging to CPU DRAM serves as a throughput lever for non-urgent traffic, but swap latency makes it unviable for latency-critical decode steps.

Example 1.8: Sarathi: Chunked prefill implementation
Scenario: An LLM inference server receives a mix of long input prompts (10k+ tokens) and interactive short decode requests (Agrawal et al. 2024).

Diagnosis: Un-chunked prompt prefills monopolize GPU compute engines for hundreds of milliseconds, blocking decode steps for concurrent requests and triggering severe inter-token latency spikes.

Systems lesson: Chunked prefill (Sarathi-Serve) divides long prompt prefills into fixed-size chunks (e.g., 200 tokens/chunk), interleaving prefill chunks with decode iterations. Bounding prefill execution time per iteration prevents decode stalls and maintains inter-token SLOs.

Agrawal, Amey, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve.” 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 117–34.

Adapter state and locality

KV-cache management makes one GPU safe for many concurrent requests; multi-tenant adapter serving asks the same serving process to manage another kind of resident state. The KV cache is per-token attention state. A low-rank adaptation (LoRA) adapter is a per-tenant weight delta. Both must be available at the moment a decode step runs, and both can erase batching gains if the scheduler ignores locality. As serving platforms scale to support thousands of concurrent users on a single foundation model, personalization introduces a new memory bottleneck. LoRA stores a small set of user- or task-specific adapter weights that modify a shared base model without copying the full model. When 10,000 users each have a unique LoRA adapter applied to the same 140 GB base model, replicating the base weights for each user is physically impossible. Instead, the base model remains pinned in HBM, while user-specific adapters are brought into the fast on-chip working set as needed.

Systems Perspective 1.2: The context switch of machine learning
This multi-tenant serving pattern shifts the fundamental performance constraint from compute throughput to adapter locality. As the continuous batching scheduler interleaves requests from different users, adapter weights may have to move repeatedly from HBM into SRAM or registers before each generation step. The effect is similar to an operating-system context switch: the accelerator has work to do, but it first waits for the state belonging to the next tenant. If the adapter swap latency exceeds the compute time of the generation step, the GPU compute cores stall. Efficient multi-tenancy therefore batches requests by active adapter when possible and coordinates adapter placement with KV-cache paging so personalization does not erase the benefit of sharing the base model.

The same locality problem reappears at a larger scale when the model itself exceeds single-GPU capacity. For a 175-billion-parameter foundation model, even the weights must be split across multiple devices, so inference becomes a sharding problem before it becomes a scheduling problem.

Self-Check: Question

  1. How does PagedAttention eliminate memory waste in LLM KV cache management, and what is its primary system-level benefit?

    1. It compresses KV cache tensors using lossless entropy coding to reduce arithmetic intensity in attention kernels.
    2. It replaces standard multi-head attention with grouped query attention by dynamically merging key-value heads.
    3. It stores attention KV states in non-contiguous, fixed-size physical memory pages mapped via logical block tables, boosting batch capacity \(2\text{--}4\times\).
    4. It offloads 100% of KV cache data to host CPU DRAM, eliminating GPU HBM usage entirely for active sequences.

  2. What is the ‘speculation tax’ in speculative decoding, and how does it influence whether speculative decoding should be enabled for a specific serving endpoint?

    1. The latency penalty paid when draft models fail to generate valid UTF-8 token encodings.
    2. The GPU memory allocated to draft model weights and draft KV cache, which reduces available HBM for batch concurrency.
    3. The serialization tax incurred when transmitting draft tokens across InfiniBand interconnects.
    4. The loss in model generation accuracy caused by greedy sampling during draft token verification.

  3. A chatbot platform serves 1,000 concurrent users who all share a common 2,000-token system prompt, generating an average of 500 response tokens each. Explain how prefix caching reduces total KV cache memory from ~6.6 TB to ~1.3 TB (~80% reduction).

  4. Why do un-chunked long prompt prefills (e.g., 10,000 tokens) cause severe tail latency spikes for concurrent decode requests, and how does chunked prefill (Sarathi-Serve) resolve this?

  5. Switching from standard Multi-Head Attention (MHA) with 64 key-value heads to Grouped Query Attention (GQA) with 8 key-value heads reduces the KV cache memory footprint by a factor of ____\(\times\).

  6. Order the steps executed during a single verification round of Speculative Decoding:

  1. Target model runs a single batched parallel forward pass across all \(K\) candidate positions
  2. Rejection sampling evaluates candidate tokens sequentially against target probability distributions
  3. Draft model autoregressively generates \(K\) candidate tokens
  4. KV cache block tables are updated to append accepted tokens and one target correction token, discarding rejected branches

See Answers →

Model Sharding for Inference

A 70-billion-parameter model requires over 140 GB of VRAM to hold its weights in FP16, exceeding a single 80 GB GPU’s capacity. Serving this model requires slicing its architecture across multiple GPUs that act together as a single logical replica. Adapting distributed training parallelism techniques, specifically tensor and pipeline parallelism, for low-latency inference introduces a distinct set of constraints.

When sharding becomes necessary

Model sharding for inference is driven by two distinct requirements. Table 14 identifies the memory and latency constraints that necessitate sharding:

The first driver is memory capacity. A model that cannot fit in single-GPU memory must be sharded regardless of performance considerations. For a model with \(P\) parameters at precision \(b\) bits, the weight memory is calculated by equation 5:

\[M_{\text{weights}} = P \times \frac{b}{8} \text{ bytes} \tag{5}\]

A 70-billion-parameter model in FP16 (16 bits) requires:

\[M_{\text{weights}} = 70 \times 10^9 \times \frac{16}{8} = 140 \text{ GB}\]

The result exceeds the 80 GB capacity of an H100 GPU, requiring at minimum 2-way sharding.

The second driver is latency. Even when a model fits in memory, sharding can reduce latency by parallelizing computation. Equation 6 formalizes the potential speedup as a function of parallelization efficiency:

\[T_{\text{parallel}} = \frac{T_{\text{compute}}}{N} + T_{\text{comm}}(N) + T_{\text{sync}}(N) - T_{\text{overlap}} \tag{6}\]

where \(N\) is the number of devices in the sharding group, \(T_{\text{comm}}(N)\) is the communication overhead, \(T_{\text{sync}}(N)\) is synchronization overhead, and \(T_{\text{overlap}}\) is communication hidden behind useful compute. In the common no-overlap simplification with negligible extra synchronization, this reduces to \(T_{\text{compute}}/N + T_{\text{comm}}(N)\). Sharding provides latency benefit only when the nonoverlapped overhead is smaller than the time saved through parallelization.

Table 14: Sharding Triggers: Model memory, KV-cache capacity, embedding table size, and forward-pass latency limits dictate minimum sharding degrees and candidate distribution strategies.
Sharding Trigger Model Examples Minimum Sharding Strategy
Memory (weights) Llama-70B (140 GB) 2-way Tensor or pipeline
Memory (KV cache) GPT-4 (long context) 4–8 way Tensor (for cache)
Memory (embeddings) DLRM (100 TB) 1000+ way Embedding sharding
Latency Any large model Varies Tensor parallelism

Tensor parallelism

Tensor parallelism (Shoeybi et al. 2019) distributes individual layers across multiple devices, enabling parallel computation within each layer. The column-row partitioning scheme introduced for training in Distributed Training applies here: splitting the first linear layer by columns and the second by rows requires only one AllReduce per transformer block. For transformer models, the primary target is the attention mechanism and feed-forward layers, which contain the majority of computation.

Shoeybi, Mohammad, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.” arXiv Preprint arXiv:1909.08053.

The multi-head attention computation naturally partitions across attention heads. For a model with \(N_{\text{heads}}\) attention heads distributed across tensor-parallel degree \(t\), each device computes \(N_{\text{heads}}/t\) heads:

\[\text{Attention}_i = \text{softmax}\left(\frac{Q_i K_i^T}{\sqrt{d_k}}\right) V_i \text{ for heads } i \in \{1, ..., N_{\text{heads}}/t\}\]

After computing local attention, an AllReduce operation (covered in Collective Communication) combines results across devices, adding communication overhead proportional to the activation size divided by the interconnect bandwidth.

The feed-forward layer (typically two linear transformations with activation) partitions along the hidden dimension. For the first linear layer, columns are distributed; for the second, rows are distributed. This column-row partitioning requires only one all-reduce per feed-forward block.

The communication pattern for tensor-parallel inference follows a two-phase synchronization per layer. Figure 11 illustrates this flow:

Figure 11: Tensor Parallelism for Inference: Computation is distributed across devices by splitting tensor operations. Attention heads are partitioned across GPUs, requiring an AllReduce operation to synchronize results. Feed-forward networks use a column-row splitting strategy that requires only one AllReduce synchronization per block. This approach reduces latency for large models but introduces communication overhead that demands high-bandwidth interconnects like NVLink.

The inference time with tensor parallelism follows equation 7:

\[T_{\text{inference}} = \frac{T_{\text{compute}}}{t} + 2 \times T_{\text{allreduce}}\left(\frac{A}{t}\right) \tag{7}\]

where \(T_{\text{compute}}\) is the sequential compute time, \(t\) is the tensor-parallel degree, and \(A\) is the activation size being reduced. The factor of 2 accounts for the two all-reduce operations per transformer layer (attention and feed-forward).

Example 1.9: Tensor parallelism for Llama-70B
Scenario: An engineering team deploys Llama-70B (70B parameters, FP16 footprint 140 GB) across an 8-GPU H100 node with NVLink (Touvron et al. 2023).

Diagnosis: Fitting the model on 2 GPUs (140 GB/80 GB) suffers from compute bottlenecks (30 ms per layer). Applying 8-way tensor parallelism shards attention and feed-forward layers across all 8 GPUs, reducing per-layer execution to 4.35 ms despite AllReduce communication overhead.

Systems lesson: 8-way tensor parallelism over NVLink achieves a 6.9\(\times\) end-to-end speedup (prefill time reduced from 2,400 ms to 348 ms). Sharding attention and feed-forward layers across intra-node NVLink interconnects buys latency by trading low-overhead NVLink transfers for compute acceleration.

Touvron, Hugo, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, et al. 2023. “Llama 2: Open Foundation and Fine-Tuned Chat Models.” arXiv Preprint arXiv:2307.09288.

Pipeline parallelism for inference

Pipeline parallelism distributes layers across devices sequentially, with each device handling a subset of layers. Unlike tensor parallelism, there is no synchronization within a layer, only between pipeline stages.

For inference, pipeline parallelism creates bubbles differently than in training. Figure 12 contrasts single-request latency (bubble-dominated) with pipelined throughput (bubble-amortized):

Figure 12: Pipeline Parallelism Bubbles: For a single inference request (top), pipeline parallelism offers no latency benefit as the request must traverse all four stages sequentially. When processing multiple concurrent requests (bottom), the pipeline fills and throughput scales with the number of stages. This makes pipeline parallelism ideal for high-throughput batch processing but less suitable for latency-critical interactive serving.

For a single request, pipeline parallelism provides no latency benefit: the request must traverse all stages sequentially. The pipeline fill time equals the sequential execution time.

However, pipeline parallelism enables throughput scaling through pipelining multiple requests:

Time →
Device 0: [Req1] [Req2] [Req3] [Req4] ...
Device 1:        [Req1] [Req2] [Req3] [Req4] ...
Device 2:               [Req1] [Req2] [Req3] [Req4] ...
Device 3:                      [Req1] [Req2] [Req3] [Req4] ...

Once the pipeline is full, throughput equals \(p\) times single-stage throughput, where \(p\) is the number of pipeline stages. The steady-state latency remains approximately the single-device latency (sum of all stage times), but throughput scales with parallelism.

Pipeline parallelism is appropriate for inference in three scenarios:

  • Memory constraints require sharding but latency requirements are relaxed
  • Throughput matters more than individual request latency
  • Network bandwidth between devices is limited (only point-to-point communication)

Table 15 captures the trade-offs between these two sharding approaches:

Table 15: Pipeline vs. Tensor Parallelism: Key trade-offs between layer-internal tensor splitting (reduces single-request latency, requires high-bandwidth NVLink) and inter-layer pipeline staging (maximizes batch throughput, introduces pipeline bubbles).
Aspect Tensor Parallelism Pipeline Parallelism
Single-request latency Reduced by ~\(t\times\) No improvement
Throughput \(t\times\) \(p\times\) (when pipelined)
Communication pattern AllReduce (bandwidth-intensive) Point-to-point (latency-sensitive)
Memory efficiency Activations replicated Activations passed along
Complexity Higher (requires custom kernels) Lower (layer-level partitioning)

The decision rule is latency first, topology second. Tensor parallelism is the right default when single-request latency matters and NVLink-class bandwidth is available; pipeline parallelism is a throughput tool for relaxed-latency workloads or placements where only point-to-point stage communication is affordable.

Expert parallelism for MoE models

Mixture-of-experts (MoE) models, such as DeepSeek-V3 or Mixtral, introduce unique serving challenges beyond those of dense models. In an MoE transformer layer, the feed-forward network is replaced by multiple parallel “expert” sub-networks and a lightweight router that selects a subset of experts for each token.

The MoE architecture enables scaling the total parameter count while keeping the per-token compute constant. However, serving such models at scale requires expert parallelism, where different experts are hosted on different GPUs.

MoE economics and capacity planning

The economics of MoE serving create a paradox: total parameter count determines the minimum hardware count (for memory), while active parameter count determines the compute cost per token. This makes MoE models efficient in compute but expensive in terms of VRAM “rent.”

The performance advantages are striking. During autoregressive decode at batch size 1, the dominant cost is reading weights from HBM. A dense 400B model in FP16 reads 800 GB per step. DeepSeek-V3, despite having more total parameters, reads only the 37B active parameters per step (approximately 74 GB in FP16), a 10.8× reduction in per-token bandwidth. The compute savings are proportional: 10.8× fewer FLOPs per token.

Memory ladder comparing MoE resident weights at 1342 GB, dense resident weights at 800 GB, and MoE active read at 74 GB.

MoE keeps every expert resident but activates few per token.

The trade-off is memory capacity: all experts must reside in memory even though only a fraction are active at any time. DeepSeek-V3’s full model in FP16 requires approximately 1342 GB, necessitating distribution across many GPUs.

Napkin Math 1.8: MoE capacity planning
Problem: A team deploys DeepSeek-V3 (671B total parameters, 37B active per token) for a chatbot application. The model uses FP8 weights (1 byte per parameter). The cluster has 8-GPU nodes, each with 8\(\times\) H100 GPUs (80 GB HBM per GPU, 640 GB per node). How many nodes are needed, and what is the per-token decode latency?

Math:

  1. Memory: Total weight memory = 671 GB. A single node (640 GB) is insufficient. 2 provide 1,280 GB, leaving enough room for KV caches.
  2. Latency: Each decode step reads 37B active parameters (37 GB). Distributed across 16 GPUs, each reads about 2.3 GB. At 3.35 TB/s HBM bandwidth, \(T_{\text{read}} \approx\) 0.69 ms. AllToAll routing adds about 0.3 ms.
  3. Floor: Estimated decode latency \(\approx\) 1.0 ms/token, or 1,000 tokens/s at batch size 1.

Systems insight: MoE allows a 671B model to approach the per-token bandwidth cost of a much smaller dense model, but capacity planning is still driven by the full parameter set that must remain resident.

Expert parallelism and load balancing

Distributing MoE models across GPUs introduces expert parallelism: each GPU holds a subset of experts, and tokens are routed to the GPU holding their assigned expert. In an MoE layer, a gating network selects \(k\) experts (out of \(E\) total) for each token:

\[\text{Output} = \sum_{i \in \text{top-}k} g_i \cdot \text{Expert}_i(\text{input})\]

Expert parallelism distributes experts across devices, with each device hosting \(E/N\) experts, where \(N\) is the number of expert-parallel devices. Figure 13 traces the routing, dispatch, and gather operations:

Figure 13: Mixture-of-Experts (MoE) Routing: Expert parallelism distributes “experts” across different devices. For each token, a gating mechanism selects the top-\(k\) experts. An AllToAll communication step dispatches tokens to the devices hosting their selected experts (1). Experts process the tokens in parallel (2). A second AllToAll step gathers the results back to the original device (3). This pattern enables massive model capacity but introduces all-to-all communication overhead.

The communication pattern differs from tensor parallelism: instead of all-reduce (same data to all devices), expert parallelism uses all-to-all (different data to different devices based on routing). This AllToAll pattern creates a critical system challenge: load balancing. If the router consistently sends more tokens to certain experts than others, those experts’ GPUs become bottlenecks while other GPUs sit idle. Three mechanisms address load imbalance:

  • Auxiliary Loss: An additional term in the training loss penalizes uneven expert utilization by rewarding uniform routing probabilities.
  • Capacity Factor: Each expert has a maximum number of tokens it will accept per batch (typically 1.25\(\times\) the fair share). Tokens exceeding this are dropped or rerouted.
  • Expert Buffering: Tokens are buffered and processed in subsequent iterations if experts are at capacity, smoothing out load spikes at the cost of latency variance.

These mechanisms keep expert utilization balanced enough that routing remains a throughput advantage instead of a new straggler source.

Routing failure modes

Router behavior directly impacts both system performance and model quality. Expert collapse is a training-time router failure in which the router converges to a small subset of experts, leaving others undertrained; the served model then behaves like a small dense model with a massive memory footprint. Routing instability is another training-time failure: assignments oscillate, preventing experts from specializing. Serving introduces runtime overload conditions as well. Distribution shift can make a router trained on formal text overload specific “grammar experts” when served colloquial chat data, creating hotspots. Token dropping occurs when capacity-limited experts drop tokens, skip their computation, and rely only on the residual connection, degrading output quality.

These failure modes follow from the basic MoE serving trade-off:9 computation is dynamically routed to different experts based on input, so the routing pattern becomes a systems workload rather than only a model-internal choice. Popular models like Mixtral (Jiang et al. 2024) use MoE to achieve high capacity with lower inference cost.

9 MoE (Mixture-of-Experts) Serving Trade-Off: Only a subset of experts activate per token (typically 2 of 8–16), keeping per-token FLOPs low while total parameter count reaches hundreds of billions. The serving constraint: all expert weights must reside in memory even though most are idle on any given token, inflating memory footprint 4–8\(\times\) relative to the active compute. This forces expert parallelism across devices with AllToAll communication at every MoE layer.

Jiang, A. Q., A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, et al. 2024. “Mixtral of Experts.” arXiv Preprint arXiv:2401.04088.

Example 1.10: Expert parallelism for Mixtral-8x7B
Scenario: Mixtral-8x7B (46.7B total parameters, ~12.9B active per token) is served across a 4-device GPU cluster using top-2 routing.

Diagnosis: Replicating all 8 experts across all 4 GPUs causes severe memory redundancy. Sharding 2 experts per GPU (4-way expert parallelism) requires AllToAll dispatch/gather collectives (~0.4 ms) to route tokens to expert devices.

Systems lesson: Expert parallelism trades AllToAll communication for reduced per-GPU parameter memory. Processing tokens through 2 sharded experts per GPU lowers MoE layer execution time to ~1.5 ms (vs. ~4 ms for an equivalent dense layer).

Expert parallelism represents the peak of model sharding complexity, requiring tight integration between the model architecture, the routing algorithm, and the physical interconnect topology. The next section examines how recommendation systems use similar sharding techniques for massive embedding tables.

Embedding sharding for recommendation systems

Recommendation systems typically contain embedding tables that dwarf dense model weights in size. Meta’s DLRM reference architecture uses model parallelism over embedding tables to mitigate memory constraints while scaling dense layers separately (Naumov et al. 2019). At production scale, this embedding-heavy structure requires sharding strategies distinct from tensor or pipeline parallelism.

Row-wise sharding partitions embedding tables by row (entity ID):

\[\text{Shard}_i = \{e_j : \text{hash}(j) \bmod N_{\text{shards}} = i\}\]

Each shard contains approximately \(N_{\text{entities}}/N_{\text{shards}}\) embeddings, where \(N_{\text{entities}}\) is the total number of entities and \(N_{\text{shards}}\) is the shard count.

Column-wise sharding partitions each embedding vector across devices:

\[e_j = [e_j^{(0)}, e_j^{(1)}, ..., e_j^{(N_{\text{shards}}-1)}]\]

Each device stores a slice of every embedding. Hybrid sharding combines both approaches: frequently accessed embeddings are column-sharded for faster access, while the long tail uses row sharding. The choice of embedding sharding strategy depends on lookup patterns and communication overhead. Table 16 compares row-wise, column-wise, and hybrid approaches:

Table 16: Embedding Sharding Strategies: Comparison of row-wise, column-wise, and hybrid embedding placement strategies, balancing lookup locality against network communication collectives and memory load balancing.
Sharding Strategy Lookup Pattern Communication Best For
Row-wise Single device per lookup AllToAll gather Uniform access patterns
Column-wise All devices per lookup AllGather Hot embeddings
Hybrid Varies by embedding Mixed Production RecSys

The trade-off is locality against load balance. Row-wise sharding keeps each embedding vector whole on one device, which localizes the lookup but needs an AllToAll gather to collect vectors scattered by entity ID; column-wise sharding parallelizes every lookup across devices at the cost of an AllGather; and hybrid sharding splits the difference, replicating hot embeddings column-wise while row-sharding the cold tail. Figure 14 shows the three layouts, where the row-wise “network gather” is the AllToAll collective named in table 16.

Figure 14: Embedding Sharding Strategies: Row-wise sharding places complete embedding vectors on specific servers based on entity ID, requiring a network gather for lookup. Column-wise sharding splits each vector across all servers, allowing parallel local lookups followed by an AllGather, which is efficient for popular “hot” embeddings. Hybrid sharding combines these approaches, using column sharding for hot items and row sharding for the “cold” long tail to balance load and memory.

All three sharding strategies operate at massive scale in production. Meta’s recommendation infrastructure provides a concrete example of how row-wise, column-wise, and hybrid approaches combine to serve trillion-entity embedding tables.

Lighthouse 1.1: Archetype B (DLRM at Scale): Embedding sharding at Meta
For the Archetype B recommendation workload, Meta’s infrastructure demonstrates embedding sharding at extreme scale:

Scale:

  • Embedding tables: 100+ TB total
  • Unique entities: 10+ trillion
  • Embedding dimension: 128–256
  • Shards: 1,000+ servers

These scale bullets describe infrastructure-level totals, not one dense resident table in which every unique entity has a 128–256-dimensional vector. A dense INT8 table with 10 trillion entities and 128 dimensions would require at least 1,280 TB, while 100 TB across the same entity count averages only 10 bytes per entity. The production system therefore relies on compression, tiering, cold storage, and uneven entity coverage rather than a single fully materialized dense matrix.

Sharding strategy:

  • Hot embeddings (top 1 percent by access frequency): Replicated across all shards
  • Warm embeddings (next 10 percent): Column-sharded with 8-way parallelism
  • Cold embeddings (remaining 89 percent): Row-sharded with consistent hashing

Each inference request requires approximately 5,000 embedding lookups. Without optimization, this would require 5,000 network round trips. Instead, the system applies several optimizations. Batch accumulation collects lookups for 1 ms. Lookup deduplication removes duplicate entities across requests. Shard-aware batching groups lookups by destination shard. Parallel dispatch sends batched requests to all shards simultaneously. Streaming assembly reconstructs embeddings as responses arrive.

The order-of-magnitude effect of these optimizations on round-trip count, lookup latency, and bandwidth appears in table 17:

Table 17: Meta Embedding Sharding Lookup Performance: Network round-trip, lookup-latency, and bandwidth measurements for the recommendation lookup path before and after batching plus shard-aware dispatch, showing the order-of-magnitude effect of these optimizations.
Metric Without Optimization With Optimization
Network round trips 5,000 1 (batched)
Lookup latency 50 ms 2 ms
Network bandwidth 10 Gbps 40 Gbps (burst)

The lighthouse lesson is that DLRM-scale serving is dominated by sparse embedding placement and lookup coordination, not by dense accelerator FLOP/s.

Hybrid sharding strategies

Production systems often combine multiple sharding strategies because no single axis satisfies every inference constraint. Hybrid sharding composes memory capacity, latency, and communication requirements: one axis may keep weights resident, another may reduce per-token latency, and a third may route sparse components without forcing dense layers onto the wrong device. Three recurring combinations illustrate the pattern:

Tensor and pipeline parallelism combine when models require both memory distribution and latency reduction:

8 GPUs organized as 2 by 4 (pipeline stages by tensor parallel):

Stage 0 (Layers 1-40):  TP across GPUs 0,1,2,3
Stage 1 (Layers 41-80): TP across GPUs 4,5,6,7

The combination achieves 4\(\times\) latency reduction (from TP) while handling models requiring 8-way sharding for memory.

Expert and tensor parallelism combine for MoE models where individual experts are large:

Mixtral with large experts:

- Expert parallelism: Distribute 8 experts across 8 GPU groups
- Tensor parallelism: Each expert spread across 2 GPUs
- Total GPUs: 16

Embedding and dense parallelism serve recommendation models with both large embeddings and large dense components:

DLRM-scale model:

- Embedding sharding: 1,000 shards across CPU servers
- Dense model: 8-way tensor parallel across GPUs
- Communication: Embeddings gathered to GPU, processed, returned

Hybrid sharding buys fit or latency by adding new communication patterns at every boundary between shards. The next question is whether the communication tax stays inside the latency and throughput budget.

Communication overhead analysis

The practical speedup from sharding depends critically on communication efficiency. Each sharding strategy has characteristic communication patterns with different bandwidth and latency requirements.

Equation 8 quantifies AllReduce communication time for tensor parallelism, where data is combined from all devices with the result available on all devices.

\[T_{\text{allreduce}} \approx 2(N-1)\alpha + \frac{2(N-1)}{N} \times \frac{M}{\beta} \tag{8}\]

where \(N\) is the number of devices, \(M\) is the collective payload size, \(\alpha\) is startup latency, and \(\beta\) is the interconnect bandwidth. The factor of 2 accounts for the reduce-scatter and all-gather phases. AllReduce derives this ring-AllReduce cost and its bandwidth-optimal \(2(N-1)/N\) scaling, establishing why the per-layer communication tax stays bounded as the tensor-parallel degree grows.

Equation 9 expresses the simpler point-to-point communication for pipeline parallelism, where data flows from one device to the next.

\[T_{\text{p2p}}(n) = \alpha + \frac{n}{\beta} \tag{9}\]

where \(n\) is the point-to-point payload size, \(\alpha\) is the network latency, and \(n/\beta\) is the transfer time. Equation 10 captures the more complex AllToAll communication for expert parallelism, where each device exchanges distinct data with every other device.

\[T_{\text{alltoall}} = (N-1) \times \left(\alpha + \frac{M/N}{\beta}\right) \tag{10}\]

Both equations make clear that the achievable bandwidth \(\beta\) and latency \(\alpha\) of the underlying interconnect determine real-world sharding performance. Production hardware differs enough on both axes that the interconnect choice can determine whether a sharding plan is viable.

Communication overhead depends heavily on the interconnect technology. Table 18 records the physical bandwidth, latency, and intended scope of each candidate interconnect:

Table 18: Datacenter Interconnect Specs: Bandwidth, latency, and intended scope for NVLink, PCIe Gen5, InfiniBand HDR, and 100G Ethernet, establishing the physical limits that bound achievable sharding speedups.
Interconnect Bandwidth Latency Use Case
NVLink (H100) 900 GB/s 500 ns Intra-node TP
PCIe Gen5 64 GB/s 1 μs Intra-node (no NVLink)
InfiniBand HDR 200 Gb/s (25 GB/s) 7 μs Inter-node
Ethernet 100G 100 Gb/s (12.5 GB/s) 50 μs Inter-node (commodity)

Table 19 translates those bandwidths into AllReduce time for an 8-way tensor-parallel layer (activation 8 MB per all-reduce; batch=1, hidden=8,192) as a fraction of a 30 ms transformer-layer budget:

Table 19: AllReduce Cost by Interconnect: AllReduce time for an 8 MB activation across NVLink, InfiniBand, and 100G Ethernet, with each as a fraction of a 30 ms transformer-layer budget, showing why intra-node TP requires NVLink-class bandwidth.
Interconnect AllReduce Time Share of 30 ms Layer
NVLink 0.03 ms 0.10%
InfiniBand 0.56 ms 1.9%
100G Ethernet 1.12 ms 3.7%

The engineering boundary is local: NVLink makes tensor parallelism efficient within a node, InfiniBand can make cross-node tensor parallelism acceptable for carefully chosen workloads, and commodity Ethernet is usually too slow for latency-sensitive inference.

NVLink bandwidth10 has evolved significantly over GPU generations.

10 NVLink Bandwidth Evolution: From 160 GB/s bidirectional on early NVLink implementations to 900 GB/s on Hopper-class GPUs to 1.8 TB/s on Blackwell-class GPUs—roughly a 10\(\times\) improvement across three hardware generations. This bandwidth growth is what made intra-node tensor parallelism across 8 GPUs practical with less than 5 percent communication overhead, directly determining the maximum model size servable without crossing the slower cross-node interconnect.

Sharding strategy selection

The choice of sharding strategy depends on model architecture and deployment priorities. Table 20 evaluates each approach across four critical factors:

Table 20: Sharding Strategy Selection Guide: Evaluation of tensor, pipeline, expert, and embedding sharding across latency targets, throughput priorities, interconnect constraints, and implementation complexity.
Factor Tensor Parallel Pipeline Parallel Expert Parallel Embedding Shard
Latency priority Best Worst Moderate N/A
Throughput priority Good Best (pipelined) Good Best
Interconnect limited Poor fit Good fit Moderate Good fit
Implementation effort High Low Moderate High

Strategy selection starts from the deployment bottleneck. Use tensor parallelism when latency dominates and the interconnect can sustain frequent collectives, pipeline parallelism when throughput matters more than per-request latency, expert parallelism when MoE routing defines the model structure, and embedding sharding when large sparse tables dominate capacity.

Once massively sharded multi-GPU replicas are assembled, dozens of them must run in parallel to handle global traffic. The challenge shifts from the internal mechanics of a single replica to the traffic control layer that routes millions of user queries across the fleet.

Self-Check: Question

  1. Why does Tensor Parallelism (TP) reduce single-request Time-to-First-Token (TTFT) whereas Pipeline Parallelism (PP) provides zero reduction in single-request latency?

    1. TP eliminates the need for activation storage, while PP doubles activation sizes across stages.
    2. TP uses point-to-point transfers over Ethernet, while PP requires NVLink broadcast collectives.
    3. TP relies on asynchronous CPU offloading, whereas PP executes purely on GPU Tensor Cores.
    4. TP partitions intra-layer matrix multiplications across devices with per-layer AllReduce, whereas PP requires a single request to traverse all sequential stages in series.

  2. Explain the ‘MoE economics paradox’ for a 671B-parameter Mixture-of-Experts model (such as DeepSeek-V3 with 37B active parameters per token): why does it require multi-node cluster VRAM for capacity while exhibiting the decode latency of a small dense model?

  3. True or False: In model sharding, Tensor Parallelism relies on AllToAll collective communication to route tokens between layers, whereas Expert Parallelism relies strictly on AllReduce to synchronize partial attention head outputs.

  4. A production recommendation system operates 100+ TB of embedding tables with trillions of entities. How does a hybrid embedding sharding architecture distribute hot, warm, and cold embeddings to balance memory footprint and network overhead?

    1. Hot (top 1%) embeddings are replicated in memory across all shards; warm (10%) are column-sharded with AllGather; cold (89%) are row-sharded on SSDs with AllToAll gather.
    2. Hot embeddings are stored exclusively on remote cold storage; cold embeddings are replicated in HBM across all GPUs.
    3. All embeddings are strictly column-sharded across all GPUs regardless of popularity to eliminate network communication.
    4. Hot embeddings are sharded row-wise across CPUs; cold embeddings are broadcast via AllReduce to all GPU Tensor Cores.

  5. Order the sequence of operations that occurs during token execution in an Expert-Parallel (MoE) transformer layer:

  1. AllToAll dispatch transmits token activation vectors to the specific GPUs hosting the selected experts
  2. Gating network evaluates input representations and computes top-\(k\) routing probabilities
  3. AllToAll gather routes expert output activations back to the originating host devices
  4. Selected expert sub-networks compute local feed-forward transformations on assigned tokens
  5. Weighted sum combines expert outputs with gating coefficients and adds residual connections
  1. Using the \(\alpha\)-\(\beta\) communication model, explain why Tensor Parallelism across nodes linked via standard 100G Ethernet introduces prohibitive latency overhead compared to intra-node NVLink.

See Answers →

Load Balancing and Request Routing

If ten model replicas are actively serving traffic, and a new request arrives asking for a 5,000-word document summarization, sending it to a replica that is already overloaded will trigger a massive latency spike for every user assigned to that node. Simple round-robin load balancing fails catastrophically for generative ML. Production serving requires intelligent request routing that evaluates the internal memory and queue states of every replica in the fleet.

Lighthouse 1.2: Archetype B (DLRM at Scale): The tail at scale
Archetype B (DLRM at Scale) relies on \(P_{99} \le 100\text{ ms}\) performance boundaries. Because high-throughput serving aggregates \(10^7\) QPS, \(10^{-4}\) tail latency events occur \(1{,}000\) times per second. This necessitates two-choice routing to suppress the queuing delay \(T_{\text{queue}}\) that otherwise dominates physical compute \(T_{\text{compute}}\).

Load balancing principles

Two trend lines: a red random-assignment tail curve rises sharply above a flatter blue two-choice routing curve, with the gap shaded red.

Two-choice routing flattens the tail imbalance of random placement.

Load balancing serves two primary goals that sometimes conflict: latency minimization and utilization maximization. Latency minimization routes requests to the replicas that can serve them fastest, considering current queue depth and processing time; utilization maximization spreads load evenly to avoid both idle replicas and overloaded replicas. The tension arises because latency-optimal routing may concentrate load on fast replicas, reducing their performance and leaving other replicas underutilized.

Evaluation therefore needs four signals that cover both tail latency and load balance:

  • Maximum queue length: This determines worst-case latency.
  • Load variance: This measures how evenly work is distributed.
  • Utilization spread: This shows whether some replicas are idle while others are saturated.
  • Decision overhead: This captures the cost of making the routing choice.

A policy that optimizes only one of these signals can look healthy in aggregate while sending unlucky requests into the tail.

Round-robin and random assignment

The simplest load-balancing strategies assign requests without considering server state. Round-robin sends request 1 to server 1, request 2 to server 2, and so on, which gives perfect distribution only when servers are homogeneous and request processing times are identical. Random assignment selects a server uniformly at random for each request; with many requests it converges to an even mean distribution, but its variance is higher than round-robin. Both strategies are reasonable baselines for identical servers, yet production serving rarely satisfies that assumption. Heterogeneous GPU generations, different memory configurations, variable request sizes, warmup behavior, and replicas nearing memory limits all make the state of the server fleet matter.

Under these realistic conditions, uninformed strategies perform poorly because unlucky assignments accumulate in the tail. The maximum queue length under random assignment follows the classical balls-into-bins result (Mitzenmacher 2001), shown in equation 11:

\[E[\text{max queue}] = \Theta\left(\frac{\log R}{\log \log R}\right) \tag{11}\]

where \(R\) is the number of servers or replicas. Evaluating the ratio at \(R = 1{,}000\) gives \(\log R/\log\log R \approx 6.9/1.9 \approx 3.6\), which the leading constant lifts to roughly 4–5 requests in the worst-case queue. This seems small, but the unlucky requests in long queues experience significantly higher latency. The power of two choices (principle 16) provides the theoretical response: one extra queue-length probe changes the tail behavior from \(\mathcal{O}(\log R / \log \log R)\) to \(\mathcal{O}(\log \log R)\).

The power of two choices

A foundational result in load balancing theory (Mitzenmacher 2001) shows that querying just two random servers before making a routing decision provides exponentially better load distribution than random assignment.11

Mitzenmacher, M. 2001. “The Power of Two Choices in Randomized Load Balancing.” IEEE Transactions on Parallel and Distributed Systems 12 (10): 1094–104. https://doi.org/10.1109/71.963420.

11 Balls-into-Bins (Power of Two Choices): With random placement, maximum bin load is \(\Theta(\log R / \log \log R)\); with two random choices and greedy selection, it drops to \(\Theta(\log \log R)\). For a 1,000-server inference fleet, this reduces worst-case queue depth from roughly 5 to roughly 2, cutting P99 tail latency by nearly half while adding only the overhead of a single extra queue-length probe per request.

The procedure is deliberately small, but the load signal must match the serving workload. For a homogeneous vision fleet, current queue length may be enough. For an LLM fleet, active tokens, estimated remaining decode work, and KV-cache pressure better approximate the work a replica has already accepted. Algorithm 2 states a representative routing loop: probe two replicas, compare normalized work, and route without polling the whole fleet.

\begin{algorithm} \caption{Power-of-two choices request routing} \begin{algorithmic} \Require replica set $\mathcal{S}$; request $r$; capacity weights $w_s$; per-replica work signal $W_s$ (queue length, active tokens, predicted work, or KV-cache occupancy) \Ensure a selected replica for $r$ \State sample two candidate replicas $s_a, s_b$ from $\mathcal{S}$ \Comment{capacity-weighted} \State query $W_{s_a}, W_{s_b}$; estimate incremental work $\hat{c}(r)$ (expected output tokens or KV pages) \State $u_s \gets (W_s + \hat{c}(r)) / w_s$ for $s \in \{s_a, s_b\}$ \Comment{post-admission load} \State $s^\star \gets$ candidate with smaller $u_s$ \Comment{ties: cache affinity} \State route $r$ to $s^\star$ \State update $s^\star$'s local admission state \State \Return $s^\star$ \end{algorithmic} \end{algorithm}

The two probes and the one state update per request are \(\mathcal{O}(1)\) coordination rather than an all-replica scan, and the tail-latency win materializes only when the load signal reflects true work: in LLM serving, request count alone can hide a 10-token request behind a 500-token one, while active-token and KV-cache signals expose the risk. The queue-length bound in equation 12 formalizes why this small change matters, reducing maximum queue length from \(\mathcal{O}(\log R / \log \log R)\) to \(\mathcal{O}(\log \log R)\):

\[E[\text{max queue}]_{\text{two choices}} = \Theta(\log \log R) \tag{12}\]

For 1,000 servers, random assignment produces a maximum queue of roughly 4–5 requests, while two choices reduces the maximum to roughly 2. The improvement is exponential: two choices with 1,000 servers achieves better balance than random assignment with just 10 servers.

Theorem 1.2: Power-of-two choices load-balancing bound
For \(R\) servers receiving independent requests, random assignment produces a maximum queue length of \(\mathcal{O}(\log R / \log \log R)\) with high probability, while routing each request to the shorter of two randomly sampled queues reduces the maximum queue length to \(\mathcal{O}(\log \log R)\).

The practical implication is that near-optimal load balancing does not require polling the whole fleet. Two probes avoid the \(R\)-probe cost of exact least-loaded routing, the improvement grows with system size, and the method remains simple enough to implement inside ordinary load balancers. Variants of power-of-two-choices appear in large-scale systems because they offer strong tail behavior without global polling.

The mechanism is intuitive: random assignment occasionally makes poor choices by routing to an already-busy server, and these mistakes compound. With two choices, the algorithm almost never makes the worst choice, avoiding the tail behavior that creates long queues. Mathematically, when \(m\) servers have queue length \(k\), random assignment grows queue length \(k+1\) with probability proportional to \(m/R\). With two choices, that probability drops to \((m/R)^2\), creating a super-exponential decay in queue length distribution.

Weighted and adaptive load balancing

When servers have different capacities, naive load balancing creates imbalance. A mix of A100 GPUs and lower-capacity T4 GPUs receiving equal request rates will overload the T4 servers while leaving A100 capacity idle. Weighted round-robin corrects the mean assignment rate by routing requests proportional to server capacity:

\[\Pr(\text{route to server } i) = \frac{w_i}{\sum_j w_j}\]

where \(w_i\) is the weight, or capacity, of server \(i\). Weighted two-choices applies the same idea to tail control: sample two servers with probability proportional to their weights, compare current load relative to capacity, and route to the lower relative load. The weighted-routing example makes this policy concrete for mixed GPU hardware.

Example 1.11: Heterogeneous GPU cluster
Scenario: A serving cluster mixes 10 H100 GPUs (1000 QPS each) and 20 A100 GPUs (600 QPS each), processing 15,000 QPS target traffic.

Diagnosis: Unweighted round-robin routes equal request counts (500 QPS) to all nodes, leaving H100s underutilized (50 percent) while overloading A100s (83.3 percent). Weighted routing assigns H100s a weight of 4.5 percent and A100s a weight of 2.7 percent, balancing utilization across tiers (68.2 percent vs. 68.2 percent).

Systems lesson: Heterogeneous inference clusters require capacity-weighted routing. Equal request volumes produce unbalanced load when hardware tiers have different service rates.

Static weights handle known capacity differences, but production replicas also change over time. Adaptive load balancing adjusts weights dynamically based on observed performance:

For each server i:
    latency[i] = exponential_moving_average(observed_latency)
    weight[i] = 1 / latency[i]  # Inverse latency weighting

Inverse latency weighting lowers the probability of routing to servers whose observed latency has risen, whether the cause is memory pressure, thermal throttling, request mix, or background work consuming resources.

Least-connections load balancing

An alternative to random selection is routing to the server with the fewest active connections or shortest queue. Least-connections maintains an active-request count for each server, routes a new request to the minimum count, increments on dispatch, and decrements on completion. For long-running requests, common in LLM serving, this outperforms round-robin because it accounts for work still in progress rather than only historical assignment order.

The challenge is maintaining accurate connection counts in a distributed system. A centralized counter gives a single source of truth but can become a bottleneck. Distributed counters with gossip scale better but may route using stale information. Sampled least-connections combines the idea with two choices by probing only a subset of servers and choosing the minimum.

Example 1.12: Least-connections for LLM serving
Scenario: An LLM serving fleet handles requests with highly variable output sequence lengths (500 ms for 10 tokens vs. 25 seconds for 500 tokens).

Diagnosis: Round-robin routing assigns work based on historical request count, leaving nodes processing long generations overloaded (P99 = 45s) while idle nodes wait. Least-connections routes new work to nodes with the fewest active in-flight requests.

Systems lesson: Least-connections routing accounts for in-flight work remaining rather than historical assignment count. For LLM serving with variable output lengths, least-connections drops P99 tail latency by 38 percent (45s to 28s).

Consistent hashing for stateful routing

Many inference workloads maintain state that benefits from routing affinity. LLM conversations reuse KV cache from previous turns, recommendation sessions carry user context and recent interactions, and streaming inference may depend on model state from previous frames. For these workloads, routing the same user or session to the same server improves performance by avoiding cache misses and state reconstruction.

Consistent Hashing12 (Karger et al. 1997) maps requests to servers based on a hash of the routing key (user ID, session ID):

12 Consistent Hashing: A ring-based load distribution algorithm originally deployed at Akamai for CDN routing. Servers are mapped onto a virtual ring so that adding or removing a node remaps only \(K/N_{\text{servers}}\) keys on average. For LLM serving, this minimal-disruption property preserves KV cache locality during autoscaling events: without it, every scale-up would invalidate cached conversation state across the fleet, forcing expensive recomputation.

Karger, D., E. Lehman, T. Leighton, R. Panigrahy, M. Levine, and D. Lewin. 1997. “Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web.” Proceedings of the Twenty-Ninth Annual ACM Symposium on Theory of Computing - STOC ’97, 654–63. https://doi.org/10.1145/258533.258660.

\[\text{server}(request) = \operatorname{arg\,min}_{s \in \mathcal{S}_{\text{srv}}} \text{distance}(\text{hash}(key), \text{hash}(s))\]

where \(\mathcal{S}_{\text{srv}}\) is the set of servers mapped onto the ring, and each request routes to the nearest server clockwise.

The important properties are determinism, minimal disruption, and balance. The same key always routes to the same server, adding or removing servers remaps only \(K/N_{\text{servers}}\) keys on average, and virtual nodes distribute load evenly enough to prevent one physical server from owning a disproportionate key range. For LLM serving, the most direct payoff is keeping each user’s requests on the server that already holds their session state.

Example 1.13: Consistent hashing for KV cache affinity
Scenario: A multi-turn LLM chat service routes user queries across a cluster of inference replicas without session affinity.

Diagnosis: Un-affinitized routing sends subsequent conversation turns to different replicas, forcing every turn to rebuild the KV cache from scratch (500 ms penalty, wasting 4.5s over a 10-turn conversation). Consistent hashing maps user IDs deterministically to specific replicas.

Systems lesson: Consistent hashing preserves KV cache locality across multi-turn sessions. Hashing user sessions to virtual nodes on a hash ring eliminates redundant prefill KV-cache reconstruction while ensuring minimal key remapping (\(1/N_{\text{servers}}\)) during server failures or autoscaling.

The same stateful routing machinery becomes a correctness boundary when cached state is user-specific.

War Story 1.1: When cache state crossed users (2023)
Context: ChatGPT used Redis Cluster through the redis-py client to cache user information and conversation titles in a high-volume serving system (OpenAI 2023).

Mechanism: Under an Asyncio connection-pooling race condition, canceled requests left unread responses in recycled connection buffers, causing subsequent requests to receive cache data belonging to previous users.

Impact: On March 20, 2023, OpenAI took ChatGPT offline after users reported seeing other active users’ conversation titles and payment details.

Fix: OpenAI patched redis-py connection handling, forced strict connection isolation per request context, and purged affected Redis cache keys.

Systems lesson: Stateful inference infrastructure needs isolation guarantees around caches, connection pools, and routing keys. A cache is part of the serving correctness boundary, not merely an optimization.

OpenAI. 2023. March 20 ChatGPT Outage: Here’s What Happened. OpenAI incident report.

Request routing for sharded models

The sharding strategies examined in section 1.4 introduce routing complexity: a single inference request may require computation on multiple devices, necessitating coordination. The routing pattern depends on the sharding strategy.

Under tensor parallelism, each request is broadcast to all devices in the shard group. Each device processes its portion of each layer, and results are synchronized via AllReduce. Figure 15 shows this fan-out, compute, and gather pattern.

Figure 15: Tensor Parallelism Request Routing: A request is broadcast from the load balancer to all devices in the shard group. Each GPU computes its assigned attention head partition, then a single AllReduce synchronizes results before the response is returned. Interconnect bandwidth—not model size—determines latency.

The key constraint visible in figure 15 is that every request requires an AllReduce synchronization across all devices in the shard group, making interconnect bandwidth the latency-determining factor rather than model size.

Under pipeline parallelism, each request traverses stages sequentially, with each stage forwarding activations to the next. Figure 16 illustrates how pipelining multiple requests achieves throughput scaling even though single-request latency equals the sum of all stage times.

Figure 16: Pipeline Parallelism Request Flow: (a) A single request flows sequentially through four stages—no latency benefit from parallelism. (b) Multiple requests fill the pipeline: once steady state is reached, throughput scales with the number of pipeline stages, while per-request latency remains approximately constant.

The critical insight from figure 16 is that pipeline parallelism offers no single-request latency benefit: one request still traverses all stages sequentially. Throughput scales only when multiple concurrent requests fill the pipeline, reaching steady-state throughput proportional to the number of stages.

Under expert parallelism, each request is dispatched to the devices hosting its selected experts based on the gating decision. Two AllToAll communication steps bookend the expert compute (figure 17).

Figure 17: Expert Parallelism (MoE) Request Flow: Gating selects the top-\(k\) experts; an AllToAll dispatch sends tokens to the GPUs hosting those experts; experts compute in parallel; a second AllToAll gathers results. The lower panel contrasts unbalanced routing (one expert bottlenecks) against balanced routing (uniform utilization via auxiliary loss).

These two-level AllToAll communication steps make expert parallelism uniquely sensitive to load imbalance: if the gating function concentrates tokens on a few experts, those devices become bottlenecks while others sit idle.

When multiple shard groups provide horizontal scaling, the load balancer routes to groups rather than individual devices. Figure 18 shows this two-level hierarchy: standard load-balancing algorithms (round-robin, two-choices, consistent hashing) operate at the group level, while each group handles its own internal AllReduce locally.

Figure 18: Horizontal Scaling with Shard Groups: The load balancer treats each shard group as a single logical inference server. Within each group, 8 GPUs coordinate via tensor parallelism. Adding groups scales throughput linearly; adding GPUs per group reduces per-request latency.

The two-level hierarchy in figure 18 separates concerns: the load balancer treats each shard group as a single logical server using standard algorithms, while internal AllReduce coordination remains local to each group. Adding shard groups scales throughput linearly; adding GPUs per group reduces per-request latency.

Health checking and failover

Load balancers can route around failure only if the health signal matches the failure mode. For ML serving, a process that is alive may still be unusable because weights are not loaded, the GPU memory pool is exhausted, or the first request will trigger a long warm-up. Production systems therefore layer health checks from cheapest to most realistic.

Liveness probes verify that the server process is running:

GET /health/live
Response: 200 OK (process alive) or timeout (process dead)

Readiness probes go further, verifying the server can handle requests (model loaded, GPU initialized):

GET /health/ready
Response: 200 OK (ready to serve) or 503 (not ready)

Deep health checks verify that actual inference works by running a test request:

POST /health/inference
Body: {"prompt": "test"}
Response: 200 OK with valid output, or error

Deep health checks are the primary defense against a failure mode that does not exist in traditional web servers: silent hardware degradation. A GPU with degraded HBM modules may continue to respond to process-level HTTP 200 checks while failing silently on real requests due to memory pressure or un-warmed model state.

Example 1.14: Health checks for GPU inference
Scenario: GPU inference servers pass standard process-level HTTP 200 checks while failing silently on real requests due to memory pressure or un-warmed model state.

Diagnosis: Standard probes fail to check GPU memory allocation (MIN_REQUEST_MEMORY) or model warmup state, allowing degraded nodes to receive production traffic and cause generation failures.

Systems lesson: GPU inference health checking requires tiered probing (Liveness, Readiness, Deep Health). Readiness probes must validate GPU memory availability and warm-up state, while deep health checks validate logit output distributions against expected bounds.

Quantitative analysis: Load balancing impact

The choice of load balancing algorithm has quantitative impact on system performance. Consider a system with 100 servers, 10,000 QPS, and variable request sizes (CV = 0.5). Table 21 quantifies the latency and overhead trade-offs:

Table 21: Load Balancing Algorithm Comparison: Trade-offs across random, round-robin, power-of-two-choices, least-connections, and hybrid routing, showing how stateful probes cut P99 latency at the expense of control-plane overhead.
Algorithm Max Queue P99 Latency CPU Overhead
Random 4.2 requests 45 ms Minimal
Round-robin 2.8 requests 32 ms Minimal
Two-choices 1.9 requests 24 ms 2 probes/request
Least-connections 1.4 requests 19 ms Global state
Two-choices + LC 1.2 requests 17 ms 2 probes + state

Table 21 shows the familiar serving trade-off: random and round-robin routing keep CPU overhead near zero but leave longer queues and higher latency variance; two-choices roughly halves p99 latency with only two probes per request; least-connections improves further when request cost varies but requires global state; and combining two-choices with least-connections produces the shortest queues at the highest implementation complexity.

For many serving systems, two-choices provides a strong trade-off between performance improvement and implementation complexity. Least-connections adds value for workloads with high request size variance (LLM serving, recommendation ranking).

Circuit breakers and backpressure

When a GPU inference server slows down, routing more requests to it can turn a local slowdown into a fleet-wide failure. Thermal throttling is a typical trigger: a replica that normally serves a request in 50 ms begins taking 500 ms, queues fill behind it, client timeouts generate retries, and those retries push load onto neighboring replicas. The load balancer needs a way to stop treating that replica as merely slow and start treating it as unavailable.

The Circuit Breaker Pattern13 (Nygard 2007) provides that control. In the closed state, the server receives normal traffic. When error rate, timeout rate, or latency exceeds a threshold, the breaker opens and requests fail fast or route elsewhere instead of waiting behind a saturated queue. After a recovery interval, the breaker becomes half-open: it admits a small number of probe requests, closes if they succeed, and reopens if they fail. The important systems property is bounded blast radius. The failed replica loses traffic quickly enough that its queue cannot export overload to the rest of the fleet.

13 Circuit Breaker: Named after the electrical safety device that cuts current before wires overheat. In inference serving, the three-state machine (closed, open, half-open) prevents a single overloaded GPU replica from cascading failure to the entire fleet: once error rate exceeds threshold, the breaker opens and fails requests fast (microseconds) rather than waiting for timeout (seconds), protecting upstream callers and preserving capacity for healthy replicas.

Backpressure propagation complements the breaker by making overload visible upstream. When queue depth crosses a threshold, the server returns a signal such as 503 Service Unavailable; the load balancer marks the replica as degraded and routes fewer requests to it. If all replicas become degraded, the system shifts from routing control to admission control, rejecting some requests at the edge rather than accepting work it cannot serve within the latency SLO. Circuit breakers isolate unhealthy replicas; backpressure tells the rest of the system to slow down before retries amplify the failure.

A saturated LLM fleet under KV cache pressure has three relief valves that generic backpressure cannot provide:

  • Context eviction: Evict older conversational turns from the KV cache to reduce context window and free memory.
  • Speculative decoding: Disable speculative decoding to reclaim the GPU memory its draft model occupies.
  • Fallback routing: Route new requests to a smaller quantized fallback model until the primary queue drains.

Each response trades output quality or latency for continued availability, which is often preferable to rejection from the user’s perspective. The system designer must specify which degradation modes are acceptable at each pressure tier, because the correct trade-off depends on the application, a real-time assistant tolerates context truncation better than silent quality degradation.

Example 1.15: Cascading failure prevention
Scenario: A single GPU inference server experiences thermal throttling (latency spiking from 50 ms to 500 ms) in a multi-replica serving fleet.

Diagnosis: Unprotected load balancers continue sending traffic to the slow replica, filling queues and generating client retries that cascade overload to healthy neighboring servers.

Systems lesson: Circuit breakers isolate degraded GPU replicas before retry traffic causes fleet-wide failure. Opening the breaker when error or latency thresholds are breached (50 percent error rate) fails fast and preserves capacity across remaining replicas.

Efficient routing assumes that inference servers own the underlying hardware. In enterprise environments, however, critical production models often run on the same physical clusters as experimental models and developer endpoints. Preventing a rogue query from degrading a production service requires strict multi-tenancy and resource isolation.

Self-Check: Question

  1. In a 1,000-server inference fleet, why does the Power-of-Two-Choices load balancing algorithm dramatically outperform random assignment while avoiding the overhead of global least-loaded routing?

    1. It eliminates all network packet headers by broadcasting requests to all 1,000 servers simultaneously.
    2. It probes two random replicas and picks the less loaded one, reducing maximum queue depth from \(\Theta(\frac{\log R}{\log \log R}) \approx 4\text{--}5\) to \(\Theta(\log \log R) \approx 2\) with \(\mathcal{O}(1)\) overhead.
    3. It guarantees zero queuing delay by dynamically compiling new CUDA kernels on the fly for each request.
    4. It forces all requests to execute on CPU fallback nodes whenever GPU utilization exceeds 50%.

  2. Explain why Consistent Hashing is critical for routing multi-turn LLM conversations, and describe what happens to KV cache state when an autoscaling event adds a new replica.

  3. True or False: Standard HTTP 200 process liveness checks are sufficient to detect GPU inference degradation caused by thermal throttling or silent HBM memory pool exhaustion.

  4. A heterogeneous inference cluster contains 10 H100 servers (delivering 1,000 QPS capacity each) and 20 A100 servers (delivering 600 QPS capacity each), serving a total target load of 15,000 QPS. What weighted routing ratio should be applied to achieve equal utilization across all hardware tiers?

    1. Route 500 QPS equally to every server regardless of type, producing 50% utilization on H100 and 83.3% on A100.
    2. Assign weight 1.0 to H100 and 1.0 to A100 so all servers receive equal traffic shares.
    3. Assign capacity-proportional weights (H100 weight \(= 1000/22000 \approx 4.55\%\) per server; A100 weight \(= 600/22000 \approx 2.73\%\) per server), achieving uniform 68.2% utilization across all nodes.
    4. Route 100% of traffic to A100 servers first and overflow remaining traffic to H100 servers.

  5. The three-state resilience pattern that fails fast and stops routing traffic to a degraded or thermally throttled GPU replica before retries cascade across the fleet is called a ____.

See Answers →

Multi-Tenancy and Isolation

Consider a critical customer-support chatbot running on the same GPU server as an experimental computer vision script. If the experimental workload accidentally monopolizes the PCIe bus or memory bandwidth, the customer-facing chatbot begins timing out, violating SLAs. Multi-tenancy and isolation ensure that co-located workloads save money without sacrificing the predictability of flagship services.

The multi-tenancy challenge

Multi-tenancy saves cost through cost efficiency, operational simplicity, and statistical multiplexing. Sharing infrastructure raises resource utilization, reduces the number of clusters to manage, and makes aggregate traffic more predictable than any one tenant’s traffic.

A red tenant workload fans gray arrows to five blue peer workloads, showing one source degrading many shared dependents.

One tenant’s burst can degrade every workload sharing the pool.

The same sharing creates the isolation problem and multi-tenancy trade-offs that table 22 frames: noisy neighbors can make one tenant’s burst degrade others, resource contention spans GPU memory, network bandwidth, and CPU cycles, security boundaries must protect tenant data, and SLO complexity increases when tenants have different requirements.

Table 22: Single vs. Multi-Tenant Tradeoffs: Multi-tenant deployments double GPU utilization and cut per-request cost but introduce noisy-neighbor risks that demand strict isolation controls.
Aspect Single-Tenant Multi-Tenant
Resource utilization 30–50% 70-90%
Cost per request Higher Lower
SLO guarantees Simple Complex
Isolation Complete Requires engineering
Operational overhead Higher (many clusters) Lower (fewer clusters)

Noisy neighbor problems

The noisy neighbor problem occurs when one tenant’s workload degrades performance for others sharing the same infrastructure. The interference manifests across three resource dimensions simultaneously.

GPU memory contention is the most severe: a tenant with unexpectedly long sequences can consume a disproportionate share of the KV cache pool. Consider three tenants sharing a 60 GB KV cache pool with equal 20 GB allocations. When one tenant begins issuing long-context requests, its allocation can swell to 45 GB, forcing evictions that reduce the other tenants from 200 concurrent sequences each to 75 – a 62 percent batch size reduction that directly degrades their throughput. Network bandwidth saturation compounds this effect when a tenant streaming many large responses consumes the available egress capacity. GPU time-sharing between tenants introduces context-switching overhead and unpredictable latency variance. Measuring noisy-neighbor impact requires capturing all three interference dimensions simultaneously.

Example 1.16: Quantifying noisy neighbor impact
Scenario: An inference platform serves 10 tenants on shared H100 GPUs at 100 QPS each under a 50 ms P99 SLO.

Diagnosis: Tenant 3 bursts to 500 QPS (5\(\times\) baseline). Without isolation, KV cache memory and compute contention cascade into P99 latency violations (22–45 ms) for all 10 tenants on the shared pool.

Systems lesson: Shared capacity increases utilization only when coupled with strict isolation. Unprotected shared pools collapse effective SLOs to the worst-case burst of any single tenant.

Example 1.17: Noisy neighbor with per-tenant quotas
Scenario: Tenant 3 bursts to 500 QPS on a shared inference cluster with per-tenant resource quotas enabled.

Diagnosis: Quota enforcement throttles Tenant 3 to 120 QPS (violating only Tenant 3’s SLO), while Tenants 1–2 and 4–10 maintain normal 15 ms latency.

Systems lesson: Per-tenant quotas transform platform-wide failures into isolated admission-control decisions, protecting co-located tenants from noisy-neighbor bursts.

Resource quotas and fair sharing

Resource quotas limit what each tenant can consume, preventing any single tenant from monopolizing shared resources. Hard quotas turn shared serving into admission control: the system checks concurrency, KV-cache memory, and request rate before admitting each request. Listing 1 uses those three gates so a tenant can be throttled at the scarce resource boundary instead of degrading the whole batch.

Listing 1: Resource Quotas: Admission control enforcement that evaluates per-tenant limits on concurrency, KV-cache memory allocation, and request rate before admitting requests. 
class TenantQuota:
    max_concurrent_requests: int  # e.g., 100
    max_kv_cache_mb: int  # e.g., 20,000
    max_qps: int  # e.g., 1,000
    max_batch_tokens: int  # e.g., 50,000


def admit_request(tenant_id, request):
    quota = get_quota(tenant_id)
    usage = get_usage(tenant_id)

    if usage.concurrent >= quota.max_concurrent_requests:
        return RateLimitError("concurrent request limit")
    if usage.kv_cache_mb >= quota.max_kv_cache_mb:
        return RateLimitError("memory limit")
    if usage.qps >= quota.max_qps:
        return RateLimitError("rate limit")

    return admit(request)

The invariant is deny before interference. A request that would exceed any tenant quota never enters the scheduler, which keeps protected tenants from paying for another tenant’s burst with higher queueing delay or KV-cache pressure.

Soft quotas with fair sharing provide a more flexible alternative. Each tenant has a nominal token generation quota (for example, 10,000 tokens/sec), but when the cluster is underutilized (50 percent capacity), tenants can burst to 2\(\times\) their quota. When the cluster saturates (90 percent capacity), quotas are enforced. This approach maximizes utilization during low-traffic periods while protecting tenants during contention.

When total demand exceeds capacity, max-min fairness allocates resources to maximize the minimum allocation across tenants. The algorithm first gives each tenant an equal share, then redistributes unused capacity from tenants whose demand falls below their equal share. Raw request counts are an inadequate unit for this calculation because, as established in this section, equal request counts are not equal load: a tenant issuing long-context summarization requests consumes far more KV cache and decode compute than a tenant issuing short classification queries. Token generation rate (tokens/sec) serves as the appropriate bounding metric, capturing both throughput and memory pressure. For three tenants with demands of 30,000, 20,000, and 80,000 tokens/sec competing for a GPU pool capable of 100,000 tokens/sec, max-min fairness yields allocations of 30,000, 20,000, and 50,000: each tenant receives up to its demand or its fair share, whichever is less, with excess capacity redistributed proportionally.

Priority scheduling

When tenants have different SLO requirements, priority scheduling ensures that high-priority requests receive resources first. Table 23 shows that preemption rights and resource guarantees move together down the three classes: critical traffic both preempts any lower class and reserves capacity, while best-effort never preempts and gets no guarantee, so an SLO tier maps directly onto a position in the preemption ladder.

Table 23: Priority Scheduling Classes: Three-tier priority hierarchy for multi-tenant inference. Critical traffic gets 100 percent reserved capacity and can preempt any lower class; standard traffic shares capacity proportionally and can preempt best-effort; best-effort runs on leftover capacity with no guarantees.
Class Use Case Preemption Resource Guarantee
Critical Revenue-generating Can preempt lower 100% reserved
Standard General traffic Can preempt best-effort Weighted share
Best-effort Background, batch Cannot preempt No guarantee

The scheduler sorts incoming requests by priority class, then by arrival time within each class. When a new critical request arrives, it jumps ahead of all standard and best-effort requests in the queue, ensuring that revenue-generating traffic never waits behind background batch jobs.

Preemption extends this principle to requests already in flight. When a critical request arrives and all GPU slots are occupied by lower-priority work, the scheduler selects a victim from the lowest priority class, saves its KV cache state to CPU memory, and yields the GPU slot. Once the critical request completes, the preempted request resumes from its saved state rather than restarting from scratch. This checkpoint-and-resume mechanism makes preemption practical for autoregressive generation, where discarding partial output would waste all tokens generated so far.

Bulkhead pattern

The bulkhead pattern14 (Nygard 2007) physically isolates tenant workloads, preventing failures from propagating across tenants. The pattern is named after ship compartments that contain flooding to isolated sections.

14 [offset=-30mm] Bulkhead Pattern: Named after ship compartments that contain flooding to isolated sections, which failed on the Titanic when water cascaded over incomplete hull dividers. The lesson for multi-tenant inference: shared GPU pools with soft quotas provide only partial isolation, as a single tenant’s burst can exhaust memory or saturate bandwidth for all others. Effective bulkheads require dedicated hardware resources per critical tenant.

Nygard, Michael T. 2007. Release It!: Design and Deploy Production-Ready Software. Pragmatic Bookshelf.

The strongest form of bulkhead dedicates entire replicas to specific tenants or tenant groups. Figure 19 illustrates this deployment-level isolation between gold and standard tiers:

Figure 19: Bulkhead Isolation Patterns: To prevent cascading failures in multi-tenant systems, bulkheads isolate resources. Deployment-level bulkheads (shown) assign dedicated physical replicas to high-priority tenants, ensuring complete isolation. Request-level bulkheads enforce strict concurrency limits within shared processes. Like ship compartments, these boundaries ensure that a failure or resource exhaustion in one segment cannot sink the entire platform.

Deployment-level bulkheads provide complete isolation for premium tenants, ensuring that their performance is never affected by other workloads. The trade-off is lower overall resource utilization and increased operational overhead from managing dedicated infrastructure.

Request-level bulkheads complement this physical separation by limiting what any single request can consume within a shared process. Capping input length (for example, 8,000 tokens), output length (2,000 tokens), and execution time (30 seconds) prevents a single pathological request from monopolizing a GPU for minutes while other requests queue behind it.

Failure isolation follows naturally from bulkhead boundaries. When a tenant submits malformed input that triggers a model error, the bulkhead confines the failure to that tenant’s request; other tenants continue processing normally rather than sharing a crashed inference worker. In multi-tenant deployments, these boundaries typically align with service tiers, each with distinct resource guarantees.

Example 1.18: Bulkhead configuration for API tiers
Scenario: An LLM API platform serves Enterprise, Professional, and Free service tiers across shared and dedicated hardware.

Diagnosis: Unpartitioned multi-tenant pools allow Free-tier traffic spikes or malformed requests to degrade Enterprise SLOs. Bulkhead isolation isolates Enterprise on dedicated GPU pools (99.9 percent availability), allocates 70 percent capacity to Professional, and caps Free tier at 30 percent best-effort capacity.

Systems lesson: Bulkheads concentrate isolation overhead where business requirements dictate. Hardware partitioning guarantees SLA compliance for high-value tiers while maximizing utilization across lower tiers.

Example 1.19: Cold start timeline for Llama-70B
Scenario: An autoscaling engine provisions a new Llama-70B serving replica on an H100 GPU instance during a sudden traffic surge.

Diagnosis: Bringing up a fresh replica takes ~7 minutes (\(T_{\text{cold start}} = T_{\text{provision}} + T_{\text{load}} + T_{\text{warmup}}\)), dominated by 5 minutes downloading 140 GB of model weights from S3 and 1 minute GPU loading/warmup. Reactive autoscaling fails to absorb the surge before requests time out.

Systems lesson: Cold-start latency for multi-gigabyte LLMs is dominated by weight download and initialization time. Scaling engines must maintain pre-warmed spare pools or trigger predictive scale-ups 5–10 minutes ahead of anticipated demand spikes.

Example 1.20: The ChatGPT traffic spike
Scenario: ChatGPT scaled from 0 to 100 million active users in two months, creating unprecedented traffic surges across global inference clusters.

Diagnosis: Rapid user adoption triggered severe cold-start bottlenecks and extreme KV cache memory pressure, causing fleet-wide queue saturation and request timeouts under unoptimized serving stacks.

Systems lesson: Internet-scale traffic spikes transform cold-start latency and KV-cache footprint from infrastructure tuning details into binding system architecture constraints. Sustaining rapid growth requires combining pre-warmed GPU pools with iteration-level continuous batching and stateful routing.

Example 1.21: Predictive scaling for traffic
Scenario: A commercial chatbot service faces predictable diurnal traffic patterns, ramping from 500 QPS at night to 3,000 QPS at peak.

Diagnosis: Reactive scaling triggers scale-ups only after traffic arrives, causing 7-minute cold-start latency overloads during morning traffic ramps (06:00 and 09:00 UTC).

Systems lesson: Predictive autoscaling converts diurnal time-series forecasts into scheduled pre-warming windows, as shown in the schedule breakdown (table 24). Triggering scale-up actions 30 minutes before expected ramps ensures replicas are fully warmed before traffic arrives.

Table 24: Predictive Scaling Schedule: Per-time-bucket scaling actions, active replicas, and replicas in cold-start warming for the chatbot workload.
Time Action Replicas Active Replicas Starting
05:30 Scale up 5 +10 warming
06:00 Traffic ramp 15 -
08:30 Scale up 15 +15 warming
09:00 Peak traffic 30 -
17:00 Scale down 20 -10 terminating
20:00 Scale down 10 -10 terminating
00:00 Scale down 5 -5 terminating

Systems insight: A reactive-only autoscaler must over-provision during each ramp (about 45 replicas at peak) because it cannot anticipate the curve. Predictive scaling sizes the fleet to the actual peak (30 replicas), recovering roughly 33 percent of GPU spend without sacrificing latency headroom.

The cost savings demonstrated in example 1.21 depend on anticipating traffic before it arrives. Proactive provisioning avoids the SLO violations that reactive-only systems suffer during ramp-up periods (figure 20).

Figure 20: Predictive vs. Reactive Scaling: Reactive scaling (red line) responds to traffic spikes after they occur, leading to periods of under-provisioning where SLOs are violated due to cold start latency. Predictive scaling (blue line) anticipates traffic and begins provisioning capacity before the spike arrives, ensuring consistent performance.

Warm pool management

Maintaining a pool of prewarmed replicas reduces effective cold start time by buying idle readiness before demand arrives. Warm pool sizing starts from the expected spike and per-replica throughput:

\[\text{Warm pool size} = \frac{\text{Max expected spike}}{\text{Per-replica throughput}} \times \text{Headroom factor}\]

For example, if the maximum expected spike is 2\(\times\) normal and the headroom factor is 1.5, the fleet needs three times the minimum pool capacity available before the spike arrives:

\[\text{Warm pool} = 2 \times 1.5 = 3 \times \text{minimum pool capacity}\]

The cost of maintaining warm replicas is \(\text{Pool size} \times \text{GPU cost/hour} \times \text{Idle fraction}\), creating a direct trade-off between response speed and idle-capacity expense. Tiered warm pools, summarized in table 25, resolve this trade-off by maintaining replicas at different readiness levels, each with different activation latency and cost:

Table 25: Tiered Warm Pool Readiness Levels: Three readiness tiers trade activation latency against idle-capacity cost. Hot replicas absorb bursts instantly at full cost; warm replicas activate in 30 seconds at 60 percent cost; cold replicas cost nothing but take five minutes to come online. Production deployments combine a small hot pool, a larger warm pool, and unlimited cold capacity.
Tier State Response Time Cost (relative)
Hot GPU loaded, running Instant 100%
Warm GPU allocated, model loaded 30s 60%
Cold GPU not allocated 5+ min 0%

When a traffic spike hits, the scaling sequence activates hot replicas immediately, promotes warm replicas within 30 seconds, and cold-starts new instances only if demand persists beyond the warm pool’s capacity. A typical configuration maintains 2 hot replicas for instant burst absorption and 5 warm replicas for sustained spikes, with unlimited cold capacity available from the cloud provider.

Scaling response time analysis

The scaling response budget is the sum of the control loop’s detection, decision, provisioning, and warmup delays. Equation 13 makes those phases explicit:

\[T_{\text{response}} = T_{\text{detect}} + T_{\text{decide}} + T_{\text{provision}} + T_{\text{warmup}} \tag{13}\]

Table 26 decomposes each term and identifies its primary optimization path:

Table 26: Scaling Response Time Components: The four phases of a scaling event with typical durations and their primary optimization lever. Provisioning dominates the budget by an order of magnitude, which is why warm pools are a high-leverage optimization for latency-sensitive serving fleets.
Component Duration Optimization
Detection 10–60s Reduce metrics interval
Decision 1–5s Faster autoscaler
Provisioning 30s–5min Warm pools
Warmup 5–30s Precompilation

Each component offers distinct optimization opportunities. Detection speed improves by collecting metrics at 1-second intervals rather than 60-second intervals, at the cost of noisier signals and higher metric volume. Decision speed improves by precomputing scaling plans for predicted scenarios so that when a trigger fires, the system executes a precomputed plan rather than computing one from scratch. Provisioning speed improves most dramatically through warm pools, which eliminate the provisioning phase entirely for anticipated demand. Warmup speed improves through precompiled inference engines that skip just-in-time (JIT) compilation, reducing the final phase from 30 seconds to under 5 seconds.

Spot and preemptible instances

Cloud providers may offer discounted GPU instances15 that can be reclaimed with short notice, so the scheduling decision is whether the workload can drain, reroute, or recompute without violating its SLO. Table 27 lays out the three GPU instance classes and the workloads each suits best:

15 Spot Instance Economics: Cloud providers have sold unused GPU capacity at large discounts, sometimes with 30-second to 2-minute termination notice. For inference, termination destroys in-flight requests and KV cache state, requiring graceful drain logic and rapid request re-routing. The savings can justify this complexity for burst and best-effort traffic tiers but are unsuitable for SLO-critical paths where termination would violate latency guarantees.

Table 27: GPU Instance Types by Pricing and Reliability: Three GPU instance classes trade discount against interruption risk. On-demand is the reference price with no interruption; reserved instances trade upfront commitment for moderate discount; spot instances trade preemption risk for large savings. Production fleets often run a baseline of reserved + on-demand instances and burst into spot capacity for best-effort traffic.
Instance Type Discount Interruption Notice Use Case
On-demand 0% Never SLO-critical
Reserved 30-60% Never Steady baseline
Spot/Preemptible 60–90% 30s–2min Burst capacity

Spot termination handling is a race against the provider’s warning window. Listing 2 orders the shutdown path so new work stops first, in-flight work drains while time remains, recoverable state is persisted, and the load balancer stops sending traffic before the instance disappears.

Listing 2: Spot Termination Handling: Graceful shutdown sequence that stops admission, deregisters from the load balancer, drains in-flight requests, and persists recoverable state. 
async def handle_spot_termination():
    # Received 2-minute warning
    # 1. Stop admission and remove the replica from routing
    stop_accepting_requests()
    deregister_from_loadbalancer()

    # 2. Complete in-flight requests (if possible)
    await complete_inflight(timeout=90)

    # 3. Save state for resumption elsewhere
    save_kv_cache_to_storage()

    # 4. Terminate gracefully
    shutdown()

The sequence matters because the steps are not interchangeable. Stopping admission and deregistering first prevent new requests from arriving during the drain window; the handler then completes in-flight work, persists recoverable state, and shuts down before the termination deadline.

A spot-aware architecture splits traffic between on-demand and spot replicas by SLO class (figure 21). SLO-critical requests always route to on-demand capacity, while best-effort requests absorb the cost savings and interruption risk of spot instances.

Figure 21: Spot-Aware Traffic Distribution: The load balancer routes 70 percent of traffic to on-demand replicas (guaranteed availability) and 30 percent to spot replicas (cost savings, preemption risk). SLO-critical requests always use on-demand; best-effort requests tolerate occasional retries on eviction. In the representative discount scenario, the mixed fleet achieves roughly 18–27 percent cost reduction vs. 100 percent on-demand.

The split architecture in figure 21 achieves approximately 18–27 percent cost reduction compared to a fully on-demand fleet in the representative discount scenario, while absorbing preemption risk only for best-effort traffic and keeping SLO-critical requests on guaranteed capacity.

Global inference infrastructure

Extending resource isolation and failure boundaries beyond a single data center leads to global infrastructure. Global serving becomes necessary when distance, failure domains, or data-residency constraints exceed what one region can hide. A user in Tokyo expects low-latency responses regardless of where models were trained or where the company headquarters is located; the engineering choice is whether to replicate computation, cache repeated work, or centralize the model and pay the network penalty. The architectural patterns are useful only when tied to that choice.

Why multi-region matters

Single-region deployment creates three fundamental limitations. The most immediate is the latency floor imposed by network round-trip time (RTT) to distant users, quantified in table 28:

Table 28: Network RTT by User Location: A single US-East deployment imposes a 75 to 200 ms RTT on European, Asian, and Australian users, an order of magnitude above the local-region floor. For interactive workloads where each request makes several model calls, this latency floor compounds quickly, motivating multi-region deployment for any globally distributed user base.
User Location RTT to US-East RTT to Local Region
New York 10 ms 10 ms
London 75 ms 10 ms
Tokyo 150 ms 10 ms
Sydney 200 ms 10 ms

For interactive applications (chatbots, autocomplete), these delays compound across multiple model calls per request. Beyond latency, single-region deployment creates a single point of failure: cloud region outages, while rare, affect all users simultaneously. Regulatory constraints add a third dimension, as data residency requirements (the General Data Protection Regulation and data sovereignty laws) may require processing user data within specific geographic boundaries. The resulting patterns form a decision ladder: regional replicas pay the highest deployment cost but minimize RTT and isolate regional failures; edge caching pays operational complexity only when queries repeat; and cross-region sharding is a last resort for capacity shortages because inter-region latency overwhelms per-stage compute.

Pattern 1: Global load balancing with regional replicas

Each region runs independent inference replicas with identical models. A global load balancer routes each user to the nearest region based on measured round-trip latency (figure 22). A shared model registry synchronizes weights across regions during rollouts.

Figure 22: Multi-Region Inference: Global Load Balancing: A global load balancer routes requests to three independent regional deployments (US-East, EU-West, Asia-Pacific) based on latency. Each region operates independently; a shared model registry synchronizes weights. During regional failure, the LB reroutes to the next nearest healthy region.

Independent regional replicas reduce RTT from 150–200 ms to under 10 ms for distant users, while the shared model registry ensures consistency across deployments. Model synchronization is the first architectural concern: updates must propagate to all regions. Push-based synchronization lets a central registry push to every region, which is simple but creates a possible inconsistency window; pull-based synchronization lets regions poll for updates, trading higher rollout latency for stronger consistency; hybrid synchronization combines push notification with pull verification. Version consistency is the second concern. During rollouts, different regions may briefly serve different versions. For most applications this is acceptable, but applications requiring strict consistency need version pinning in request routing.

Pattern 2: Edge caching with central inference

For models too large to replicate globally, caching responses at the edge eliminates redundant inference for repeated queries. Figure 23 shows the two paths: a cache hit returns in under 10 ms from the nearest edge node; a cache miss traverses the full backend and populates the cache on return.

Figure 23: Edge Caching: Cache Hit vs. Cache Miss Paths: A cache hit returns immediately from the CDN edge node (<10 ms). A cache miss flows through to central inference and populates the cache on return. Hit rate determines the effective cost saving; open-ended generative workloads have low repeatability and benefit little from this pattern.

The two paths in figure 23 highlight the order-of-magnitude latency difference: cache hits return in under 10 ms, while cache misses incur the full backend round-trip. Effectiveness depends on request repeatability across workloads such as autocomplete, FAQ chatbots, open-ended chat, and code generation, as table 29 quantifies:

Table 29: Edge Cache Hit Rates by Workload: Cache effectiveness depends almost entirely on request repeatability. Autocomplete and FAQ workloads have high prefix overlap and benefit dramatically; open-ended chat has near-unique inputs and barely benefits. The hit-rate range determines whether edge caching is worth the operational complexity for a given workload.
Workload Cache Hit Rate Suitability
Autocomplete 60–80% Excellent
FAQ chatbot 40-60% Good
Open-ended chat 5-15% Poor
Code generation 20–40% Moderate

Semantic caching16 (caching based on embedding similarity rather than exact match) can improve hit rates for open-ended workloads.

16 [offset=-10mm] Semantic Caching: Returns cached responses for queries that are embedding-similar rather than string-identical, requiring a vector database lookup per request. The trade-off: higher hit rates (potentially 2–3\(\times\) for open-ended workloads) at the cost of added lookup latency (1–5 ms) and the risk of incorrect cache hits when semantically similar queries require different answers. Setting the similarity threshold is a precision-recall trade-off with direct impact on response correctness.

Pattern 3: Cross-region model sharding

Cross-region model sharding is a last-resort capacity strategy rather than a normal latency optimization. For the largest models, pipeline parallelism can theoretically span regions: a router assigns early layers to one region and later layers to another. This pattern is rarely practical because inter-region network latency (75–150 ms) dominates compute time per stage (1–5 ms), making cross-region pipeline parallelism 10–100\(\times\) slower than co-located deployment. It applies only when no single region has sufficient GPU capacity for the full model.

Because cross-region sharding is so costly on the steady-state path, most production designs keep inference co-located and use other regions for resilience. The cross-region question therefore shifts from splitting one request across regions to deciding how quickly traffic can move when a region fails.

Cross-region failover

When a region becomes unavailable, traffic must reroute to healthy regions. Listing 3 shows active-active failover routing logic that falls back to the next healthy region after timeout or unavailability.

Listing 3: Active-Active Failover: Global routing implementation that attempts the primary regional endpoint with a strict timeout before falling back to the secondary healthy region. 
# Simplified global routing logic
def route_request(user_region, request):
    primary = get_nearest_healthy_region(user_region)
    secondary = get_second_nearest_healthy_region(user_region)

    try:
        return call_region(primary, request, timeout=2.0)
    except (Timeout, RegionUnavailable):
        # Failover with increased latency
        return call_region(secondary, request, timeout=5.0)

Stateful LLM serving makes failover harder than ordinary HTTP routing because the receiving region must absorb both traffic and lost session state. Session affinity loss means users mid-conversation lose KV cache state, so the fallback region must regenerate context from conversation history. The receiving region also sees a sudden capacity spike, requiring preprovisioned headroom, typically 30–50 percent over steady state, or accepted degraded latency during failover. When the failed region recovers, gradual recovery shifts traffic back slowly enough to avoid oscillation.

Global model deployment

Deploying model updates across regions requires careful coordination. A phased rollout begins with a canary deployment to a single region (typically 1 percent of traffic), monitors for 1–4 hours, then progressively expands to additional regions. Each expansion point is a gate: if error rates exceed 0.1 percent or P99 latency exceeds the target, the rollout halts. Rollback switches the affected region back to the previous model version while preserving the new version for debugging. The key invariant is that no region proceeds to a new version until the prior region has demonstrated healthy metrics for a sufficient monitoring window.

Global deployment health requires monitoring four metrics both per-region and globally, as table 30 specifies:

Table 30: Global Deployment Health Metrics: Four metrics monitored at both per-region and global granularity during phased rollouts. Per-region thresholds catch localized regressions; global thresholds catch cross-region effects (such as a P99 inflation greater than 2\(\times\) the single-region target) that no per-region check would surface.
Metric Per-Region Global
Error rate \(< 0.1\%\) \(< 0.1\%\)
P99 latency \(< \text{target}\) \(< 2\times\) single-region
Throughput Stable Stable
Model quality Within bounds Consistent across regions

Cost optimization across regions

GPU pricing varies by region. Table 31 shows representative H100 prices so placement can balance cost against latency requirements:

Table 31: H100 Pricing and Latency by Region: Representative spot and on-demand H100 prices alongside RTT to US users. The 20 percent price spread across regions is modest compared to the latency spread, so cost-aware routing typically sends only latency-tolerant workloads (batch inference, background processing) to the cheapest region while keeping interactive traffic local.
Region H100 Spot Price On-Demand Latency to US Users
US-East $2.50/hr $4.00/hr 10–50 ms
US-West $2.30/hr $3.80/hr 30–70 ms
EU-West $2.80/hr $4.20/hr 75–100 ms

Cost-aware routing directs latency-tolerant workloads (batch inference, background processing) to the cheapest available region (listing 4).

Listing 4: Cost-Aware Routing: Priority-based request router that dispatches low-priority batch requests to the lowest-cost region with available capacity while routing latency-sensitive interactive traffic to the nearest region. 
def route_batch_request(request):
    if request.priority == "low":
        # Route to cheapest region with capacity
        return get_cheapest_region_with_capacity()
    else:
        # Route to nearest region
        return get_nearest_region(request.user_location)

Time-of-day routing can reduce costs by 20–40 percent for batch workloads while maintaining SLOs for interactive traffic.

Scaling global infrastructure aggressively is highly effective, but it is also exceptionally expensive. To further bend the cost curve of large deployments, system designers alter the model’s fundamental numerical representation to execute faster and consume less memory through quantization.

Self-Check: Question

  1. Which statement best distinguishes deployment-level bulkheads from request-level bulkheads in a multi-tenant inference platform?

    1. Deployment-level bulkheads enforce rate limits using software counters; request-level bulkheads isolate GPU power supplies.
    2. Deployment-level bulkheads apply only to batch inference; request-level bulkheads apply only to interactive chat.
    3. Deployment-level bulkheads dynamically quantize weights to INT4; request-level bulkheads dynamically compile CUDA graphs.
    4. Deployment-level bulkheads physically isolate tenant tiers onto dedicated GPU replicas, while request-level bulkheads cap input/output token lengths and execution time within shared processes.

  2. The scaling response time budget is defined as \(T_{\text{response}} = T_{\text{detect}} + T_{\text{decide}} + T_{\text{provision}} + T_{\text{warmup}}\). When autoscaling a 70B-parameter LLM replica on fresh cloud instances, which component overwhelmingly dominates the delay, and what strategy mitigates it?

    1. \(T_{\text{provision}}\) dominates (5–7 minutes downloading 140 GB weights from remote storage); mitigated by maintaining pre-warmed pools or predictive scaling.
    2. \(T_{\text{decide}}\) dominates (10–15 minutes evaluating autoscaling PID loops); mitigated by switching to manual scaling.
    3. \(T_{\text{detect}}\) dominates (30 minutes gathering Prometheus metrics); mitigated by removing metric telemetry entirely.
    4. \(T_{\text{warmup}}\) dominates (1–2 hours running CUDA kernel JIT compilation); mitigated by disabling Tensor Cores.

  3. When a cloud provider issues a 2-minute termination warning for a spot/preemptible GPU inference instance, describe the 4-step graceful shutdown sequence that must be executed to prevent data corruption and user-visible failures.

  4. True or False: Cross-region model sharding (spanning pipeline stages between US-East and Asia-Pacific) is an effective strategy for minimizing single-request P99 latency.

  5. When multiple tenants with unequal demand compete for a shared GPU pool, the fair-allocation algorithm that maximizes the minimum allocation by bounding per-tenant consumption using token generation rate (tokens/sec) is called ____ fairness.

See Answers →

Weight Quantization for Serving

Memory ladder comparing a 140 GB FP16 70B model with a 35 GB INT4 70B model, with the 4x capacity reduction marked as a ratio annotation.

INT4 turns a two-GPU model into a one-GPU candidate.

Weight quantization for serving is a bottleneck-selection decision: reducing weights from 16-bit floats to 8-bit or 4-bit integers can turn a 140 GB FP16 language model that requires two A100 GPUs into a 35 GB INT4 representation that fits on a single GPU. The trade-off is a small accuracy loss in exchange for a 4\(\times\) reduction in memory bandwidth and capacity demand.

Systems Perspective 1.3: Quantization as a serving constraint
In serving systems, quantization is not merely a model-compression technique; it changes the binding resource. Post-training quantization can shrink a deployed model without retraining, while quantization-aware training spends training effort to preserve accuracy at lower precision. The systems question is which constraint the lower-precision representation relaxes: GPU memory capacity, HBM bandwidth during decode, accelerator instruction support, or cost per served token. The precision foundations appear in Precision Engineering and Post-training vs. quantization-aware training.

Quantization reduces numerical precision of model weights and activations, decreasing memory footprint by 2–4\(\times\) while increasing decode throughput, which is memory-bandwidth limited rather than compute limited. The serving decision is which quantity should shrink: resident weights for fit, bytes moved per decode step for bandwidth, activations for prefill throughput, or the precision path supported by target hardware. Serving at scale also introduces distinct challenges: models must be quantized after training without access to training data, quality must be preserved across diverse inputs, and hardware deployment targets vary from data center GPUs to edge accelerators. Production inference therefore needs a decision model, not a method roster. The decision model in table 32 keeps the method choice tied to the serving constraint that binds the system.

Table 32: Serving Quantization Decision Model: Quantization methods are useful only when they relax the resource that binds the serving system. The method family follows from the bottleneck: fit, bandwidth, prefill compute, or outlier robustness.
Binding serving constraint Representation change Method family Risk to test before rollout
Model does not fit in memory Quantize weights only GPTQ, AWQ Calibration mismatch and quality regression
Decode is bandwidth-bound Reduce bytes read per token W4A16 weight-only paths Kernel support and KV-cache headroom
Prefill is compute-bound Quantize weights and activations SmoothQuant W8A8 Activation outliers and INT8 hardware path
Outliers block low precision Change the coordinate basis before quantization Rotation-based methods Transform overhead and limited runtime path

LLM-specific quantization challenges

Large language models present unique quantization challenges distinct from vision or recommendation models. The outlier activation problem occurs because certain attention heads produce activation magnitudes orders of magnitude larger than typical values. Naive quantization clips these outliers, causing significant quality degradation.

Consider a large language-model layer where most activation values are modest but specific channels produce much larger outliers (Xiao et al. 2023). Symmetric INT8 quantization with range [-127, 127] has no painless scale choice: a wide range preserves outliers but collapses typical values into too few bins, while a narrow range preserves resolution for typical values but clips outliers and introduces large errors.

The outlier distribution explains why the specialized methods that follow differ: each one protects a different part of the serving contract.

GPTQ: Layer-by-layer weight quantization

Generalized post-training quantization (GPTQ) (Frantar et al. 2023) is the weight-only choice when deployment needs 4-bit compression after training and can afford calibration data. Its serving value is that the model gets smaller without a full retraining run. The risk is that rounding one weight changes the layer output in a way later weights must absorb. GPTQ addresses that risk by using calibration activations to estimate which weight errors matter most,17 then compensating the remaining unquantized weights as each column is rounded.

Frantar, Elias, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. “GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers.” In CoRR, abs/2210.17323. https://doi.org/10.48550/ARXIV.2210.17323.

17 Hessian Matrix in Quantization: The Hessian \(\mathbf{H} = \mathbf{X}^T \mathbf{X}\) captures second-order sensitivity: weights with large diagonal entries have outsized impact on model outputs. GPTQ exploits this to quantize insensitive weights aggressively while preserving sensitive ones, achieving 3–4-bit quantization with less than 1 percent perplexity degradation. Without Hessian guidance, naive uniform quantization below 8-bit typically destroys model quality.

18 GPTQ (Generalized Post-Training Quantization) Column-Wise Update: GPTQ factors the inverse Hessian \(H^{-1}\) via Cholesky decomposition so the compensation becomes a stable triangular update: after rounding column \(q\), the residual error is propagated to columns \(q{+}1{:}\) weighted by \([H^{-1}][:,q{+}1{:}] / [H^{-1}]_{qq}\). The factorization avoids recomputing a full inverse per column, giving \(\mathcal{O}(d_{\text{row}} \cdot d_{\text{col}}^2)\) work instead of the naive \(\mathcal{O}(d_{\text{row}} \cdot d_{\text{col}}^3)\). Per-group scaling (group size 128) gives outlier channels a finer step than a single per-matrix scale.

GPTQ processes the model one layer at a time, quantizing each weight matrix column by column18 and compensating the still-unquantized columns so the layer’s output drifts as little as possible. The serving consequence is a two-part deploy-time cost: a calibration pass over 128–256 representative samples that must match production traffic, and a per-layer second-order update that makes quantization minutes-to-hours of offline work rather than a one-shot cast. Both are paid once before deployment, so they do not touch serving latency, but a calibration set that misrepresents the deployed traffic mix is the failure mode to test for before rollout.

Table 33 summarizes GPTQ performance across Llama model sizes:

Table 33: GPTQ Performance Across Llama Sizes: Quantization quality holds remarkably steady as model size scales: the 70B model loses less perplexity than the 7B model despite the same 4-bit target. Memory reduction is uniformly 4\(\times\). Quantization time scales near-linearly with model size, dominated by the per-layer second-order updates.
Model Bits Perplexity Increase Memory Reduction Quantization Time
Llama-7B 4 +0.3 4\(\times\) 15 min
Llama-13B 4 +0.2 4\(\times\) 30 min
Llama-70B 4 +0.15 4\(\times\) 3 hours

GPTQ’s strengths include fast quantization without retraining, minimal quality loss for 4-bit weights, and broad hardware compatibility. Its limitations include requiring calibration data, sensitivity to calibration set selection, and per-layer processing that cannot use cross-layer information.

AWQ: Activation-aware weight quantization

Activation-aware weight quantization (AWQ) (Lin et al. 2024) attacks the same weight-only serving goal from the activation side. A calibration pass measures per-channel activation magnitudes, and the weights feeding the few high-magnitude channels are scaled up before quantization (with a matching scale folded into the next layer), so those salient weights keep their resolution while the rest are quantized aggressively. Its serving advantage over GPTQ is that protecting salient channels avoids the per-column error-feedback pass, so calibration is lighter and data-free of any reference output; the risk to test is that the salient-channel scaling must fuse into the deployed graph without a separate runtime op, or the memory win is eaten by an extra kernel.

Lin, Ji, Jiaming Tang, Haotian Tang, Shang Yang, Guangxuan Xiao, and Song Han. 2024. “AWQ: Activation-Aware Weight Quantization for on-Device LLM Compression and Acceleration.” GetMobile: Mobile Computing and Communications 28 (4): 12–17. https://doi.org/10.1145/3714983.3714987.

Table 34 summarizes how AWQ compares to GPTQ across error compensation, calibration cost, quality, speed, and hardware compatibility.

Table 34: AWQ vs. GPTQ: AWQ scales salient channels rather than compensating residual error, achieves higher 4-bit quality at marginally slower calibration speed, and matches GPTQ on hardware compatibility.
Aspect GPTQ AWQ
Error compensation Adjusts remaining weights Scales salient channels
Calibration data 128–256 samples 128 samples
Quality (4-bit) High Excellent
Speed Faster Slightly slower
Hardware compatibility Broad Broad

AWQ can achieve 0.5-1 percent lower perplexity degradation than GPTQ at the same bit-width in the representative setting here, making it a plausible choice when the quality budget is tight.

SmoothQuant: Migrating quantization difficulty

SmoothQuant (Xiao et al. 2023) is the serving path for W8A8 deployments where activation quantization matters, especially compute-bound prefill. Activations carry unpredictable outliers that defeat INT8; weights do not. SmoothQuant divides activations by a per-channel scale and multiplies the corresponding weights by the same scale, an algebraically identical transform that leaves the layer output unchanged while moving the hard-to-quantize range out of the activations and into the more forgiving weights. Its serving consequence is that the scales fold into the existing weights and a cheap activation division, so the smoothed model runs on a standard INT8 path at near-zero runtime overhead; the risk to test is that the migration only shifts outliers rather than removing them, so a calibration set whose activation extremes differ from production can still clip.

Xiao, Guangxuan, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. “SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.” International Conference on Machine Learning (ICML), 38087–99.

In the W8A8 deployment regime, SmoothQuant enables INT8 quantization for both weights and activations. Table 35 shows how W8A8 compares to FP16 baseline and weight-only quantization:

Table 35: W8A8 SmoothQuant vs. Weight-Only Quantization: W8A8 quantizes both weights and activations to INT8, doubling memory savings and unlocking compute-bound prefill speedup, but its decode speedup is lower than W8A16’s because decode is memory-bound and W8A8’s smaller weights are already captured by the simpler weight-only scheme.
Configuration Memory Prefill Speedup Decode Speedup Quality
FP16 (baseline) 1\(\times\) 1\(\times\) 1\(\times\) Baseline
W8A16 (weights only) 2\(\times\) 1.3\(\times\) 1.8\(\times\) \(<0.5\%\) loss
W8A8 (SmoothQuant) 2\(\times\) 1.8–2\(\times\) 1.3–1.5\(\times\) \(<1\%\) loss

The critical distinction is that W8A8 provides near 2\(\times\) speedup for compute-bound prefill (large batch processing initial prompt) but only 1.3–1.5\(\times\) speedup for memory-bound decode (generating tokens one at a time). LLM serving is typically decode-heavy, so real-world throughput improvements from W8A8 are often 1.3–1.7\(\times\) rather than the theoretical 2\(\times\) compute throughput of INT8 Tensor Cores.

Rotation-based quantization

Traditional quantization methods (GPTQ, AWQ, SmoothQuant) address outliers through compensation or migration. Rotation-Based Quantization (Ashkboos et al. 2024) is the aggressive path when activation outliers block lower precision: it mathematically transforms the weight and activation space to eliminate outliers entirely.

Ashkboos, Saleh, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. “QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs.” Advances in Neural Information Processing Systems (NeurIPS), 100213–40. https://doi.org/10.52202/079017-3180.

The foundational insight is that outliers are artifacts of the coordinate basis representation. Rotating to a different basis spreads extreme values uniformly, making all values quantization-friendly.

QuaRot (quantization with rotation)

QuaRot rotates the weight and activation spaces with an orthogonal Hadamard transform19 so that any single outlier is spread evenly across all dimensions, leaving a distribution with no dominant value that uniform low-bit quantization can capture. Because the rotation is orthogonal, the layer output is unchanged and the transform needs no calibration data, which is what lets QuaRot reach W4A4 where the compensation and migration methods stop at higher precision. The serving consequence is a runtime cost the other methods do not pay: the rotation runs inline on every forward pass, adding roughly 3 percent overhead, so the test before rollout is whether the W4A4 memory-and-bandwidth win clears that standing tax on the target hardware.

19 Hadamard Rotation in QuaRot (Quantized Rotation): QuaRot transforms activations as \(X' = X \cdot H\) and weights as \(\mathbf{W}' = \mathbf{H}^T \cdot \mathbf{W}\), where \(H\) is an orthogonal Hadamard matrix, computed structurally without being stored. Orthogonality preserves \(\mathbf{X} \mathbf{W} = \mathbf{X}' \mathbf{W}'\), so the layer output is unchanged. The outlier-spreading effect is concrete: a vector \([1000, 1, 1, 1]\) with one extreme value becomes roughly \([502, 500, 500, 500]\) after the transform, so no single coordinate dominates the quantization range.

Table 36 contrasts QuaRot’s rotation-based approach with SmoothQuant’s migration. The operating trade-off is calibration and runtime cost: SmoothQuant avoids runtime overhead after calibration but stops at W8A8, while QuaRot reaches W4A4 by paying an inline rotation cost.

Table 36: QuaRot vs. SmoothQuant: SmoothQuant migrates outliers from activations into weights and runs at zero runtime overhead but requires calibration data and stops at W8A8. QuaRot eliminates outliers through Hadamard rotation, runs data-free, reaches W4A4, and pays a ~3 percent runtime cost for the inline transforms.
Aspect SmoothQuant QuaRot
Calibration data Required Not required (data-free)
Minimum precision W8A8 W4A4
Runtime overhead ~0% ~3% (Hadamard transforms)
Outlier handling Migration Elimination

Table 37 compares perplexity across quantization methods and precision choices on Llama 2 70B:

Table 37: Quantization Perplexity on Llama 2 70B: Perplexity degradation across four quantization methods on a 70B model. QuaRot’s W4A4 lands within 0.2 perplexity of FP16 baseline while delivering roughly 4\(\times\) memory reduction for quantized tensors, demonstrating that rotation-based methods make 4-bit activation quantization viable for production inference.
Method Precision Llama 2 70B Perplexity vs. Baseline
FP16 W16A16 3.12 Baseline
SmoothQuant W8A8 3.18 +0.06
GPTQ W4A16 3.24 +0.12
QuaRot W4A4 3.31 +0.19

QuaRot achieves 4-bit weights and 4-bit activations with quality competitive to GPTQ’s 4-bit weights only, enabling approximately 4\(\times\) memory reduction vs. FP16 for quantized weights and activations. SpinQuant extends QuaRot by learning optimal rotation matrices during a short fine-tuning phase, improving quality at the cost of training compute. Rotation methods therefore extend low-precision reach, but whether the trade-off pays depends on the deployment hardware’s support for the target precision.

Hardware-deployment co-design

The quantization method only pays off when the deployment hardware accelerates the chosen precision. Different accelerators support formats such as BF16, INT8, and INT4 with varying performance multipliers.

Table 38 summarizes NVIDIA Tensor Core precision support across Ampere and Hopper:

Table 38: Tensor Core Precision Support and Speedup: Native precision support across NVIDIA Ampere and Hopper Tensor Cores. The 2\(\times\) speedup at INT8 and FP8 reflects the doubled throughput of those native paths. A100 provides native INT4 Tensor Core operations, although realized LLM speedup still depends on framework and kernel support; H100 INT4 deployment remains software/framework-dependent, and native FP4 first appears in Blackwell.
Format Ampere (A100) Hopper (H100) Speedup vs. FP16
FP16 Yes Yes 1\(\times\)
BF16 Yes Yes 1\(\times\)
INT8 Yes Yes 2\(\times\)
FP8 (E4M3) No Yes 2\(\times\)
INT4 Yes (native Tensor Cores) Software/framework-dependent on H100; native FP4 is a Blackwell feature workload-dependent

Memory bandwidth dominates autoregressive LLM decode throughput because each token reads the entire model:

\[\text{Decode throughput} \propto \frac{\text{Memory bandwidth}}{\text{Model size in bytes}}\]

In the ideal memory-bound case, reducing FP16 weights to 4-bit values cuts weight traffic by 4\(\times\), but realized decode throughput depends on kernel support, dequantization overhead, KV-cache precision, batch shape, and whether HBM bandwidth is truly the bottleneck. The value of 4-bit serving is therefore strongest when it increases model residency or batch capacity, and throughput gains should be validated on the target serving workload.

Table 39 shows that the configuration follows two things the deployment fixes in advance: the precision path the target hardware supports and whether the goal is memory residency or throughput. Memory-constrained consumer GPUs take W4A16, INT8 accelerators take W8A8 for Tensor Core throughput, and H100-class hardware takes its native FP8 path.

Table 39: Quantization Deployment Configurations: Mapping from quantization scheme to deployment target and supporting frameworks. W4A16 is common for consumer and memory-constrained deployments; W8A8 unlocks INT8 Tensor Core throughput; FP8 is the native low-precision path on H100/H200-class deployments; W4A4 is an aggressive compression target with more limited framework support.
Quantization Best For Framework Support
W4A16 (GPTQ/AWQ) Consumer GPUs, memory-constrained vLLM, TensorRT-LLM, llama.cpp
W8A8 (SmoothQuant) INT8 accelerators, high throughput TensorRT-LLM, ONNX Runtime
FP8 H100/H200 deployments TensorRT-LLM
W4A4 Research, extreme compression Limited

Runtime contract for quantized serving

Production serving frameworks matter because quantized weights reduce cost only when the runtime preserves the theoretical savings. The durable contract is framework-independent: the serving stack must load the quantized artifact without silently dequantizing it, choose kernels that execute the intended low-precision path, allocate KV cache around the smaller weight footprint, and expose enough telemetry to detect quality or latency regressions. Table 40 lists these responsibilities alongside the way each one silently leaks the theoretical saving when the runtime fails it, which is why all four together form the minimum contract a framework must satisfy before quantization becomes a production-serving win.

Table 40: Quantized Serving Runtime Contract: Framework-specific APIs differ, but every production path has to preserve representation, choose supported kernels, plan memory, and validate quality under the deployed traffic mix.
Runtime responsibility Why it matters
Preserve the quantized artifact Silent dequantization restores the FP16 memory and bandwidth cost.
Select compatible low-precision kernels Unsupported operators fall back to slower or higher-precision execution.
Plan memory around weights and KV cache The capacity gain matters only if the freed memory becomes useful batch headroom.
Validate quality and latency together A lower-bit path that meets latency but fails task quality is not a serving win.

Quantized models combine with PagedAttention for maximum memory efficiency:

\[\text{Max batch} = \frac{\text{GPU Memory} - \text{Quantized Weights}}{\text{KV Cache per Sequence}}\]

A 70B model with 4-bit weights requires approximately 35 GB, leaving 45 GB on an 80 GB A100 for KV cache. With FP16 KV cache at about 2.6 GB per 1K-token sequence for the MHA baseline, this supports about 17 concurrent 1K-token sequences before other runtime overheads. With FP16 weights, the same 70B model would not fit on one A100 at all, so weight-only quantization changes both feasibility and batch capacity.

Quantization selection guidelines

The final selection matches the method to the binding constraint. Latency-sensitive paths should prefer FP16 or BF16 when quantization overhead would dominate, while throughput-oriented paths are more likely to benefit from quantization. Hardware support narrows the viable choices: H100 and H200 deployments can consider FP8 because the hardware provides native support with minimal quality loss; A100 and A10G deployments usually choose between W8A8 and W4A16 depending on the workload; consumer GPUs often require W4A16 to fit the model. The quality budget then sets the lower precision bound. If less than 0.5 percent degradation is acceptable, AWQ 4-bit is a plausible target; if less than 1 percent degradation is acceptable, GPTQ 4-bit or SmoothQuant W8A8 may be viable; if no degradation is acceptable, FP16 or BF16 remains the safest choice. The binding resource makes the final distinction: compute-bound prefill favors W8A8 because it can provide a 2\(\times\) speedup, whereas memory-bound decode favors W4A16 because it can provide a 4\(\times\) effective bandwidth increase.

Table 41 shows how quantization reshapes serving cost per million tokens:

Table 41: Quantization Impact on Serving Cost: Estimated cost per million tokens at three configurations. Halving the GPU count via 4-bit quantization halves the cost; halving it again brings the per-million-token cost to one quarter of the FP16 baseline. The cost savings come almost entirely from fewer GPUs rather than from per-GPU throughput gains.
Configuration Cost per 1M tokens (estimated)
FP16 on 8\(\times\) A100 $2.40
AWQ 4-bit on 4\(\times\) A100 $1.20
AWQ 4-bit on 2\(\times\) A100 $0.60

Quantization can reduce serving costs by 2–4\(\times\) while maintaining acceptable quality, making it an important lever for cost-effective LLM deployment.

These serving optimizations span continuous batching, PagedAttention, global load balancing, and extreme quantization. The case studies that follow show how production systems at global scale combine these techniques to meet specific cost and latency targets.

Self-Check: Question

  1. How does SmoothQuant enable efficient 8-bit weight and 8-bit activation (W8A8) inference, and why is its speedup significantly higher during prefill than during decode?

    1. It prunes 50% of model weights using structured sparsity, accelerating decode but having no effect on prefill.
    2. It multiplies activations by a per-channel smoothing scale \(s\) and divides weights by \(s\) to migrate outlier difficulty from activations to weights; prefill benefits from native INT8 Tensor Core compute (near \(2\times\)), while decode is memory-bandwidth bound (\(1.3\text{--}1.5\times\)).
    3. It rotates the coordinate basis using randomized Walsh-Hadamard matrices, eliminating memory bandwidth bottlenecks during decode.
    4. It converts floating-point weights into binary spikes, enabling single-cycle bitwise operations during token generation.

  2. What is the core algorithmic distinction between Generalized Post-Training Quantization (GPTQ) and Activation-Aware Weight Quantization (AWQ)?

    1. GPTQ quantizes activations while leaving weights in FP16; AWQ quantizes weights while leaving activations in FP32.
    2. GPTQ requires full model retraining for 100 epochs; AWQ is an online dynamic quantization method with no calibration.
    3. GPTQ compensates residual quantization error across remaining unquantized columns using inverse Hessian information (\(H^{-1}\)); AWQ protects salient weight channels based on activation magnitudes without error feedback.
    4. GPTQ applies only to convolutional vision models; AWQ applies exclusively to sparse recommendation embedding tables.

  3. Explain how rotation-based quantization (QuaRot) eliminates activation outliers in a data-free manner to unlock 4-bit weights and 4-bit activations (W4A4), and identify the runtime cost it incurs.

  4. True or False: Modern NVIDIA Hopper (H100) GPUs provide native Tensor Core instruction acceleration for FP8 (E4M3/E5M2) delivering a \(2\times\) compute throughput multiplier over FP16, whereas Ampere (A100) Tensor Cores do not natively support FP8 execution.

  5. Order the sequential steps in the GPTQ column-wise layer quantization pipeline:

  1. Compute the Hessian matrix \(H = 2 X^T X\) from calibration activations
  2. Perform Cholesky decomposition on the inverse Hessian matrix \(H^{-1}\)
  3. Round the current column weights to the nearest discrete quantized grid
  4. Calculate the residual quantization error vector for the column
  5. Propagate compensation updates to all remaining unquantized columns using the Cholesky inverse weights

See Answers →

Case Studies

Production-scale serving systems force the chapter’s techniques to meet real workload variance, latency budgets, and operational constraints. The cases bind on different constraints: embedding scale, variable-length generation, cascade cost, or multimodal freshness. The examples show how systems combine batching, sharding, caching, routing, and quantization to meet specific cost and latency targets. The Orca and vLLM mechanisms developed earlier in this chapter provide the LLM-serving primitives; the case studies in this section broaden the view to recommendation serving, global request routing, ranking cascades, and multimodal freshness.

Meta recommendation serving

Meta’s recommendation infrastructure binds on embedding scale rather than dense forward-pass compute. It serves predictions for feeds, ads, and content ranking across Facebook, Instagram, WhatsApp, and Messenger, making it one of the largest production inference deployments in the world.

The scale numbers explain why embedding locality dominates the design:

  • Request volume: Billions of requests per day
  • Latency target: \(<10\text{ ms}\) P99
  • Model diversity: Hundreds of model variants
  • Feature cardinality: Trillions of unique entities

The serving stack separates sparse lookup from dense ranking:

User request → Feature collection → Embedding lookup → Model inference → Response

                    |                     |                  |
                    v                     v                  v
             Feature Store        Embedding Servers      GPU Inference
             (CPU, DRAM)         (CPU + SSD, 1000s)     (GPU, 100s)

In Meta’s architecture, the binding design constraint is embedding scale: tables total over 100 TB, requiring 1,000+ shards. Meta uses a hybrid sharding strategy:

  • Hot embeddings (top 1 percent): Replicated across memory on all inference servers
  • Warm embeddings (next 10 percent): Column-sharded with 8-way parallelism
  • Cold embeddings (remaining 89 percent): Row-sharded with consistent hashing, SSD-backed

Hybrid sharding reduces embedding lookup latency from 50 ms (naive) to 2 ms through batching and locality optimization.

Instead of batching entire requests, Meta batches at the feature level. Each inference request triggers 5,000+ embedding lookups, but these lookups are batched across requests within a 1 ms window. This achieves 90 percent+ memory bandwidth utilization on embedding servers.

The resulting GPU-CPU hybrid architecture runs dense model computation (ranking towers) on GPUs, while sparse embedding lookups run on CPU servers with large memory and SSD storage. Table 42 places each component on the hardware whose strength matches its bottleneck: the memory-bound sparse lookups land on SSD-backed CPU servers, the low-intensity feature processing stays on CPU, and only the compute-dense ranking pass uses the GPU.

Table 42: Meta DLRM Hybrid CPU/GPU Architecture: Hardware-workload mapping for Meta’s recommendation serving stack. CPU + SSD handles the bandwidth-bound sparse path; CPU handles low-arithmetic-intensity feature processing; GPU handles the compute-dense ranking forward pass. The three latency budgets sum to ~5 ms, well within typical recommendation SLOs.
Component Hardware Latency Throughput
Embedding lookup CPU + SSD 2 ms 50M lookups/s
Feature processing CPU 1 ms 10M ops/s
Dense ranking GPU 1.5 ms 100K infs/s

The serving lesson is that recommendation latency is often dominated by embedding lookup rather than dense model inference. Feature-parallel batching and a hybrid CPU-GPU architecture match the hardware to the sparse and dense halves of the workload, which sets up a contrast with GPT-style API serving where request variance and autoregressive state dominate.

OpenAI API infrastructure

OpenAI-style API infrastructure binds on request variance: short prompts, long-context requests, and different model sizes share capacity while users expect predictable TTFT. The infrastructure serves GPT-class models to large developer and application workloads while maintaining quality of service across diverse workloads.

The scale numbers explain why continuous batching and admission control dominate:

  • Request volume: Millions of requests per hour
  • Latency target: TTFT \(<2\text{s}\), throughput varies by model
  • Model sizes: Billions to hundreds of billions of parameters
  • Context lengths: Up to 128K tokens

The serving stack separates routing, scheduling, cache management, and shard groups:

API Gateway → Rate Limiting → Request Router → Model Cluster → Response Streaming

                                     |
                                     v
                              +-------------+
                              | Model Pool  |
                              | +---------+ |
                              | | Large   | |
                              | | 8×H100  | |
                              | +---------+ |
                              | +---------+ |
                              | | Smaller | |
                              | | 4×A100  | |
                              | +---------+ |
                              +-------------+

The main design decision is to prevent long prompts from blocking decode service for everyone else.

An Orca-style long-context serving design uses continuous batching to maintain high GPU utilization despite variable output lengths. Chunked prefill bounds decode latency by processing long prompts in chunks that interleave with ongoing generation. Table 43 contrasts the three approaches:

Table 43: Batching Strategy Comparison: Three batching approaches with the resulting GPU utilization and 128K-prompt behavior. Static batching lets the longest request idle the whole batch; continuous batching lifts utilization to 75 percent but still stalls decode behind long prefills; chunked prefill interleaves prefill chunks with decode, capping decode stalls at a few seconds while reaching 85 percent utilization.
Batching Strategy GPU Utilization 128K Prompt Behavior
Static batching 45% 30s TTFT; decode blocked
Continuous batching 75% 30s TTFT; decode still blocked by prefill
Continuous + chunked 85% 30s TTFT; decode stalls bounded to ~3s

Large GPT-class models often require 8-way or greater tensor parallelism for memory capacity and latency:

  • 8\(\times\) H100 per large-model shard group
  • NVLink for intra-node communication
  • Consistent hashing for session affinity (KV cache reuse)

OpenAI implements rate limiting at multiple levels to prevent noisy neighbors:

  • Per-API-key request rate limits
  • Per-API-key token-per-minute limits
  • Organization-level capacity quotas
  • Global model capacity limits

During peak demand, OpenAI shifts capacity between models based on queue depth:

if large_model_queue_depth > threshold:
    # Migrate some smaller-model capacity to the larger model
    reallocate_cluster_capacity(from="smaller-model", to="large-model", fraction=0.2)

LLM APIs are governed by variance control. Continuous batching, prefix caching, and multi-tier rate limiting keep long prompts, repeated conversational context, and traffic spikes from turning one user’s request into everyone else’s latency. Search ranking faces a different shape of the same problem. Many cheaper models must cooperate under one strict deadline.

Google search ranking

Google Search binds on cascade cost: each query coordinates many smaller models under one end-to-end latency budget rather than serving one large model. The ensemble combines specialized models for query understanding, document relevance, and result ranking.

The scale numbers explain why a cascade is mandatory:

  • Request volume: Billions of searches per day
  • Latency target: \(<200\text{ ms}\) end-to-end
  • Model count: Dozens of models per query
  • Result processing: Thousands of documents per query

To meet this latency budget while evaluating thousands of documents, the system employs a ranking cascade (figure 24) that progressively narrows the candidate set through increasingly expensive models.

Figure 24: Ranking Cascade Architecture: To optimize latency and cost, search and recommendation systems use a cascade of increasingly complex models. Early stages (Retrieval) filter millions of candidates down to thousands using fast, cheap heuristics. Later stages use expensive, high-precision models only on the most promising candidates.

The central design decision is progressive refinement: rather than running one expensive model on all candidates, Google uses a ranking cascade. Table 44 shows the trade the cascade makes: each stage spends a larger latency budget on a smaller candidate set, so the expensive L3 ensemble is affordable only because L0 through L2 have already cut the candidates by four orders of magnitude.

Table 44: Google Ranking Cascade: Four-stage cascade that filters a million candidates down to ten while spending compute proportional to depth. Each stage uses a more expensive model on fewer candidates, so total cost is dominated by the cheap early stages rather than the expensive final ensemble.
Stage Model Complexity Candidates Latency Budget
L0 (Retrieval) Embedding lookup 1,000,000 → 10,000 10 ms
L1 (First pass) Linear model 10,000 → 1,000 20 ms
L2 (Second pass) Small transformer 1,000 → 100 50 ms
L3 (Final rank) Large ensemble 100 → 10 100 ms

The cascade reduces final L3 evaluations by 10,000× compared to running L3 on all candidates. Under this simplified request-budget model, the L3 stage costs 1 ms per candidate. Applying L3 to the full million-candidate set would therefore consume about 1000 s of model work, while the cascade budget sums to 180 ms. The resulting model-work reduction is roughly 5,600×.

Given tight latency budgets, Google uses speculative execution for model ensembles. Here, speculation means launching candidate ranking work in parallel under a deadline, not draft-token verification for LLM decoding:

# Instead of sequential:
#   q1 = model1(query)
#   q2 = model2(query)
#   q3 = model3(query, q1, q2)

# Speculative parallel:
async_q1 = async model1(query)
async_q2 = async model2(query)
async_q3 = async model3(query, predicted_q1, predicted_q2)

# Use actual results if they arrive in time, otherwise use speculative

Search ranking deployments can run ranking models on Tensor Processing Units (TPUs)20 optimized for transformer inference. TPU pods provide:

20 [offset=-10mm] TPU (Tensor Processing Unit) for Inference: TPU systolic arrays provide deterministic, low-variance matrix execution for ranking-style inference. GPU latency can vary with batch composition, while TPU-style execution can deliver consistent per-request timing for strict search SLOs where P99 latency directly affects revenue.

  • 2D mesh topology for efficient AllReduce
  • High memory bandwidth for attention operations
  • Custom quantization for serving efficiency

Each sub-request carries a deadline, and workers prioritize by deadline proximity:

Worker queue: [Doc1: 50 ms left] [Doc2: 30 ms left] [Doc3: 80 ms left]
                                      ↑ Process first

If deadline will be missed:
  Return cached/default result rather than timing out

The serving lesson is that ranking cascades reduce cost by spending expensive models only on a shrinking candidate set, while deadline propagation and priority scheduling keep the ensemble inside a strict latency budget. Custom hardware can improve predictability when the workload matches the accelerator, but the next case adds a freshness constraint that hardware alone cannot solve.

TikTok multimodal recommendation

TikTok’s recommendation system binds on freshness under a tight ranking SLO: video understanding is expensive, new content arrives continuously, and user modeling remains online. It combines video understanding (vision) with user modeling (recommendation) for personalized content ranking, creating a multimodal inference challenge where different model types must coordinate.

The scale numbers explain why online/offline separation matters:

  • Request volume: Millions of video rankings per second
  • Latency target: \(<50\text{ ms}\) P99
  • Content volume: Millions of new videos daily
  • Modalities: Video, audio, text, user signals

The serving stack separates content and user paths:

User request → User embedding → Candidate videos → Video understanding → Ranking

                    |                  |                    |
                    v                  v                    v
             User Tower          Video Cache          Vision Models
           (Transformer)        (Pre-computed)        (On-demand)

TikTok’s architecture separates user understanding (online) from content understanding (offline) through a two-tower architecture with caching.21 Table 45 contrasts the online and offline tower update cadences:

21 Two-Tower Architecture: Encodes user features and item features through separate neural networks (“towers”) into embeddings, then scores via dot product. The serving advantage: item embeddings can be precomputed offline and cached, reducing online inference to the user tower forward pass plus a sub-millisecond vector similarity lookup. The trade-off is reduced model expressiveness from late interaction, since the towers cannot attend to each other’s features during encoding.

Table 45: TikTok Two-Tower Update Cadence: User and video towers run on different cadences: user features change second-to-second and must run inference online; video features change much more slowly and are precomputed in hourly batches. This asymmetric scheduling is the design that takes vision inference out of the critical path for almost every request.
Tower Update Frequency Latency Compute
User tower Real-time 5 ms GPU (online)
Video tower Hourly N/A GPU (batch)

Video embeddings are precomputed and cached, eliminating vision inference from the critical path for most requests. Only new videos (uploaded within the hour) require online vision inference.

Like Meta, TikTok uses CPU for embedding operations and GPU for dense model computation:

User features → CPU preprocessing (1 ms)
             → Embedding lookup (2 ms, CPU+DRAM)
             → Dense ranking (10 ms, GPU)
             → Response formatting (1 ms)

New video content is processed with different priorities, as table 46 shows:

Table 46: TikTok Video Ingestion Priority Tiers: Three-tier SLA for video ingestion. High-follower-count creators get a five-minute SLA to maximize content freshness for their audience; standard uploads get 30 minutes; bulk imports get two hours of slack so the system can absorb them during quiet periods.
Priority SLA Use Case
Critical 5 min Creator with large following
Standard 30 min Normal uploads
Background 2 hours Bulk/imported content

The priority scheme ensures popular creators’ content reaches recommendations quickly while managing compute costs.

TikTok combines multiple understanding modalities through late fusion, which combines independently computed modality embeddings or scores near the end of ranking rather than running one joint model over all raw modalities:

Video embedding (512d) -+
Audio embedding (256d) -+- Concat → Fusion MLP → Final embedding (256d)
Text embedding (256d)  -+

Late fusion allows independent updates to each modality’s model without retraining the full system.

The serving lesson is that freshness requires separating online and offline work. Two-tower architectures make aggressive caching possible because item embeddings can be refreshed asynchronously, while priority-based processing spends the freshest compute where it has the largest user impact.

Cross-cutting observations

Across these cases, the common pattern is specialization instead of a universal serving stack. Systems separate embedding and retrieval from ranking and generation, use hybrid CPU-GPU architectures to match hardware to workload characteristics, cache at multiple levels while accepting staleness, progressively refine candidate sets to avoid expensive work on unlikely items, and propagate deadlines so quality/latency trade-offs stay explicit under pressure. Table 47 identifies the primary technique and key innovation from each system. The four differ precisely because each is specialized to its own workload; that specialization is the shared pattern, not a contradiction of it.

Table 47: Case Study Summary: Synthesis of primary serving techniques and key architectural innovations across Meta, Google, and TikTok production workloads.
System Primary Technique Key Innovation
Meta Embedding sharding Feature-parallel batching
OpenAI Continuous batching Chunked prefill
Google Ranking cascade Speculative execution
TikTok Two-tower caching Multimodal fusion

The case studies show why serving architectures must be adapted to their workloads. Applying conventional web-service assumptions to GPU-bound systems creates the fallacies and pitfalls examined next.

Self-Check: Question

  1. In Google Search ranking, how does a four-stage ranking cascade (L0 Retrieval: 1M \(\to\) 10k items in 10 ms; L1 First-pass: 10k \(\to\) 1k in 20 ms; L2 Second-pass: 1k \(\to\) 100 in 50 ms; L3 Final rank: 100 \(\to\) 10 in 100 ms) reduce model evaluation work compared to scoring all candidates with the L3 ensemble?

    1. It runs L3 on all 1,000,000 candidates using asynchronous DMA transfers without CPU involvement.
    2. It replaces neural networks entirely with BM25 keyword matching across all stages.
    3. It dynamically recompiles transformer weights at each stage using TensorRT-LLM.
    4. It cuts final expensive L3 evaluations by \(10{,}000\times\) (from 1,000,000 to 100), reducing total candidate-scoring compute by \(\approx 5{,}600\times\) within a 180 ms budget.

  2. Compare the primary binding bottlenecks and serving topologies of Meta’s recommendation infrastructure (DLRM) versus OpenAI’s GPT API infrastructure.

  3. True or False: In TikTok’s two-tower multimodal recommendation architecture, both the user tower and video content understanding tower must execute synchronously in real-time on GPU clusters for every incoming video swipe.

  4. Which cross-cutting architectural principle is demonstrated by comparing production serving architectures across Meta, OpenAI, Google, and TikTok?

    1. Production inference rejects one-size-fits-all stacks: systems specialize hardware and batching to match their binding bottleneck (embeddings, variable decode, cascade cost, or asynchronous freshness).
    2. All modern inference workloads are compute-bound and converge onto identical 8-way tensor-parallel GPU clusters.
    3. Traditional static batching with First-Come-First-Served scheduling remains optimal for all production workloads when scaled to 1,000 GPUs.
    4. Quantization to 1-bit binary representations is universally deployed across all production recommendation and search systems.

See Answers →

Fallacies and Pitfalls

A DevOps team applies their standard web-server load balancing rules to a new fleet of GPU inference nodes, only to watch tail latency explode because they ignored the extreme variance of autoregressive token generation. Inference at scale breaks the rules of traditional microservices, creating dangerous fallacies for engineers transitioning from standard web development to machine learning systems.

Fallacy: Inference at scale is synonymous with LLM serving.

The misconception, reinforced by LLM-centric discourse, leads to over-focus on LLM-specific techniques while ignoring the broader inference landscape. Production data shows that recommendation models generate the bulk of daily query traffic and capacity demand across hyperscale fleets, while vision, fraud detection, and classification models account for the rest (Gupta et al. 2020; Hazelwood et al. 2018). A practitioner who only understands continuous batching and KV cache management is unprepared for the feature-parallel batching and embedding sharding that dominate high-volume recommendation inference. Technique selection must match the actual workload.

Hazelwood, Kim, Sarah Bird, David Brooks, Soumith Chintala, Utku Diril, Dmytro Dzhulgakov, Mohamed Fawzy, et al. 2018. “Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective.” 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), 620–29. https://doi.org/10.1109/hpca.2018.00059.

Pitfall: Using training infrastructure for production serving.

Training and serving have different requirements. Training optimizes for aggregate throughput over hours or days; serving optimizes for per-request latency under strict SLOs. Training tolerates batch sizes of thousands; serving often requires batch sizes in single digits. Training accepts checkpoint-based recovery; serving requires graceful failover without user impact. Teams that deploy training clusters for serving often discover unacceptable latency variance, poor resource utilization, and difficulty meeting SLOs. Purpose-built serving infrastructure with appropriate batching, load balancing, and autoscaling is essential.

Fallacy: Continuous batching solves the whole LLM serving problem.

Continuous batching dramatically improves GPU utilization for LLM serving, but it addresses only one dimension of the problem. Prefill remains a bottleneck for long contexts, as the quadratic attention computation cannot be avoided regardless of how subsequent decode iterations are batched. Section 1.3 demonstrates that KV cache memory, not compute, often limits batch size. Network bandwidth between sharded model components can dominate latency for large models. Continuous batching is necessary but not sufficient for efficient LLM serving.

Pitfall: Sizing the fleet by average throughput.

Queuing theory establishes that systems provisioned for average load violate SLOs during traffic peaks. At 80 percent average utilization, a modest 25 percent traffic spike pushes utilization above 100 percent, causing unbounded queue growth. Predictive autoscaling and SLO management details how the cold start problem exacerbates this: by the time new capacity is available, which can take multiple minutes for GPU instances, SLO violations have already occurred. Capacity planning must account for peak load plus headroom, not average load.

Fallacy: Load balancing does not matter much for inference.

Simple load balancing strategies like round-robin seem adequate until examined quantitatively. Section 1.5 shows that random assignment produces maximum queue lengths of \(\mathcal{O}(\log R / \log \log R)\) across \(R\) servers. Power-of-two-choices reduces this to \(\mathcal{O}(\log \log R)\), an exponential improvement. For a 1,000-server cluster, this translates from ~4-5 requests maximum queue to ~2 requests. At the tail latencies that determine SLO compliance, this difference is substantial. The choice of load balancing algorithm has first-order impact on system performance.

Pitfall: Ignoring the serving tax.

Distributed inference introduces overhead absent from single-machine serving: network round-trips, serialization, load balancer decisions, and coordination for sharded models. This “serving tax” often consumes 10–30 percent of the latency budget. A team that achieves 70 ms model inference on a single GPU may be surprised when end-to-end latency reaches 100 ms in production due to these overheads. Latency budgets must explicitly account for distribution overhead, not just compute time.

Fallacy: More GPU memory always means larger batch size and higher throughput.

Larger GPU memory enables larger batches only for models whose throughput is capacity-bound. The binding constraint in LLM decode is memory bandwidth, not capacity: once HBM bandwidth saturates, adding memory does not increase tokens per second. The bottleneck hierarchy makes the mistake visible.

While larger GPU memory enables larger batches for models that fit in memory, the bottleneck often shifts before memory is exhausted. Memory bandwidth limits throughput for bandwidth-bound operations (LLM decode). Compute limits throughput for compute-bound operations (prefill, vision inference). A bandwidth-bound decode workload on an H100 with 80 GB memory eventually plateaus when HBM3 bandwidth (3.35 TB/s) saturates; adding memory alone does not raise token throughput once that point is reached. Understanding whether the workload is compute-bound, memory-bandwidth-bound, or capacity-bound guides appropriate resource allocation.

Pitfall: Neglecting multi-tenancy isolation until production.

In development and staging, single-tenant deployments work well. In production, noisy neighbors cause sudden, unpredictable performance degradation that is difficult to diagnose and resolve. A tenant bursting to 5\(\times\) normal traffic can degrade latency for all other tenants on shared infrastructure. As emphasized in section 1.6, resource quotas, priority scheduling, and bulkhead isolation must be designed into the system from the start, not retrofitted after production incidents.

Avoiding these pitfalls keeps serving infrastructure resilient to traffic variance, cost-efficient under fixed SLOs, and consistently responsive across tenants.

Self-Check: Question

  1. A team upgrades a cluster serving a 13B LLM from 80 GB GPUs to 140 GB GPUs with identical HBM3 memory bandwidth (3.35 TB/s). Why does this hardware upgrade fail to increase single-request or small-batch decode token throughput?

    1. Autoregressive decode is strictly compute-bound; increasing memory without adding Tensor Cores leaves execution time unchanged.
    2. Autoregressive decode is memory-bandwidth bound (\(D_{\text{vol}} / \text{BW}\)); once model weights fit resident in memory, token generation speed is constrained by memory bandwidth, not total memory capacity.
    3. 140 GB GPUs automatically disable CUDA graph compilation, adding driver launch latency to every iteration.
    4. The larger memory capacity forces PagedAttention block sizes to shrink from 16 tokens to 2 tokens.

  2. A capacity planner provisions an inference serving fleet for 80% average utilization based on mean daily traffic. According to queuing theory and autoscaling dynamics, why does this lead to catastrophic P99 SLO violations during traffic spikes?

  3. True or False: Deploying continuous batching alone completely solves all throughput and latency bottlenecks in LLM serving.

See Answers →

Summary

Inference at scale is the service layer that turns trained models and distributed hardware into low-latency global predictions. While training optimizes aggregate throughput (\(O/R_{\text{peak}}\)) over a fixed run, production serving inverts every assumption: OpEx compounds continuously on every request, and user-visible tail latency (\(L_{\text{lat}}\)) dictates whether the service remains viable under strict P99 service-level objectives.

At the core of serving performance is a fundamental physical split between compute and memory bandwidth. The prefill phase is compute-bound (\(O/R_{\text{peak}}\)), achieving high arithmetic intensity as prompt tokens are processed in parallel. The autoregressive decode phase, by contrast, is strictly memory-bandwidth bound (\(D_{\text{vol}}/\text{BW}\)): each generated token forces a full read of model parameters and an expanding key-value cache across HBM. This divergence means single-replica efficiency (\(\eta_{\text{hw}}\)) fluctuates wildly across a request’s lifecycle, making static batching inefficient.

To maximize throughput under memory-bandwidth constraints, modern serving architectures co-design the memory hierarchy and the iteration scheduler. Continuous batching eliminates idle accelerator cycles by inserting new requests at token boundaries; PagedAttention eliminates physical memory fragmentation (\(D_{\text{frag}}\)) by dynamically allocating KV-cache pages; and prefill/decode disaggregation separates compute-bound prompt processing from bandwidth-bound token generation into specialized hardware pools.

When model footprint or request volume exceeds single-device HBM capacity, model sharding (tensor and expert parallelism) and global load balancing distribute state across accelerators. However, distribution incurs a network synchronization tax (\(L_{\text{comm}}\)) governed by interconnect latency floors (\(\alpha\)) and transfer bandwidth (\(\beta\)). Across all tiers of the serving hierarchy—from request-level batching to platform-level multi-tenancy—the engineering objective is to maximize resource utilization and admitted request capacity while strictly bounding tail latency within the user’s SLO budget.

Key Takeaways: Serving inverts every assumption
  • Serving cost compounds forever: Training is paid once, but inference OpEx accrues on every request (principle 15); for high-traffic production systems it can dominate by 100\(\times\) or more over a model’s lifetime. Milliseconds, memory fragments, and utilization points are financial levers, not local optimizations.
  • Tail latency governs architecture: Inference systems optimize under P99 SLOs, so batching, sharding, caching, and autoscaling must raise utilization without spending the user-visible deadline. Aggregate throughput is necessary but insufficient when a slow request can break downstream services.
  • Decode turns memory into capacity: LLM prefill is compute-bound, while autoregressive decode rereads weights and expands private KV state token by token (principle 14). Continuous batching, PagedAttention, prefix reuse, and KV compression matter because they convert scarce HBM bandwidth and memory into admitted requests.
  • Model class sets the primitive: Vision models benefit from predictable static or dynamic batches, recommenders from feature-parallel embedding sharding, and LLMs from iteration-level scheduling. A fleet scheduler must match the workload shape instead of applying one universal batch-size rule.
  • Distribution always sends a bill: Tensor parallelism, expert routing, global load balancing, multi-tenancy, and failover (principle 16) all buy capacity or isolation by adding communication and coordination. Production serving works only when that tax is explicitly budgeted inside latency, cost, and reliability envelopes.

Ultimately, serving efficiency is governed by the serving cost dominance law (principle 15): training CapEx is paid once, but inference OpEx accrues continuously on every request over the model’s operational lifetime. Because autoregressive decode is physically constrained by HBM bandwidth (\(D_{\text{vol}} / \text{BW}\)) rather than raw arithmetic (\(R_{\text{peak}}\)), system efficiency depends on converting idle memory capacity and bandwidth into admitted concurrency. At fleet scale, recovering a single megabyte of fragmented KV cache or reclaiming ten milliseconds of synchronization tax translates directly into millions of dollars in sustained operational savings.

What’s Next: From data center to edge
The architectural foundations for inference at scale developed here—continuous batching, PagedAttention, model sharding, and queue-depth load balancing—all assume data center-grade hardware: enterprise accelerators, high-speed NVLink interconnects (Collective Communication), and terabytes of HBM. Building on local runtime compilation (Performance Engineering), this chapter scaled serving to data center fleets. Many applications, however, require predictions closer to the user: on smartphones, IoT devices, or local gateways where power budgets are measured in milliwatts, memory in megabytes, and connectivity is intermittent. Edge Intelligence examines how serving principles change when intelligence moves to the edge and every data center assumption is inverted.

Self-Check: Question

  1. Which statement best captures the central physical and economic synthesis of inference at scale presented in the chapter?

    1. Inference costs are fixed one-time CapEx expenditures, while training costs scale continuously with user traffic.
    2. All inference workloads should be served using static batching on single-GPU instances to eliminate network communication.
    3. Serving cost compounds continuously as OpEx; decode throughput is physically bound by HBM bandwidth (\(D_{\text{vol}} / \text{BW}\)), requiring co-design of iteration schedulers, paged memory managers, and sharding topologies.
    4. Multi-tenancy isolation can be safely omitted if GPUs are upgraded to Blackwell architectures.

  2. Why does production inference reverse the core optimization priority of distributed training from aggregate throughput (samples/hour) to tail latency (P99 ms)?

  3. The economic principle stating that lifetime model serving operational costs (OpEx) compound continuously and typically dwarf one-time training capital expenditures (CapEx) by orders of magnitude is known as the ____ Law.

See Answers →

Self-Check Answers

Self-Check: Answer

  1. Which combination of architectural design choices correctly reflects the requirements of autoregressive large language model (LLM) serving compared to vision and recommendation workloads?

    1. Continuous batching, paged memory management, preemptive scheduling, and stateful routing
    2. Static batching, preallocated memory management, first-come-first-served (FCFS) scheduling, and stateless routing
    3. Feature-parallel batching, distributed memory tables, priority-aware scheduling, and stateless routing
    4. Streaming frame batching, pinned buffer allocation, non-preemptive scheduling, and edge-only topology

    Answer: The correct answer is A. Autoregressive LLM serving requires continuous batching (iteration-level scheduling to handle variable sequence lengths), paged memory management (PagedAttention to eliminate KV cache fragmentation), preemptive scheduling (to prevent long requests from causing head-of-line blocking), and stateful routing (sticky sessions for KV cache reuse). The static and preallocated configuration characterizes uniform vision models; the feature-parallel configuration characterizes sparse recommendation systems; and the streaming frame configuration describes real-time audio and robotics pipelines.

    Learning Objective: Evaluate how the computational characteristics of autoregressive LLMs dictate their specific profile across the five serving architecture dimensions.

  2. Why does preallocated memory management in LLM serving waste 60% to 80% of GPU KV cache memory, and how does paged memory management solve this problem?

    Answer: Preallocated memory reserves a contiguous memory buffer for each request sized to the maximum possible sequence length (e.g., 4,096 tokens) upon admission, causing massive internal fragmentation when requests generate far fewer tokens (e.g., 100 tokens), as well as external fragmentation from non-contiguous free gaps. Paged memory management (PagedAttention) divides KV cache storage into fixed-size physical blocks (e.g., 16 tokens/block) mapped through a dynamic block table, allocating pages strictly on demand and reclaiming them immediately upon completion to achieve near-100% memory utilization.

    Learning Objective: Explain the root causes of KV cache memory fragmentation under preallocation and how paged allocation mitigates them.

  3. True or False: The statefulness of an inference serving system is an optional configuration feature of the serving framework rather than an inherent property of the model architecture.

    Answer: False. Statefulness is a direct consequence of the model architecture. Autoregressive LLMs maintain per-session KV cache across decoding iterations and multi-turn conversations, making them inherently stateful and necessitating sticky routing and complex session-draining logic. In contrast, fixed-computation vision models and embedding lookups are stateless by nature, allowing any replica to serve any request independently.

    Learning Objective: Justify why statefulness in inference serving is determined by model architectural mechanics rather than framework configuration.

  4. Why is pure First-Come-First-Served (FCFS) scheduling problematic for multi-tenant LLM serving, and what mechanism does preemptive scheduling use to resolve it?

    1. FCFS requires excessive GPU memory for block tables; preemptive scheduling replaces block tables with static pointers.
    2. FCFS causes extreme AllReduce communication overhead; preemptive scheduling converts tensor parallelism to pipeline parallelism.
    3. FCFS forces all requests to run at INT4 precision; preemptive scheduling dynamically promotes high-priority requests to FP16.
    4. FCFS suffers from head-of-line blocking when long generations delay short queries; preemptive scheduling swaps long-running KV caches to CPU DRAM to free GPU slots.

    Answer: The correct answer is D. In LLM generation, output lengths exhibit high variance (e.g., 50 tokens vs. 4,000 tokens). Under FCFS, a long-generation request monopolizes GPU execution and KV cache slots, creating severe head-of-line blocking for subsequent short requests. Preemptive scheduling resolves this by pausing lower-priority or long-running requests, paging their KV cache to CPU DRAM, and allocating the freed GPU capacity to shorter or higher-priority requests before later resuming the victim.

    Learning Objective: Analyze the causes of head-of-line blocking in FCFS LLM scheduling and the operational trade-offs of KV-cache preemption.

  5. The scheduling technique that decouples batch membership from request boundaries and evaluates sequence entry and termination at every decode iteration is called ____ batching (or iteration-level scheduling).

    Answer: continuous (or continuous batching / iteration-level). Continuous batching dynamically admits waiting requests and releases completed request slots at every single decoding step, eliminating the idle bubbles caused by static batching.

    Learning Objective: Identify the exact technical term for iteration-level dynamic batching in LLM serving engines.

← Back to Questions

Self-Check: Answer

  1. A vision model has fixed latency overhead \(T_{\text{fixed}} = 2\text{ ms}\) and variable compute time \(T_{\text{var}} = 1\text{ ms/item}\). An LLM decode step has \(T_{\text{fixed}} = 40\text{ ms}\) (loading 140 GB weights from HBM) and \(T_{\text{var}} = 0.5\text{ ms/token}\). At what approximate batch size \(B\) does each workload reach the ‘knee’ of its batching efficiency curve (\(B \approx T_{\text{fixed}} / T_{\text{var}}\))?

    1. Vision knee at \(B \approx 10\); LLM decode knee at \(B \approx 20\)
    2. Vision knee at \(B \approx 80\); LLM decode knee at \(B \approx 2\)
    3. Vision knee at \(B \approx 4\); LLM decode knee at \(B \approx 40\)
    4. Vision knee at \(B \approx 2\); LLM decode knee at \(B \approx 80\)

    Answer: The correct answer is D. The knee of the batching efficiency curve occurs where the variable compute cost (\(B \times T_{\text{var}}\)) begins to exceed fixed overhead (\(T_{\text{fixed}}\)). For the vision model, \(B_{\text{knee}} \approx 2\text{ ms} / 1\text{ ms} = 2\). For LLM decode, \(B_{\text{knee}} \approx 40\text{ ms} / 0.5\text{ ms} = 80\). This demonstrates why LLM serving requires massive continuous batches: the huge fixed cost of streaming model weights across HBM requires packing dozens to hundreds of concurrent requests to amortize bandwidth overhead. The inverted result swaps the two workloads; the intermediate configurations miscalculate the quotient of fixed and variable parameters.

    Learning Objective: Calculate the knee point on the batching efficiency curve for compute-bound versus memory-bandwidth-bound inference workloads.

  2. Define the Waste Ratio (\(W = 1 - \frac{\bar{S}}{S_{\text{max}}}\)) in traditional static LLM batching, and explain why continuous batching reduces this waste ratio to near zero.

    Answer: In traditional static batching, all \(B\) sequences in a batch must remain on the GPU until the longest sequence (\(S_{\text{max}}\)) completes, performing \(B \times S_{\text{max}}\) total decode iterations while only \(\sum S_i = B \times \bar{S}\) iterations represent useful work. The fraction of idle, padded compute is \(W = 1 - \bar{S}/S_{\text{max}}\), which often exceeds 50% under realistic length variance. Continuous batching evaluates sequences at each iteration step, immediately evicting finished sequences and inserting new waiting requests into freed slots, eliminating idle padding slots.

    Learning Objective: Analyze the mathematical formulation of batching waste in static LLM execution and the mechanism of continuous batching.

  3. **Order the four stages of a production recommendation system pipeline under feature-parallel batching:

  1. Feature Interaction (computing cross-feature embeddings via attention/factorization)
  2. Dense Feature Processing (transforming and normalizing continuous dense features)
  3. Sparse Feature Lookup (retrieving embeddings from sharded embedding tables across servers)
  4. Ranking Head (evaluating MLP/transformer layers to produce final candidate scores)**

Answer: The correct sequence is (3) -> (2) -> (1) -> (4). - (3) Sparse Feature Lookup retrieves embedding vectors for user, item, and context IDs in parallel across sharded feature stores. - (2) Dense Feature Processing transforms continuous/numerical signals on local CPU/GPU compute. - (1) Feature Interaction computes cross-terms between dense representations and retrieved sparse embedding vectors. - (4) Ranking Head executes the final deep neural network forward pass to compute click-through or engagement probability scores.

Learning Objective: Structure the sequential execution stages of a feature-parallel recommendation model pipeline.

  1. A production inference service targets an aggregate arrival rate \(\lambda_{\text{arr}} = 1{,}000\text{ requests/sec}\) with a strict P99 latency SLO of \(T_{\text{lat}} = 100\text{ ms}\) (\(0.1\text{ s}\)). Each GPU replica executes batches of size \(B = 8\) with an observed replica latency of \(80\text{ ms}\) (\(0.08\text{ s}\)). According to Little’s Law (\(Q_{\text{req}} = \lambda_{\text{arr}} \cdot T_{\text{lat}}\)), what is the required concurrency and the minimum number of SLO-sized replicas needed?

    1. Required concurrency \(Q_{\text{req}} = 100\); minimum SLO-sized fleet \(= 13\) replicas
    2. Required concurrency \(Q_{\text{req}} = 1{,}000\); minimum SLO-sized fleet \(= 125\) replicas
    3. Required concurrency \(Q_{\text{req}} = 80\); minimum SLO-sized fleet \(= 10\) replicas
    4. Required concurrency \(Q_{\text{req}} = 100\); minimum SLO-sized fleet \(= 10\) replicas

    Answer: The correct answer is A. By Little’s Law, required system concurrency is \(Q_{\text{req}} = \lambda_{\text{arr}} \times T_{\text{lat}} = 1{,}000\text{ req/s} \times 0.1\text{ s} = 100\text{ concurrent requests}\). Sizing for SLO concurrency across replicas with batch size \(B=8\) requires \(\lceil 100 / 8 \rceil = 13\text{ replicas}\). Note that a throughput-only calculation (\(1{,}000 / (8/0.08) = 10\text{ replicas}\)) runs right at the 100% saturation stability boundary, whereas 13 replicas provides the required concurrency headroom and operates at \(\approx 76.9\%\) service utilization to protect against tail queuing spikes.

    Learning Objective: Apply Little’s Law to calculate system concurrency requirements and size an inference replica fleet to satisfy latency SLOs.

  2. What is the ‘Logic Wall’ in test-time compute scaling, and how does it fundamentally shift the serving optimization objective from tokens-per-second to steps-per-second?

    Answer: The Logic Wall occurs when generative models utilize deliberative ‘slow thinking’ (generating chain-of-thought tokens, tree search, or self-correction steps) where compute per request scales dynamically with problem difficulty rather than remaining fixed. This shifts the serving constraint from maximizing raw token throughput (tokens/sec across memory bandwidth) to managing search steps per second and enforcing ‘reasoning SLOs,’ requiring dynamic compute allocation that budgets thinking depth based on prompt complexity and fleet load.

    Learning Objective: Explain the systems implications of test-time compute scaling and the concept of the Logic Wall.

  3. True or False: In adaptive batching systems like NVIDIA Triton, increasing the batching timeout window (\(T_{\text{window}}\)) during sudden high-traffic load spikes is the optimal policy to reduce tail latency.

    Answer: False. During high traffic arrival rates (\(\lambda_{\text{current}}\)), the batch window should shrink (\(T_{\text{window}} = \min(T_{\text{max}}, B_{\text{target}}/\lambda_{\text{current}})\)) because batches fill rapidly, allowing immediate dispatch to minimize queuing delay. Expanding the window during high load creates unnecessary batch formation delay and inflates P99 latency.

    Learning Objective: Evaluate adaptive batching control policies under varying traffic arrival intensities.

← Back to Questions

Self-Check: Answer

  1. How does PagedAttention eliminate memory waste in LLM KV cache management, and what is its primary system-level benefit?

    1. It compresses KV cache tensors using lossless entropy coding to reduce arithmetic intensity in attention kernels.
    2. It replaces standard multi-head attention with grouped query attention by dynamically merging key-value heads.
    3. It stores attention KV states in non-contiguous, fixed-size physical memory pages mapped via logical block tables, boosting batch capacity \(2\text{--}4\times\).
    4. It offloads 100% of KV cache data to host CPU DRAM, eliminating GPU HBM usage entirely for active sequences.

    Answer: The correct answer is C. PagedAttention adapts OS virtual memory principles to KV cache allocation. Logical token positions are mapped to fixed-size physical blocks (e.g., 16 tokens/page) using a per-sequence block table. This eliminates internal fragmentation (allocating only active pages) and external fragmentation (any free block can be assigned to any sequence), increasing KV pool utilization from 30–40% to over 95% and enabling 2–4\(\times\) higher concurrent batch capacity on the same GPU hardware. The lossy compression claim misstates PagedAttention as an entropy encoder; the head-merging claim confuses PagedAttention with GQA; and the CPU DRAM offloading claim mischaracterizes a paging fallback as standard on-device execution.

    Learning Objective: Analyze the virtual memory mapping mechanism of PagedAttention and its impact on KV cache fragmentation and serving throughput.

  2. What is the ‘speculation tax’ in speculative decoding, and how does it influence whether speculative decoding should be enabled for a specific serving endpoint?

    1. The latency penalty paid when draft models fail to generate valid UTF-8 token encodings.
    2. The GPU memory allocated to draft model weights and draft KV cache, which reduces available HBM for batch concurrency.
    3. The serialization tax incurred when transmitting draft tokens across InfiniBand interconnects.
    4. The loss in model generation accuracy caused by greedy sampling during draft token verification.

    Answer: The correct answer is B. Speculative decoding requires resident GPU memory for the draft model’s weights (e.g., ~14 GB for a 7B draft pairing with a 70B target) plus concurrent draft KV cache buffers. This memory cannot be allocated to concurrent request batches. Therefore, speculation is advantageous for latency-bound endpoints (interactive chat with strict TPOT SLOs) where higher individual token speed justifies lower concurrency, but detrimental to throughput-bound endpoints (bulk processing) where lost batch concurrency degrades aggregate tokens/sec.

    Learning Objective: Evaluate the memory and throughput trade-offs of speculative decoding as an admission and scheduling policy under service-level agreements.

  3. A chatbot platform serves 1,000 concurrent users who all share a common 2,000-token system prompt, generating an average of 500 response tokens each. Explain how prefix caching reduces total KV cache memory from ~6.6 TB to ~1.3 TB (~80% reduction).

    Answer: Without prefix caching, all 1,000 users store the full 2,500 tokens (2,000 prompt + 500 response) separately, requiring \(1{,}000 \times 2{,}500 = 2{,}500{,}000\) token-cache allocations (~6.6 TB on Llama-70B FP16). With prefix caching enabled via PagedAttention copy-on-write block sharing, the 2,000-token system prompt is stored once in physical blocks and referenced by all user block tables, so the system only allocates memory for \(2{,}000 + (1{,}000 \times 500) = 502{,}000\) tokens (~1.3 TB), achieving an ~80% memory reduction and enabling 5\(\times\) higher concurrent capacity.

    Learning Objective: Calculate KV cache memory savings achieved through prefix caching in shared-context multi-tenant workloads.

  4. Why do un-chunked long prompt prefills (e.g., 10,000 tokens) cause severe tail latency spikes for concurrent decode requests, and how does chunked prefill (Sarathi-Serve) resolve this?

    Answer: Prompt prefill is compute-heavy and executes over hundreds of milliseconds. In a unified engine, an un-chunked prefill monopolizes GPU Tensor Cores, completely blocking ongoing decode iterations for all concurrent active requests and triggering severe Time Per Output Token (TPOT) spikes. Chunked prefill divides long prompts into fixed-size token chunks (e.g., 200–512 tokens) and interleaves prefill chunk computation with regular decode steps, bounding prefill execution time per step and maintaining steady decode pacing.

    Learning Objective: Explain the operational mechanism of chunked prefill in preventing decode stalls and mitigating tail latency.

  5. Switching from standard Multi-Head Attention (MHA) with 64 key-value heads to Grouped Query Attention (GQA) with 8 key-value heads reduces the KV cache memory footprint by a factor of ____\(\times\).

    Answer: 8 (or 8x / eight). Because KV cache size is directly proportional to the number of key-value heads (\(H_{\text{KV}}\)), reducing \(H_{\text{KV}}\) from 64 to 8 yields an exact \(64/8 = 8\times\) reduction in cached key-value memory.

    Learning Objective: Calculate the architectural compression factor of Grouped Query Attention on KV cache capacity.

  6. **Order the steps executed during a single verification round of Speculative Decoding:

  1. Target model runs a single batched parallel forward pass across all \(K\) candidate positions
  2. Rejection sampling evaluates candidate tokens sequentially against target probability distributions
  3. Draft model autoregressively generates \(K\) candidate tokens
  4. KV cache block tables are updated to append accepted tokens and one target correction token, discarding rejected branches**

Answer: The correct sequence is (3) -> (1) -> (2) -> (4). - (3) The fast draft model autoregressively produces \(K\) speculative tokens in lightweight forward passes. - (1) The large target model verifies all \(K\) proposed tokens simultaneously in a single compute-bound forward pass. - (2) Rejection sampling determines how many draft tokens meet the target distribution criteria until a rejection or end-of-window. - (4) Accepted tokens plus one newly sampled target token are committed to the KV cache, and any unaccepted speculative allocations are released.

Learning Objective: Structure the sequence of execution and verification steps in speculative decoding.

← Back to Questions

Self-Check: Answer

  1. Why does Tensor Parallelism (TP) reduce single-request Time-to-First-Token (TTFT) whereas Pipeline Parallelism (PP) provides zero reduction in single-request latency?

    1. TP eliminates the need for activation storage, while PP doubles activation sizes across stages.
    2. TP uses point-to-point transfers over Ethernet, while PP requires NVLink broadcast collectives.
    3. TP relies on asynchronous CPU offloading, whereas PP executes purely on GPU Tensor Cores.
    4. TP partitions intra-layer matrix multiplications across devices with per-layer AllReduce, whereas PP requires a single request to traverse all sequential stages in series.

    Answer: The correct answer is D. Tensor parallelism partitions weight matrices within each transformer layer (attention heads and MLP column-row splits) across \(t\) GPUs, parallelizing the computation of each layer to reduce single-request forward-pass time by \(\approx t\times\) (minus AllReduce communication overhead). Pipeline parallelism distributes entire layers sequentially across \(p\) stages; a single request must traverse stage 0, then stage 1, …, through stage \(p-1\) in series, yielding a latency equal to the sum of all stages. PP only improves aggregate throughput when multiple concurrent requests fill the pipeline.

    Learning Objective: Compare the latency and throughput mechanics of Tensor Parallelism versus Pipeline Parallelism during inference.

  2. Explain the ‘MoE economics paradox’ for a 671B-parameter Mixture-of-Experts model (such as DeepSeek-V3 with 37B active parameters per token): why does it require multi-node cluster VRAM for capacity while exhibiting the decode latency of a small dense model?

    Answer: The paradox arises because total parameter count (671B) determines the static memory footprint (~1,342 GB in FP16, or ~671 GB in FP8), forcing the model to be distributed across multiple GPU nodes (e.g., two 8\(\times\)H100 nodes) so all expert weights remain resident in HBM. However, during autoregressive decode, the router activates only 37B parameters per token, reading only ~74 GB (FP16) from HBM and executing compute for 37B parameters per step, delivering the fast per-token memory bandwidth and FLOP speed of a much smaller model.

    Learning Objective: Analyze the memory capacity versus memory bandwidth trade-offs in serving large Mixture-of-Experts models.

  3. True or False: In model sharding, Tensor Parallelism relies on AllToAll collective communication to route tokens between layers, whereas Expert Parallelism relies strictly on AllReduce to synchronize partial attention head outputs.

    Answer: False. The communication collectives are inverted. Tensor Parallelism relies on AllReduce collectives (two per transformer block: one after attention, one after MLP row-split) to sum partial activation results across all devices. Expert Parallelism (MoE) relies on AllToAll collectives (dispatch to route token activations to expert-hosting devices, and gather to retrieve expert outputs) because different tokens are dynamically routed to different target GPUs based on gating scores.

    Learning Objective: Classify the specific communication collective patterns required by Tensor Parallelism versus Expert Parallelism.

  4. A production recommendation system operates 100+ TB of embedding tables with trillions of entities. How does a hybrid embedding sharding architecture distribute hot, warm, and cold embeddings to balance memory footprint and network overhead?

    1. Hot (top 1%) embeddings are replicated in memory across all shards; warm (10%) are column-sharded with AllGather; cold (89%) are row-sharded on SSDs with AllToAll gather.
    2. Hot embeddings are stored exclusively on remote cold storage; cold embeddings are replicated in HBM across all GPUs.
    3. All embeddings are strictly column-sharded across all GPUs regardless of popularity to eliminate network communication.
    4. Hot embeddings are sharded row-wise across CPUs; cold embeddings are broadcast via AllReduce to all GPU Tensor Cores.

    Answer: The correct answer is A. In hybrid embedding sharding (exemplified by Meta’s DLRM infrastructure), the top 1% most frequently accessed ‘hot’ embeddings are replicated in local memory across all servers to eliminate network lookup latency. The next ~10% ‘warm’ embeddings are column-sharded across devices and combined via AllGather. The remaining ~89% ‘cold’ tail embeddings are row-sharded by entity hash across large SSD-backed CPU servers and retrieved via batched AllToAll network gathers.

    Learning Objective: Design a hybrid embedding sharding topology that balances access locality, network collectives, and memory tiering for trillion-parameter recommendation systems.

  5. **Order the sequence of operations that occurs during token execution in an Expert-Parallel (MoE) transformer layer:

  1. AllToAll dispatch transmits token activation vectors to the specific GPUs hosting the selected experts
  2. Gating network evaluates input representations and computes top-\(k\) routing probabilities
  3. AllToAll gather routes expert output activations back to the originating host devices
  4. Selected expert sub-networks compute local feed-forward transformations on assigned tokens
  5. Weighted sum combines expert outputs with gating coefficients and adds residual connections**

Answer: The correct sequence is (2) -> (1) -> (4) -> (3) -> (5). - (2) The gating network evaluates token embeddings to select the top-\(k\) experts per token. - (1) An AllToAll dispatch collective routes token representations to the devices hosting their assigned experts. - (4) Expert devices execute feed-forward matrix multiplications in parallel on received tokens. - (3) A second AllToAll gather collective transmits computed expert outputs back to originating devices. - (5) The host device performs a weighted summation of expert outputs using routing probabilities and adds residual connections.

Learning Objective: Structure the sequence of computation and communication steps in Expert-Parallel MoE token execution.

  1. Using the \(\alpha\)-\(\beta\) communication model, explain why Tensor Parallelism across nodes linked via standard 100G Ethernet introduces prohibitive latency overhead compared to intra-node NVLink.

    Answer: Tensor parallelism requires two AllReduce operations per transformer block (160 AllReduce collectives for an 80-layer model per forward pass). Intra-node NVLink delivers 900 GB/s bandwidth with ~500 ns startup latency (AllReduce takes ~0.02 ms per layer, <0.1% of a 30 ms layer budget). Standard 100G Ethernet provides only 12.5 GB/s with ~50 \(\mu\)s latency, increasing per-layer AllReduce time to >1.1 ms (consuming >4% of each layer budget and accumulating tens of milliseconds across 80 layers), thereby erasing parallel compute gains.

    Learning Objective: Analyze the quantitative impact of interconnect bandwidth and latency on Tensor Parallelism communication overhead.

← Back to Questions

Self-Check: Answer

  1. In a 1,000-server inference fleet, why does the Power-of-Two-Choices load balancing algorithm dramatically outperform random assignment while avoiding the overhead of global least-loaded routing?

    1. It eliminates all network packet headers by broadcasting requests to all 1,000 servers simultaneously.
    2. It probes two random replicas and picks the less loaded one, reducing maximum queue depth from \(\Theta(\frac{\log R}{\log \log R}) \approx 4\text{--}5\) to \(\Theta(\log \log R) \approx 2\) with \(\mathcal{O}(1)\) overhead.
    3. It guarantees zero queuing delay by dynamically compiling new CUDA kernels on the fly for each request.
    4. It forces all requests to execute on CPU fallback nodes whenever GPU utilization exceeds 50%.

    Answer: The correct answer is B. Under pure random assignment, the maximum queue length across \(R\) servers follows the classical balls-into-bins bound \(\Theta(\log R / \log \log R)\) (yielding ~4–5 requests at \(R=1{,}000\)). Querying just two randomly sampled servers and routing to the one with the lighter load drops the maximum queue bound exponentially to \(\Theta(\log \log R)\) (~2 requests at \(R=1{,}000\)). This achieves near-optimal load distribution with only two lightweight probes per request, completely avoiding the expensive \(\mathcal{O}(R)\) state polling of global least-loaded algorithms.

    Learning Objective: Analyze the theoretical queue-length bounds and operational advantages of the Power-of-Two-Choices load balancing algorithm.

  2. Explain why Consistent Hashing is critical for routing multi-turn LLM conversations, and describe what happens to KV cache state when an autoscaling event adds a new replica.

    Answer: Consistent hashing maps user or session IDs onto a virtual hash ring to deterministically route subsequent conversation turns to the same replica holding the accumulated KV cache, avoiding expensive prefill context reconstruction (~500 ms per turn). When an autoscaling event adds a new node, consistent hashing remaps only a minimal fraction (\(1/N_{\text{servers}}\)) of keys to the new node while the remaining \((N-1)/N\) keys retain their replica mappings, preserving cache locality across the vast majority of ongoing sessions.

    Learning Objective: Explain the role of consistent hashing in maintaining KV cache session affinity and minimizing cache invalidation during fleet scaling.

  3. True or False: Standard HTTP 200 process liveness checks are sufficient to detect GPU inference degradation caused by thermal throttling or silent HBM memory pool exhaustion.

    Answer: False. Process-level HTTP liveness checks only confirm that the web server daemon is running. A GPU experiencing thermal throttling, corrupted memory modules, unallocated KV pools, or un-warmed model weights will continue returning HTTP 200 while failing or stalling on real inference queries. Production fleets require Tiered Probing, specifically Deep Health checks that execute lightweight test inference requests and validate logit output distributions.

    Learning Objective: Differentiate between process liveness probes, readiness checks, and deep inference health checks in GPU serving fleets.

  4. A heterogeneous inference cluster contains 10 H100 servers (delivering 1,000 QPS capacity each) and 20 A100 servers (delivering 600 QPS capacity each), serving a total target load of 15,000 QPS. What weighted routing ratio should be applied to achieve equal utilization across all hardware tiers?

    1. Route 500 QPS equally to every server regardless of type, producing 50% utilization on H100 and 83.3% on A100.
    2. Assign weight 1.0 to H100 and 1.0 to A100 so all servers receive equal traffic shares.
    3. Assign capacity-proportional weights (H100 weight \(= 1000/22000 \approx 4.55\%\) per server; A100 weight \(= 600/22000 \approx 2.73\%\) per server), achieving uniform 68.2% utilization across all nodes.
    4. Route 100% of traffic to A100 servers first and overflow remaining traffic to H100 servers.

    Answer: The correct answer is C. Total cluster capacity is \((10 \times 1{,}000) + (20 \times 600) = 10{,}000 + 12{,}000 = 22{,}000\text{ QPS}\). Capacity-proportional routing assigns each H100 server a weight of \(1{,}000 / 22{,}000 \approx 4.55\%\) (\(681.8\text{ QPS}\) load, \(68.2\%\) utilization) and each A100 server a weight of \(600 / 22{,}000 \approx 2.73\%\) (\(409.1\text{ QPS}\) load, \(68.2\%\) utilization). Routing equal requests to heterogeneous nodes overloads the lower-capacity A100s (\(83.3\%\) utilization) while underutilizing H100s (\(50\%\) utilization).

    Learning Objective: Calculate capacity-proportional routing weights to balance resource utilization across heterogeneous accelerator fleets.

  5. The three-state resilience pattern that fails fast and stops routing traffic to a degraded or thermally throttled GPU replica before retries cascade across the fleet is called a ____.

    Answer: circuit breaker (or Circuit Breaker / circuit breaker pattern). A circuit breaker transitions from closed to open when error rates or latencies exceed a threshold, isolating the degraded replica and protecting the rest of the fleet.

    Learning Objective: Identify the circuit breaker pattern as the primary mechanism for preventing cascading failures in inference clusters.

← Back to Questions

Self-Check: Answer

  1. Which statement best distinguishes deployment-level bulkheads from request-level bulkheads in a multi-tenant inference platform?

    1. Deployment-level bulkheads enforce rate limits using software counters; request-level bulkheads isolate GPU power supplies.
    2. Deployment-level bulkheads apply only to batch inference; request-level bulkheads apply only to interactive chat.
    3. Deployment-level bulkheads dynamically quantize weights to INT4; request-level bulkheads dynamically compile CUDA graphs.
    4. Deployment-level bulkheads physically isolate tenant tiers onto dedicated GPU replicas, while request-level bulkheads cap input/output token lengths and execution time within shared processes.

    Answer: The correct answer is D. Deployment-level bulkheads physically partition hardware by assigning dedicated GPU instances to high-priority (e.g., Enterprise/Gold) tiers, ensuring complete isolation from noisy neighbors and infrastructure failures. Request-level bulkheads operate within shared processes by enforcing strict limits on maximum input context length, maximum output tokens, and timeout deadlines to prevent any single query from monopolizing shared GPU memory or compute.

    Learning Objective: Compare the architectural mechanisms and isolation guarantees of deployment-level versus request-level bulkheads.

  2. The scaling response time budget is defined as \(T_{\text{response}} = T_{\text{detect}} + T_{\text{decide}} + T_{\text{provision}} + T_{\text{warmup}}\). When autoscaling a 70B-parameter LLM replica on fresh cloud instances, which component overwhelmingly dominates the delay, and what strategy mitigates it?

    1. \(T_{\text{provision}}\) dominates (5–7 minutes downloading 140 GB weights from remote storage); mitigated by maintaining pre-warmed pools or predictive scaling.
    2. \(T_{\text{decide}}\) dominates (10–15 minutes evaluating autoscaling PID loops); mitigated by switching to manual scaling.
    3. \(T_{\text{detect}}\) dominates (30 minutes gathering Prometheus metrics); mitigated by removing metric telemetry entirely.
    4. \(T_{\text{warmup}}\) dominates (1–2 hours running CUDA kernel JIT compilation); mitigated by disabling Tensor Cores.

    Answer: The correct answer is A. Bringing up a fresh 70B LLM replica requires provisioning a GPU node and downloading ~140 GB of model weights from object storage (e.g., S3), which typically consumes 5–7 minutes (\(T_{\text{provision}}\)), far exceeding metric detection (10–60s) or autoscaler decision (1–5s) times. Because reactive autoscaling cannot absorb sudden traffic surges during this multi-minute delay, serving platforms must maintain pre-warmed replica pools (Hot/Warm tiers) or initiate predictive scaling ahead of expected traffic ramps.

    Learning Objective: Analyze the components of autoscaling response latency and evaluate strategies for mitigating cold-start delays.

  3. When a cloud provider issues a 2-minute termination warning for a spot/preemptible GPU inference instance, describe the 4-step graceful shutdown sequence that must be executed to prevent data corruption and user-visible failures.

    Answer: The 4-step graceful termination sequence is:

  4. Stop Admission & Deregister: Immediately reject new incoming requests and deregister the replica from the load balancer to halt new traffic.

  5. Drain In-Flight Requests: Allow currently active decode requests to finish generating within a strict timeout window (e.g., 90 seconds).

  6. Persist State: Save recoverable session state and active KV cache buffers to shared CPU DRAM or distributed storage for resumption.

  7. Graceful Shutdown: Cleanly terminate worker processes and release accelerator resources before the provider reclaims the instance.

    Learning Objective: Structure the graceful termination and state-preservation workflow for preemptible/spot inference instances.

  8. True or False: Cross-region model sharding (spanning pipeline stages between US-East and Asia-Pacific) is an effective strategy for minimizing single-request P99 latency.

    Answer: False. Cross-region network round-trip time (RTT) between distant data centers is 75–150 ms, which is 10–100\(\times\) larger than the per-stage compute time of an individual layer (1–5 ms). Inter-region pipeline communication severely bottlenecks execution, making cross-region sharding a last-resort disaster-recovery capacity workaround rather than a latency optimization. Latency minimization requires co-locating all sharded stages within a single datacenter and deploying independent regional replicas.

    Learning Objective: Evaluate the latency implications and feasibility constraints of cross-region distributed model sharding.

  9. When multiple tenants with unequal demand compete for a shared GPU pool, the fair-allocation algorithm that maximizes the minimum allocation by bounding per-tenant consumption using token generation rate (tokens/sec) is called ____ fairness.

    Answer: max-min (or Max-Min / max-min fairness). Max-min fairness allocates equal capacity up to each tenant’s demand, redistributing unused headroom from low-demand tenants to high-demand tenants.

    Learning Objective: Identify the max-min fairness principle applied to multi-tenant inference resource allocation.

← Back to Questions

Self-Check: Answer

  1. How does SmoothQuant enable efficient 8-bit weight and 8-bit activation (W8A8) inference, and why is its speedup significantly higher during prefill than during decode?

    1. It prunes 50% of model weights using structured sparsity, accelerating decode but having no effect on prefill.
    2. It multiplies activations by a per-channel smoothing scale \(s\) and divides weights by \(s\) to migrate outlier difficulty from activations to weights; prefill benefits from native INT8 Tensor Core compute (near \(2\times\)), while decode is memory-bandwidth bound (\(1.3\text{--}1.5\times\)).
    3. It rotates the coordinate basis using randomized Walsh-Hadamard matrices, eliminating memory bandwidth bottlenecks during decode.
    4. It converts floating-point weights into binary spikes, enabling single-cycle bitwise operations during token generation.

    Answer: The correct answer is B. SmoothQuant observes that activation outliers occur in specific persistent channels while weights are easy to quantize. It applies a per-channel scale factor \(s\) to divide activations and multiply weights (\(Y = (X \operatorname{diag}(s)^{-1}) \cdot (\operatorname{diag}(s) W)\)), mathematically preserving layer output while migrating quantization difficulty into the weights. Prefill processes large token batches in parallel and is compute-bound, achieving near \(2\times\) speedup on INT8 Tensor Cores. Decode processes single tokens and is memory-bandwidth bound on weight loading, achieving a lower \(1.3\text{--}1.5\times\) speedup.

    Learning Objective: Analyze the mathematical mechanism of SmoothQuant W8A8 quantization and contrast its performance gains during prefill versus decode phases.

  2. What is the core algorithmic distinction between Generalized Post-Training Quantization (GPTQ) and Activation-Aware Weight Quantization (AWQ)?

    1. GPTQ quantizes activations while leaving weights in FP16; AWQ quantizes weights while leaving activations in FP32.
    2. GPTQ requires full model retraining for 100 epochs; AWQ is an online dynamic quantization method with no calibration.
    3. GPTQ compensates residual quantization error across remaining unquantized columns using inverse Hessian information (\(H^{-1}\)); AWQ protects salient weight channels based on activation magnitudes without error feedback.
    4. GPTQ applies only to convolutional vision models; AWQ applies exclusively to sparse recommendation embedding tables.

    Answer: The correct answer is C. GPTQ performs layer-by-layer second-order error compensation: as each weight column is rounded to discrete grid points, the residual error is propagated to unquantized columns using the inverse Hessian matrix (\(H = X^T X\)). In contrast, AWQ observes activation magnitudes on a calibration set, identifies the top ~1% most salient channels, and scales those specific weight channels up prior to quantization to preserve precision without requiring iterative error compensation.

    Learning Objective: Compare the algorithmic formulations and error-mitigation strategies of GPTQ versus AWQ.

  3. Explain how rotation-based quantization (QuaRot) eliminates activation outliers in a data-free manner to unlock 4-bit weights and 4-bit activations (W4A4), and identify the runtime cost it incurs.

    Answer: QuaRot recognizes that activation outliers are coordinate-basis artifacts. It applies orthogonal randomized Hadamard rotation matrices (\(H\)) to transform activation and weight spaces (\(X' = XH, W' = H^T W\)), which spreads peak outlier values evenly across all dimensions without changing the matrix product (\(XW = X'W'\)). Because the rotation is orthogonal, it is completely data-free and enables uniform W4A4 quantization with minimal perplexity loss, at the cost of an inline Hadamard transform overhead (~3% latency) on every forward pass.

    Learning Objective: Explain the theoretical foundation of rotation-based quantization (QuaRot) and its operational trade-offs.

  4. True or False: Modern NVIDIA Hopper (H100) GPUs provide native Tensor Core instruction acceleration for FP8 (E4M3/E5M2) delivering a \(2\times\) compute throughput multiplier over FP16, whereas Ampere (A100) Tensor Cores do not natively support FP8 execution.

    Answer: True. Hopper (H100) introduced native FP8 Tensor Core support (with Transformer Engine scaling), providing a \(2\times\) arithmetic speedup over FP16 with minimal accuracy degradation. Ampere (A100) supports FP16, BF16, and INT8 (and INT4), but lacks native FP8 Tensor Core hardware instructions.

    Learning Objective: Identify native Tensor Core precision support across NVIDIA Ampere and Hopper accelerator architectures.

  5. **Order the sequential steps in the GPTQ column-wise layer quantization pipeline:

  1. Compute the Hessian matrix \(H = 2 X^T X\) from calibration activations
  2. Perform Cholesky decomposition on the inverse Hessian matrix \(H^{-1}\)
  3. Round the current column weights to the nearest discrete quantized grid
  4. Calculate the residual quantization error vector for the column
  5. Propagate compensation updates to all remaining unquantized columns using the Cholesky inverse weights**

Answer: The correct sequence is (1) -> (2) -> (3) -> (4) -> (5). - (1) Pass calibration data through the model to compute activation cross-products and form the Hessian matrix \(H\). - (2) Compute the inverse Hessian and factor it via Cholesky decomposition to enable stable triangular error updates. - (3) Round the target column’s continuous weights to the nearest low-bit quantized values. - (4) Compute the quantization error difference between original and rounded column weights. - (5) Propagate the scaled error vector to remaining unquantized columns to adjust their values before subsequent quantization steps.

Learning Objective: Structure the computational workflow of GPTQ layer-by-layer second-order quantization.

← Back to Questions

Self-Check: Answer

  1. In Google Search ranking, how does a four-stage ranking cascade (L0 Retrieval: 1M \(\to\) 10k items in 10 ms; L1 First-pass: 10k \(\to\) 1k in 20 ms; L2 Second-pass: 1k \(\to\) 100 in 50 ms; L3 Final rank: 100 \(\to\) 10 in 100 ms) reduce model evaluation work compared to scoring all candidates with the L3 ensemble?

    1. It runs L3 on all 1,000,000 candidates using asynchronous DMA transfers without CPU involvement.
    2. It replaces neural networks entirely with BM25 keyword matching across all stages.
    3. It dynamically recompiles transformer weights at each stage using TensorRT-LLM.
    4. It cuts final expensive L3 evaluations by \(10{,}000\times\) (from 1,000,000 to 100), reducing total candidate-scoring compute by \(\approx 5{,}600\times\) within a 180 ms budget.

    Answer: The correct answer is D. Running the expensive L3 model (costing 1 ms/candidate) across all 1,000,000 items would require 1,000 seconds of compute time. The four-stage cascade uses cheap heuristics (L0: embeddings/BM25) and fast linear models (L1) to eliminate 99.9% of irrelevant documents early, reserving the expensive L3 transformer ensemble for only the top 100 candidates. This reduces L3 evaluations by \(10{,}000\times\) and total model work by \(\approx 5{,}600\times\), completing the full cascade within 180 ms.

    Learning Objective: Calculate candidate reduction factors and compute savings achieved by multi-stage ranking cascades.

  2. Compare the primary binding bottlenecks and serving topologies of Meta’s recommendation infrastructure (DLRM) versus OpenAI’s GPT API infrastructure.

    Answer: Meta’s recommendation infrastructure binds on sparse embedding scale and memory bandwidth across 100+ TB tables, using a hybrid topology of 1,000+ CPU servers for feature-parallel embedding lookups coupled with GPUs for dense ranking under strict <10 ms SLOs. In contrast, OpenAI’s GPT API binds on autoregressive decode memory bandwidth and request length variance, utilizing multi-GPU tensor-parallel clusters (8\(\times\)H100) with iteration-level continuous batching, chunked prefill, and PagedAttention KV cache management under ~2s TTFT targets.

    Learning Objective: Compare the hardware topologies and bottleneck constraints of large-scale recommendation systems versus LLM serving APIs.

  3. True or False: In TikTok’s two-tower multimodal recommendation architecture, both the user tower and video content understanding tower must execute synchronously in real-time on GPU clusters for every incoming video swipe.

    Answer: False. TikTok’s two-tower architecture decouples online and offline computation: user embeddings change dynamically and execute online in real-time (~5 ms), whereas video content embeddings change slowly and are precomputed asynchronously in hourly batch jobs and cached in memory. Online serving is thus reduced to the user tower forward pass plus a fast vector similarity lookup against precomputed video embeddings.

    Learning Objective: Analyze the asynchronous online-versus-offline execution cadence of two-tower recommendation architectures.

  4. Which cross-cutting architectural principle is demonstrated by comparing production serving architectures across Meta, OpenAI, Google, and TikTok?

    1. Production inference rejects one-size-fits-all stacks: systems specialize hardware and batching to match their binding bottleneck (embeddings, variable decode, cascade cost, or asynchronous freshness).
    2. All modern inference workloads are compute-bound and converge onto identical 8-way tensor-parallel GPU clusters.
    3. Traditional static batching with First-Come-First-Served scheduling remains optimal for all production workloads when scaled to 1,000 GPUs.
    4. Quantization to 1-bit binary representations is universally deployed across all production recommendation and search systems.

    Answer: The correct answer is A. The cross-cutting lesson across hyperscale case studies is specialization: Meta optimizes for distributed sparse embedding locality; OpenAI optimizes for iteration-level decode variance and KV caching; Google optimizes ranking cascade depths under strict deadlines; and TikTok optimizes online-offline two-tower freshness. No single serving architecture satisfies all workloads.

    Learning Objective: Synthesize the cross-cutting architectural principles governing production inference systems at scale.

← Back to Questions

Self-Check: Answer

  1. A team upgrades a cluster serving a 13B LLM from 80 GB GPUs to 140 GB GPUs with identical HBM3 memory bandwidth (3.35 TB/s). Why does this hardware upgrade fail to increase single-request or small-batch decode token throughput?

    1. Autoregressive decode is strictly compute-bound; increasing memory without adding Tensor Cores leaves execution time unchanged.
    2. Autoregressive decode is memory-bandwidth bound (\(D_{\text{vol}} / \text{BW}\)); once model weights fit resident in memory, token generation speed is constrained by memory bandwidth, not total memory capacity.
    3. 140 GB GPUs automatically disable CUDA graph compilation, adding driver launch latency to every iteration.
    4. The larger memory capacity forces PagedAttention block sizes to shrink from 16 tokens to 2 tokens.

    Answer: The correct answer is B. During autoregressive decoding, every generated token requires loading all model weights from HBM to on-chip SRAM (\(T_{\text{decode}} \approx \text{Model Bytes} / \text{Memory Bandwidth}\)). If the model already fits comfortably in 80 GB, expanding memory capacity to 140 GB does not increase memory bandwidth (which remains 3.35 TB/s). Token generation speed remains completely unchanged unless batch size is increased sufficiently to transition the workload from bandwidth-bound to compute-bound.

    Learning Objective: Analyze why memory bandwidth, rather than memory capacity, bounds autoregressive decode throughput.

  2. A capacity planner provisions an inference serving fleet for 80% average utilization based on mean daily traffic. According to queuing theory and autoscaling dynamics, why does this lead to catastrophic P99 SLO violations during traffic spikes?

    Answer: In queuing systems, waiting time grows nonlinearly as utilization \(\rho \to 1\). An 80% baseline utilization leaves only a 20% margin; a modest 25% traffic spike pushes \(\rho > 1.0\), causing queues to grow unboundedly. Because provisioning new GPU replicas requires several minutes (\(T_{\text{provision}}\) for weight downloads and warmup), reactive autoscaling cannot react in time, resulting in massive tail latency spikes, queue overflows, and cascading request timeouts.

    Learning Objective: Explain the queuing theory and autoscaling risks of provisioning inference fleets based on average rather than peak-plus-headroom demand.

  3. True or False: Deploying continuous batching alone completely solves all throughput and latency bottlenecks in LLM serving.

    Answer: False. Continuous batching addresses only decode-phase iteration scheduling. It does not eliminate the quadratic compute bottleneck of long prompt prefills, does not solve physical KV-cache memory capacity exhaustion, and does not mitigate cross-node communication overhead in sharded models. A production serving stack requires continuous batching combined with chunked prefill, PagedAttention, prefix caching, model sharding, and load balancing.

    Learning Objective: Evaluate the scope and limitations of continuous batching within the end-to-end LLM serving pipeline.

← Back to Questions

Self-Check: Answer

  1. Which statement best captures the central physical and economic synthesis of inference at scale presented in the chapter?

    1. Inference costs are fixed one-time CapEx expenditures, while training costs scale continuously with user traffic.
    2. All inference workloads should be served using static batching on single-GPU instances to eliminate network communication.
    3. Serving cost compounds continuously as OpEx; decode throughput is physically bound by HBM bandwidth (\(D_{\text{vol}} / \text{BW}\)), requiring co-design of iteration schedulers, paged memory managers, and sharding topologies.
    4. Multi-tenancy isolation can be safely omitted if GPUs are upgraded to Blackwell architectures.

    Answer: The correct answer is C. The chapter synthesizes two central realities: economically, serving cost accumulates continuously as operational expenditure (OpEx) that can dominate training cost by 100–1000\(\times\); physically, LLM decode is memory-bandwidth bound (\(D_{\text{vol}}/\text{BW}\)). High-efficiency serving requires co-designing iteration-level continuous batching, PagedAttention memory virtualization, prefill/decode disaggregation, and load balancing while budgeting the distribution tax.

    Learning Objective: Synthesize the core physical and economic principles governing large-scale inference systems.

  2. Why does production inference reverse the core optimization priority of distributed training from aggregate throughput (samples/hour) to tail latency (P99 ms)?

    Answer: Distributed training is an offline batch workload where progress is measured in aggregate samples processed over hours or days, tolerating high variance and periodic checkpoint rollbacks. Production inference is a real-time, user-facing service where revenue and user engagement depend on strict P99 latency bounds (milliseconds); high latency variance or dropped requests cause immediate user abandonment and cascading failures in dependent microservices.

    Learning Objective: Justify why production inference prioritizes tail latency over aggregate batch throughput.

  3. The economic principle stating that lifetime model serving operational costs (OpEx) compound continuously and typically dwarf one-time training capital expenditures (CapEx) by orders of magnitude is known as the ____ Law.

    Answer: Serving Cost Dominance (or Serving Cost Dominance Law). The Serving Cost Dominance Law highlights that inference efficiency improvements provide continuous financial return over a model’s entire deployment lifetime.

    Learning Objective: Identify the Serving Cost Dominance Law governing the lifetime economics of machine learning systems.

← Back to Questions

Back to top