Acceleration Fundamentals

Hardware Acceleration

Purpose

Why does moving data cost more than computing it?

Arithmetic is inexpensive relative to many memory accesses. In the time it takes to fetch a value from main memory, a processor can perform many calculations. This imbalance, the “memory wall,” reflects the latency, bandwidth, and energy costs of moving data across a system. It explains why specialized accelerators are not merely faster at math but are architected to hide, amortize, and reduce the cost of moving data through deep memory hierarchies, massive parallelism, and specialized data paths. Hardware acceleration allows many large AI workloads to grow when general-purpose processor scaling alone is insufficient. It also explains why some optimizations that reduce theoretical computation fail to improve runtime. If an operation is memory bound, computing less may not help because computation is not the bottleneck. Hardware selection therefore cannot be reduced to comparing peak FLOP/s. What matters is whether a workload’s data movement patterns align with what the hardware was designed to accelerate. A model with large embedding tables and irregular lookups needs a different accelerator than one performing dense matrix multiplications over compact weight tensors. Software must expose and schedule the available reuse; otherwise specialized units wait while data moves and headline throughput remains theoretical. Matching workload and hardware determines whether execution remains at a fraction of theoretical peak or approaches the hardware’s practical ceiling. In D·A·M terms, the accelerator makes the machine constraint concrete and shapes which algorithms remain practical in production.

Learning Objectives

Explain hardware acceleration as machine-axis specialization for tensor workloads, data reuse, and performance per watt
Calculate arithmetic intensity and roofline ceilings to classify kernels as compute bound or memory bound
Diagnose memory-wall bottlenecks using bandwidth, cache hierarchy, host-device transfer, and energy-movement costs
Compare Tensor Cores, systolic arrays, SIMD/SIMT units, and sparse execution for ML compute primitives
Select dataflow, tiling, and mapping strategies that maximize reuse under memory-capacity constraints
Analyze compiler and runtime optimizations that fuse kernels, plan memory, and schedule accelerators
Evaluate accelerator choices across throughput, latency, power, cost, and deployment-context constraints

Hardware acceleration turns on the machine axis.

Reducing parameters, precision, or operations only matters when the machine can execute the resulting representation efficiently. Data selection reduced the data term, and compression reduced the algorithm’s work; hardware acceleration asks what the machine can actually deliver. The answer starts with the memory wall: arithmetic is often cheap relative to data movement. An off-chip memory access can take many processor cycles and consume orders of magnitude more energy than a low-precision arithmetic operation, with the exact ratio depending on the device and memory level. Specialized hardware matters because it raises compute throughput while organizing memory, dataflow, and parallelism so those arithmetic units stay fed.

Definition 1.1: Hardware acceleration

Hardware acceleration is the practice of mapping work from general-purpose execution onto specialized processors or functional units optimized for a narrower operation class. For ML workloads, that specialization commonly trades some programmability for the compute density $(R_{\text{peak}})$ and performance-per-watt gains that regular tensor operations can exploit.

Significance: An A100 GPU delivers 312 TFLOP/s for FP16/BF16 matrix multiplication; against this chapter’s illustrative 1–2 TFLOP/s reference-CPU baseline, this is a 156–312× gap. Its 54.2 billion transistors devote far more silicon to parallel arithmetic than to branch prediction, out-of-order scheduling, and large caches (NVIDIA Corporation 2020a; Choquette et al. 2021).
Distinction: A general-purpose CPU is designed for flexible execution, low-latency control, and varied workloads. An accelerator instead concentrates resources on a narrower operation class, so it achieves its gains only when the workload exposes the supported operations and enough parallel work to keep its arithmetic units busy.
Common pitfall: A frequent misconception is that an accelerator’s advertised peak throughput is the throughput a workload receives. Delivered performance is the lower of the compute ceiling and what memory bandwidth can feed, the roofline constraint: a low-arithmetic-intensity kernel can sit at a small fraction of peak FLOP/s no matter how fast the silicon is rated, because it starves for data rather than for arithmetic.

This definition frames the chapter’s central engineering trade-off. General-purpose processors allocate silicon area to complex control logic, speculative execution, branch prediction, and multi-level cache hierarchies to handle arbitrary control flow. An accelerator redirects that transistor budget toward dense arithmetic arrays, dedicated register files, and high-bandwidth memory paths tailored to a narrower workload class. When the software path exposes sufficient tensor parallelism, this specialization yields order-of-magnitude improvements in energy efficiency and throughput per watt.

Hardware alone cannot achieve these gains. Algorithms must structure operations to expose data reuse and regular memory access, while silicon architectures must allocate physical units to the mathematical primitives and numeric precisions that algorithms actually execute. This symbiosis motivates a complementary principle: hardware-software co-design.

Definition 1.2: Hardware-software co-design

Hardware-Software Co-design is an ML accelerator development method that crosses conventional abstraction boundaries, allowing algorithm constraints to inform silicon design and hardware capabilities to shape algorithm formulation.

Significance: On a supported tensor-core path, 8-bit integer (INT8) quantization can deliver multi-fold throughput gains because hardware executes lower-precision operations more densely and moves fewer bytes than an FP32 path; the exact gain depends on the architecture and workload (NVIDIA Corporation 2020a; Dally et al. 2021; Dally 2023).
Distinction: Unlike layered abstraction (where software calls a hardware API without knowing the silicon details), co-design exposes hardware constraints directly to algorithm and compiler authors: data alignment requirements, precision formats, and memory access patterns all become visible inputs to global cross-layer optimization.
Common pitfall: A frequent misconception is that co-design is a one-time hardware design choice. In practice, the supported features continue to evolve: NVIDIA Tensor Cores began with FP16 matrix multiplication, later generations added TF32 and broader integer support, and Ampere added acceleration for 2:4 structured sparsity (NVIDIA 2017; NVIDIA Corporation 2020a).

Co-design explains why the compression techniques introduced in Model Compression deliver real speedups. The quantization techniques in Quantization and Precision show why converting FP32 to INT8 accelerates execution on supported hardware: reduced precision cuts memory traffic, while specialized arithmetic units execute more low-precision operations per cycle than FP32 equivalents (NVIDIA Corporation 2020a). Structured pruning improves performance when it produces smaller dense shapes or a sparsity pattern recognized by silicon, whereas arbitrary unstructured sparsity incurs metadata handling and non-coalesced memory accesses that often erase theoretical gains. Evaluating accelerator performance requires tracing this path from workload to silicon: compute primitives, memory hierarchies, roofline diagnosis, mapping and dataflow, and compiler scheduling. The recurring question is why algorithmic improvements frequently fail to deliver proportional end-to-end speedups. Amdahl’s law provides the governing limit.

Theorem 1.1: The fundamental limit of acceleration (Amdahl's Law)

Hardware acceleration does not speed up the entire system; it speeds only the baseline-time fraction using the accelerated path. For that fraction $p$ and path gain $G_{\text{accel}}$, Amdahl’s law for AI (Amdahl 1967) gives the bound in equation 1. \[ \text{Speedup} = \frac{1}{(1 - p) + \frac{p}{G_{\text{accel}}}} \tag{1}\]

Accelerated fraction ($p$): Share of baseline time receiving the stated gain, often in matrix multiplication or another supported kernel.
Accelerator gain ($G_{\text{accel}}$): The raw speed advantage of the GPU or Tensor Processing Unit (TPU) over the CPU for the accelerated portion of the workload.
Unaccelerated fraction ($1-p$): Work outside the chosen path, such as data loading, host overhead, unsupported operators, and kernel launch latency.

Pitfall: Unaccelerated work caps total speedup. If 10 percent of baseline time is unaffected ($p=0.9$), even infinite path gain ($G_{\text{accel}}=\infty$) yields at most 10$\times$ overall. Once the accelerated component is fast enough, unaffected work dominates.

Amdahl, Gene M. 1967. “Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities.” Proceedings of the April 18-20, 1967, Spring Joint Computer Conference on - AFIPS ’67 (Spring), AFIPS ’67 (spring), 483–85. https://doi.org/10.1145/1465482.1465560.

¹ Amdahl’s law: Maps onto the iron law only after the proposed change is specified. If a faster tensor unit reduces the computation term $(O/(R_{\text{peak}} \cdot \eta_{\text{hw}}))$ but leaves data transfer and fixed latency unchanged, those unaffected terms bound total speedup. A different accelerator or memory system may improve bandwidth as well. The unaccelerated fraction is therefore defined by the intervention, not by a permanent classification of each term as serial.

Hardware acceleration targets specific terms in the iron law of ML systems (Iron Law of ML Systems), which decomposes end-to-end time into data volume $(D_{\text{vol}}/\text{BW})$, computation $(O/(R_{\text{peak}} \cdot \eta_{\text{hw}}))$, and fixed latency $(L_{\text{lat}})$. While data selection reduced total data volume and model compression reduced operations $O$ per sample, hardware acceleration increases the rate of execution by improving $R_{\text{peak}}$, $\eta_{\text{hw}}$, and $\text{BW}$. Physics of Computing supplies the analytical performance models that diagnose which of these terms dominates a given workload, including the dimensional analysis that confirms each iron law term resolves to seconds. Yet acceleration encounters a hard ceiling established by Amdahl’s law.¹

This dynamic explains why hardware upgrades frequently fail to produce proportional end-to-end speedups. Figure 1 visualizes this acceleration wall: as long as an unaccelerated fraction $1-p$ remains, the asymptotic ceiling $1/(1-p)$ enforces sharp diminishing returns on path gain $G_{\text{accel}}$ unless $p$ approaches unity.

Figure 1: **The Iron Law Heatmap**: Total system speedup as a function of accelerator gain ($G_{\text{accel}}$) and accelerated fraction ($p$). High speedup appears only near the top-right corner, where both accelerator gain and accelerated fraction are high. At $p=0.9$, for example, even an infinite gain on the accelerated path cannot exceed 10$\times$ end-to-end speedup. Contours span roughly 1$\times$–200$\times$ speedup.

Raw accelerator speedup translates into system-level throughput only after the unaccelerated fraction has been minimized. The accelerated fraction $p$ differs substantially across workload archetypes running on identical hardware, determining whether an accelerator investment pays off or stalls on non-matrix operations.

Checkpoint 1.1: The parallelism gate

These checks turn Amdahl’s bound into a hardware-selection test.

Amdahl’s reality

Serial bottlenecks: use Amdahl’s bound to explain why a 1,000$\times$ faster GPU may only speed up training by 5$\times$ when data loading is slow.
Speedup ceiling: for $p=0.95$ and $G_{\text{accel}}=100$, estimate total speedup and compare it with the infinite-accelerator ceiling.
Workload variation: compare ResNet-50 and MobileNet accelerated fractions for a proposed hardware change, then predict which benefits more.

The serial bottleneck becomes concrete on real hardware, where the same accelerator that nears its parallel ceiling on one workload can stall on another. Numbers to Know collects the reference $R_{\text{peak}}$ figures across accelerator generations and the latency hierarchy that ground these hardware comparisons in order-of-magnitude terms. Evaluating an NVIDIA H100 running compute-bound convolution versus memory-bound autoregressive decoding illustrates how differing unaccelerated fractions govern realized speedup.

Lighthouse 1.1: An illustrative Amdahl budget on H100

Illustrative ResNet-50 inference on NVIDIA H100:

Assume H100 provides $G_{\text{accel}}$ = 247× on matrix multiplication relative to a reference CPU without matrix extensions (1979 TOPS INT8 vs. ~8 TOPS) (Choquette 2023).
Assume $p$ = 0.95: 95 percent of baseline time receives that gain, while 5 percent remains in data loading, preprocessing, postprocessing, and other work. \[ \text{$\text{Speedup} = \frac{1}{(1-0.95) + \frac{0.95}{247}} = \frac{1}{0.05 + 0.0038} \approx 18.6\times$} \] The 247× path gain produces only 18.6× end-to-end speedup because the 5 percent unaffected fraction sets the ceiling.

Contrast with GPT-2 (autoregressive):

For GPT-2 token generation, assume $p$ = 0.80; the remaining 20 percent includes host overhead, sampling, and decoding work outside the accelerated matrix path. \[ \text{$\text{Speedup} = \frac{1}{(1-0.80) + \frac{0.80}{247}} = \frac{1}{0.20 + 0.0032} \approx 4.9\times$} \] Less token-generation time receives the assumed gain, so even an infinite path gain yields at most $1/(1-p)$ = 5×. Serving optimizations address this limit differently: batching improves cross-request utilization, while speculative decoding uses a draft model to reduce sequential target-model steps.

² Arithmetic intensity: The ratio of compute operations to bytes transferred across the memory interface being modeled (FLOP/byte). Workloads above that hardware level’s ridge point, such as well-tiled matrix multiplications with substantial reuse, are compute bound and can benefit from more TFLOP/s. Low-intensity operations, including some low-batch attention and decoding kernels, are bandwidth bound at that level and need more bandwidth or reuse rather than only more compute throughput.

Whether an operation is limited by compute rate or by data movement dictates both hardware selection and software optimization. Upgrading to an accelerator with $10\times$ higher peak compute yields negligible speedup if data delivery is the binding bottleneck. The Roofline Model provides the analytical framework for making this diagnosis (Williams et al. 2009); it is introduced formally in The Roofline model and section 1.5 applies it to AI workloads. It plots an operation’s arithmetic intensity,² defined as the ratio of floating-point operations to bytes of memory traffic (FLOP/byte), against hardware capabilities, revealing whether performance is capped by compute or bandwidth. A dense matrix multiplication with high arithmetic intensity benefits from additional TFLOP/s; a LayerNorm with low arithmetic intensity benefits only from greater memory bandwidth. High-reuse ResNet-50 convolutions operate in the compute-bound regime, while low-batch autoregressive attention operates in the memory-bound regime. Consequently, compute-bound workloads require strategies that maximize arithmetic throughput and functional-unit utilization, whereas memory-bound kernels require operator fusion and memory-hierarchy staging to minimize off-chip traffic.

Architectural design begins with the physical constraints that forced specialization: the breakdown of classical Dennard voltage scaling and the prohibitive energy cost of moving data across memory hierarchies. When general-purpose microarchitectures encountered these thermal and memory barriers, domain-specific hardware specialization emerged as the primary engineering path to sustain performance scaling.

Hardware Specialization

When Google deployed its first Tensor Processing Unit (TPUv1) into production data centers alongside NVIDIA K80 GPUs, the comparison highlighted a fundamental efficiency divide. General-purpose processors dedicate substantial silicon area and power budgets to instruction fetching, branch prediction, and cache coherence hardware designed for arbitrary code. For deep learning workloads dominated by regular, dense linear algebra, that control overhead yields poor arithmetic density. When an algorithmic pattern becomes sufficiently dominant and structured, specialized hardware outperforms general-purpose platforms by replacing dynamic control machinery with dedicated arithmetic arrays and software-managed data movement. Google’s production inference evaluation quantified this efficiency gap (example 1.1).

Example 1.1: The TPUv1 vs. K80 efficiency shock

Scenario: In 2015, Google deployed its first Tensor Processing Unit (TPUv1) into production data centers, comparing its throughput and energy efficiency against general-purpose NVIDIA K80 GPUs (Jouppi et al. 2017).

Diagnosis: The K80 was a programmable GPU designed to support a broader mix of graphics and compute workloads. The TPUv1 domain-specific architecture (DSA) made a narrower inference-focused bet, pairing a large 8-bit systolic matrix unit with software-managed memory and simpler control.

Systems lesson: On the six Google inference workloads and contemporary CPU/GPU baselines evaluated in the TPU paper, tailoring silicon to the dominant matrix operations delivered 15–30$\times$ higher throughput and 30–80$\times$ better performance per watt (Jouppi et al. 2017). Those ratios describe the evaluated systems, not every workload.

In D·A·M terms, this efficiency gain stems from tailoring machine resources to algorithmic structure. By replacing hardware caches with software-managed scratchpad buffers and routing data directly between neighboring arithmetic units in a systolic array, specialized architectures maximize arithmetic intensity and minimize off-chip memory traffic. Yet this specialization imposes an architectural trade-off: rigid execution pipelines forfeit flexibility. When workloads introduce irregular memory access, dynamic control flow, or unsupported numeric precisions, fixed-function data paths stall, leaving expensive silicon idle.

Modern ML accelerators—including DianNao-class neural-network accelerators (Chen et al. 2014), GPUs with Tensor Cores, Google’s TPUs,³ and Apple’s Neural Engine—reflect this continuous balancing act between arithmetic specialization and programmable flexibility. Their designs emerge from a historical trajectory spanning four distinct phases: specialized computing origins, parallel graphics processing, domain-specific architectures, and the emergence of ML-specific hardware. Each phase addressed the specific resource bottleneck that constrained scaling in its era.

Chen, Tianshi, Zidong Du, Ninghui Sun, Jia Wang, Chengyong Wu, Yunji Chen, and Olivier Temam. 2014. “DianNao: A Small-Footprint High-Throughput Accelerator for Ubiquitous Machine-Learning.” Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’14), 269–84. https://doi.org/10.1145/2541940.2541967.

³ TPU (tensor processing unit): The first TPU made a deliberately narrow bet, centering the design on a single $256{\times}256$ systolic array for 8-bit matrix multiplication and avoiding much of the control machinery required by a general-purpose core (Jouppi et al. 2017). That trade buys high compute density on dense matrix multiplication at the cost of flexibility. Irregular or branch-heavy code remains a poor fit, and compiler tiling, padding, and layer dimensions determine how fully the array is used.

Specialized computing

Hardware specialization emerges when specific computational patterns become the primary system bottleneck, preventing general-purpose processors from scaling efficiently. This history highlights three recurring limits: slow implementation of scalar floating-point, insufficient throughput for parallel graphics, and poor integration between memory and parallel compute.

The first phase, the precision bottleneck, occurred when scientific and engineering applications required floating-point arithmetic on microprocessors with integer-only arithmetic logic units. In the late 1970s, CPUs emulated floating-point operations through multi-instruction software routines that unpacked sign, exponent, and mantissa fields, shifted bits to align binary points, and normalized the output. This software emulation imposed severe scalar instruction overhead for even a single multiplication, motivating the first major wave of hardware specialization: the mathematics coprocessor.

The Intel 8087 (1980)⁴ addressed this bottleneck by executing floating-point instructions directly on dedicated hardware pipelines (Palmer 1980). Replacing multi-instruction emulation routines with hardwired arithmetic logic established a core systems principle: when an inner loop is dominated by a specific operation, specialized silicon eliminates general-purpose instruction overhead and shifts the execution ceiling.

⁴ Intel 8087: The coprocessor implemented floating-point logic directly in silicon, avoiding the CPU’s slow, multi-instruction software emulation for each calculation. Its benefit depended on how much time an application spent in supported floating-point operations, illustrating why specialization helps most when the accelerated work dominates execution.

As specialized functions demonstrated their utility, they followed a recurring pattern of integration. The Intel 486DX (1989) moved the FPU directly onto the CPU die, eliminating off-chip bus communication latency and making high-precision math a standard feature rather than an optional coprocessor (Hennessy and Patterson 2017). This cycle—specialization to resolve an acute bottleneck followed by architectural integration—recurs across hardware eras.

Figure 2 traces this progression across five eras, illustrating how recurring computational bottlenecks drove hardware evolution from floating-point coprocessors and digital signal processors to programmable GPUs, streaming media codecs, and deep-learning tensor accelerators. Modern machine learning workloads depend directly on principles established across these eras: dedicating silicon to dominant matrix operations, staging operands in local memories, and minimizing off-chip data movement.

Figure 2: **Hardware Specialization Timeline**: Computing architectures evolve through recurring cycles of specialization and integration across five eras. Each era introduces domain-specific hardware to resolve an important memory or arithmetic bottleneck—from off-chip FPUs to GPUs, codecs, and TPU/Tensor Core array units. Many of these capabilities later appear alongside CPUs in broader systems, even when the specialized units remain distinct.

Parallel computing and graphics processing

General-purpose CPUs allocate most of their silicon budget to control logic, branch prediction, and multi-level caches designed to minimize execution latency for a single instruction stream. When a workload exhibits massive, uniform data parallelism—applying identical arithmetic across millions of independent elements—this latency-oriented architecture expends chip area and thermal budget on control structures that do not increase computational throughput. Specialized accelerators resolve this imbalance by reallocating silicon from cache and control to dense arrays of arithmetic logic units (ALUs), prioritizing aggregate throughput over single-thread latency.

Real-time 3D graphics drove this throughput-oriented specialization throughout the 1990s. Early graphics accelerators offloaded fixed rasterization tasks such as bitmap transfers and polygon filling. In 1999, NVIDIA’s GeForce 256 integrated hardware-accelerated transform and lighting (T&L), shifting coordinate transformations from the host CPU to dedicated on-chip silicon. While not yet programmable, these initial Graphics Processing Units (GPUs) demonstrated how specialized pipelines could accelerate fixed mathematical operations like vertex transformation and texture mapping. However, rigid fixed-function pipelines caused severe load imbalances: complex geometries left rasterization hardware idle, while dense fragment shading starved vertex pipelines. The transition to programmable shaders in the GeForce 3 (2001) and unified shader architectures in the GeForce 8 (2006) resolved this imbalance by replacing segregated functional units with arrays of identical, programmable compute cores. This structural unification decoupled the hardware from fixed graphics stages and established the GPU as a general-purpose parallel processor capable of dense matrix arithmetic. By 2004, high-end GPUs could process over 100 million polygons per second (Owens et al. 2008).

Lyons, Richard G. 2011. Understanding Digital Signal Processing. 3rd ed. Prentice Hall.

In parallel with graphics hardware, Digital Signal Processing (DSP) processors addressed a different physical constraint: deterministic, low-latency execution within tight energy budgets. General-purpose cache hierarchies introduce variable latency whenever cache misses occur, violating strict real-time deadlines in signal processing. DSPs replaced unpredictable caches with software-managed scratchpad memories and paired specialized multiply-accumulate (MAC) units with circular addressing buffers to execute continuous filtering and Fourier transforms without memory-stall jitter. Texas Instruments’ TMS32010 (1983) demonstrated how domain-specific instruction sets and dedicated MAC data paths could outpace general-purpose microprocessors on vector operations (Lyons 2011).

Network processing confronted an even tighter latency deadline: sustaining packet inspection at line rate. When packets arrive every few nanoseconds, an off-chip memory access stall drops packets. Intel’s IXP2800 network processor shows the consequence of this hard constraint: meeting line-rate packet deadlines leaves no slack for cache misses, so the design arranges many parallel cores around tiered on-chip memory to keep data adjacent to compute. That compute-near-memory organization, forced by packet arrival deadlines, prefigured the memory hierarchies that modern machine learning accelerators use to keep processing-element arrays continuously fed with tensor tiles.

These domains diverged in implementation because their computational profiles differed. DSPs optimized for low-latency 1D signal streams under milliwatt thermal envelopes, while network processors optimized for packet routing with minimal state retention. In contrast, deep learning workloads are dominated by dense matrix multiplications and multidimensional convolutions. These operations exhibit high arithmetic intensity and massive data parallelism. The unified GPU provided the ideal machine match: thousands of parallel ALU lanes backed by wide memory buses, using hardware-managed multithreading to hide off-chip memory latency during dense tensor operations.

The practical impact of this architectural alignment became clear in 2012, when AlexNet⁵ (Krizhevsky et al. 2012) won the ImageNet competition by more than ten percentage points. Training was performed on two consumer-grade NVIDIA GTX 580 graphics cards, each with only 3 GB of VRAM. The underlying algorithm—convolutional networks trained with gradient descent—had existed for decades, but CPU compute throughput made large-scale training computationally impractical. By mapping dense 2D convolutions directly to GPU execution units, the authors compressed months of CPU computation into five to six days. In D·A·M terms, the machine constraint had previously dictated which algorithms were feasible; aligning the parallel structure of convolutions with throughput-oriented GPU hardware made deep models practical to train on large datasets.

⁵ AlexNet: Krizhevsky, Sutskever, and Hinton’s 60-million-parameter convolutional neural network (CNN) won ImageNet 2012 by more than ten percentage points on two consumer GTX 580 GPUs with only 3 GB of VRAM each. Because the model exceeded single-GPU memory, the authors partitioned the network across two cards and limited communication between them, an early form of model parallelism that foreshadowed systematic tensor and pipeline parallelism. The reported training run took five to six days (Krizhevsky et al. 2012).

Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. 2012. “ImageNet Classification with Deep Convolutional Neural Networks.” Advances in Neural Information Processing Systems (NeurIPS) 25.

Emergence of domain-specific architectures

These diverse acceleration patterns converged in a broader architectural shift. The emergence of domain-specific architectures (DSAs)⁶ marks a transition in computer system design, driven by the slowing benefits of traditional scaling (Esmaeilzadeh et al. 2011) and the increasing computational demands of specialized workloads. Moore’s Law⁷ described a long-running increase in transistor density (Moore 1998). Dennard scaling⁸ (Dennard et al. 1974) allowed voltage to fall as transistors shrank; its breakdown removed the accompanying path to routine frequency gains at constant power density. Together, these shifts constrained general-purpose performance and efficiency. As Hennessy and Patterson (2019) argued in the 2017 Turing Lecture, the response was a new era of domain-specific solutions optimized for important workloads.

⁶ DSA (domain-specific architecture): Hardware optimized for an application domain, sacrificing some general-purpose programmability for efficiency. Google’s TPUv1 achieved 15–30$\times$ better performance and 30–80$\times$ better performance per watt than the contemporary CPU and GPU systems evaluated on Google’s inference workloads by centering its design on a systolic array and software-managed memory (Jouppi et al. 2017). A DSA that excels at dense matrix multiplication may perform worse than a CPU on irregular workloads such as graph traversal, so the workload limits and efficiency gain must justify the software, compiler, and ecosystem cost (Hennessy and Patterson 2019, 2017).

Hennessy, John L., and David A. Patterson. 2019. “A New Golden Age for Computer Architecture.” Communications of the ACM 62 (2): 48–60. https://doi.org/10.1145/3282307.

Hennessy, John L., and David A. Patterson. 2017. Computer Architecture: A Quantitative Approach. 6th ed. Morgan Kaufmann.

⁷ Moore’s law: In the illustrative assumptions used by figure 3, model compute demand grows roughly 6.1× per year while accelerator peak supply improves roughly 1.7× per year, widening the modeled demand/supply gap by about 3.5× per year. Published compute trends motivate the divergence, but the plotted rates are assumptions rather than forecasts. The figure therefore shows how sensitive the systems gap is to sustained differences in growth rates, not a fixed future trajectory. Under such a gap, algorithmic efficiency techniques such as compression, quantization, and sparsity become increasingly valuable (Amodei and Hernandez 2018; Epoch AI 2024).

Moore, G. E. 1998. “Cramming More Components onto Integrated Circuits.” Proceedings of the IEEE 86 (1): 82–85. https://doi.org/10.1109/jproc.1998.658762.

⁸ Dennard scaling: The 1974 scaling model held that operating voltage could fall as transistor dimensions shrank, keeping power density approximately constant (Dennard et al. 1974). Its breakdown after about 2005 constrained further clock-frequency growth within a chip’s thermal design power; the resulting dark silicon literature estimated, for modeled advanced nodes and design assumptions, that thermal limits could prevent large fractions of available transistors from operating simultaneously. That fraction is not a universal constant, but the power constraint is durable: specialized units make the active silicon budget more useful by dedicating it to high-value operation classes (Esmaeilzadeh et al. 2011).

Dennard, Robert H., Frank H. Gaensslen, Hwa-Nien Yu, Victor L. Rideout, Elias Bassous, and Antoine R. LeBlanc. 1974. “Design of Ion-Implanted MOSFET’s with Very Small Physical Dimensions.” IEEE J. Solid-State Circuits 9 (5): 256–68. https://doi.org/10.1109/jssc.1974.1050511.

Esmaeilzadeh, Hadi, Emily Blem, Renee St. Amant, Karthikeyan Sankaralingam, and Doug Burger. 2011. “Dark Silicon and the End of Multicore Scaling.” Proceedings of the 38th Annual International Symposium on Computer Architecture, 365–76. https://doi.org/10.1145/2000064.2000108.

⁹ Huang’s law: The observation that GPU performance for AI workloads has improved through architectural innovations such as Tensor Cores as well as transistor scaling. The normalized figure uses illustrative GPU-supply and model-demand curves to show how unequal growth rates create a widening systems gap; it is not a forecast of either rate (Amodei and Hernandez 2018; Epoch AI 2024; NVIDIA Corporation 2020a; Choquette 2023).

Amodei, Dario, and Danny Hernandez. 2018. “AI and Compute.” OpenAI Blog 2.

Epoch AI. 2024. Machine Learning Trends. Epoch AI Research Database.

The scale of this challenge becomes stark in figure 3, which plots an illustrative systems gap between model demand and hardware supply, sometimes discussed under the informal label Huang’s Law.⁹ The normalized scenario assumes GPU supply rises about 1.7$\times$ per year while model demand rises about 6$\times$ per year, producing a gap that widens by roughly 3–4$\times$ each year. Published compute trends motivate the qualitative divergence, while the plotted rates are scenario assumptions rather than a forecast (Amodei and Hernandez 2018; Epoch AI 2024).

Figure 3: **The Systems Gap**: Relative compute growth (log scale) comparing model demand to hardware supply, normalized to 2012 = 1.0. The gray dotted line (CPU) and blue dashed line (GPU) reflect hardware progress, which lags the exponential red solid line (Model Demand). The purple region is the ‘Systems Gap’ that must be bridged through parallelism and co-design.

The plot is normalized to a 2012 baseline to emphasize relative growth. The purple-shaded region highlights this divergence: the systems gap cannot be closed by process node advances alone, but requires architectural innovation, parallel distribution, and hardware-software co-design.

The technology S-curve: Why architectures shift

Many computing technologies are described by a lifecycle of ferment (initial slow progress), take-off (rapid growth), and saturation (diminishing returns near physical or economic limits). Figure 4 uses two conceptual S-curves to express the pattern: as gains from one design approach taper, domain-specific architectures can open a new efficiency curve for workloads with stable computational structure.

Figure 4: **The Twin S-Curves of Specialized Computing**: This conceptual diagram represents a general-purpose design approach (gray) reaching diminishing returns as a specialized approach (blue) begins a new efficiency curve. It is a schematic of an architectural transition, not a measured historical performance series. Specializing hardware for stable linear-algebra workloads can produce large gains at the cost of general programmability.

The routine gains once associated with shrinking transistors have slowed, while AI model compute demand has expanded faster than general-purpose hardware capacity. Closing this gap cannot be accomplished by waiting for successive CPU process shrinks; it requires architectural changes that restructure how workloads use silicon area, memory bandwidth, and thermal budgets.

During the growth phase of the general-purpose S-curve, semiconductor scaling delivered predictable performance gains to existing software. Under classical Dennard scaling, shrinking transistor dimensions permitted operating voltages to scale down proportionally, keeping power density constant while clock frequencies increased. When Dennard scaling broke down in the mid-2000s, subthreshold leakage currents prevented further voltage reductions, forcing clock frequencies to stall against a thermal dissipation wall. Architects initially responded by adding processor cores, but multicore expansion encountered two physical boundaries: Amdahl’s Law and dark silicon—the thermal constraint where only a fraction of a chip’s transistors can switch simultaneously at full frequency. Furthermore, general-purpose CPUs expend the vast majority of their energy budget on control overhead—instruction fetch and decode, branch prediction, speculative out-of-order execution, and cache coherence—rather than arithmetic. For regular linear algebra workloads, executing an instruction on a CPU core can consume hundreds of picojoules of control energy to perform a 3-picojoule floating-point operation.

Domain-specific architectures shift to a new efficiency curve by eliminating this control overhead and dedicating silicon to matching data paths. The primary architectural innovation is a customized data path: matrix multiplication units in AI accelerators, for example, implement systolic arrays, grid-like networks of processing elements that compute and forward data directly to neighboring units. Instead of continually writing intermediate results back to register files or caches, a systolic array streams activations and weights across adjacent cells, enabling a value fetched once from on-chip memory to be reused across an entire row or column of multiply-accumulate operations. Once that data path is established, the memory hierarchy can be tailored directly to workload reuse patterns, replacing hardware-managed caches with software-managed scratchpads, prefetching logic, and memory controllers designed for deterministic tensor streams.

Domain-specific instruction sets reinforce this physical specialization. Rather than fetching and decoding scalar operations individually, domain-specific instruction sets encode coarse-grained tensor primitives into single macro-instructions that configure thousands of arithmetic units concurrently, amortizing decode and dispatch overhead. Fixed-function circuit blocks bypass software execution entirely for invariant operations such as activation functions and normalization layers. The resulting efficiency stems from unified hardware specialization across every layer: data movement, memory locality, instruction overhead, and circuit implementation all align around identical computational invariants.

Mobile system-on-chip (SoC) architectures demonstrate this efficiency trade-off directly. Modern smartphones decode high-resolution video within milliwatt thermal envelopes, despite video processing requiring billions of operations per second. This efficiency is achieved through dedicated hardware video codecs¹⁰ implementing industry standards such as H.264/AVC and H.265/HEVC (Sullivan et al. 2012). These specialized circuits provide order-of-magnitude performance-per-watt gains compared with software decoding on general-purpose CPU cores, with the exact gain depending on codec profile, resolution, process node, and CPU baseline.

¹⁰ Codec: A portmanteau of “coder-decoder,” reflecting the hardware’s dual function. Encoding (compression) is compute-intensive because it searches for optimal representations, while decoding (decompression) is bandwidth-intensive because it reconstructs full-resolution frames from compressed streams. Dedicated codec silicon implements both paths in fixed-function hardware, so neither path wastes transistors on unrelated general-purpose control logic.

Sullivan, Gary J., Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. 2012. “Overview of the High Efficiency Video Coding (HEVC) Standard.” IEEE Transactions on Circuits and Systems for Video Technology 22 (12): 1649–68. https://doi.org/10.1109/tcsvt.2012.2221191.

Turakhia, Yatish, Gill Bejerano, and William J. Dally. 2018. “Darwin: A Genomics Co-Processor Provides up to 15,000X Acceleration on Long Read Assembly.” Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 199–213. https://doi.org/10.1145/3173162.3173193.

¹¹ ASIC (application-specific integrated circuit): These circuits achieve their extreme efficiency by implementing a single algorithm directly in silicon, often improving performance-per-watt by $10^3\times$ to $10^5\times$. Examples include cryptographic hashing for blockchain mining and sequence alignment for genomics. The trade-off is total inflexibility: if that core algorithm changes, the ASIC cannot be reprogrammed and becomes obsolete, locking the hardware design to the specific problem version it was built to solve.

Bedford Taylor, Michael. 2017. “The Evolution of Bitcoin Hardware.” Computer 50 (9): 58–66. https://doi.org/10.1109/mc.2017.3571056.

This architectural shift recurs across data-intensive computing domains whenever a workload’s computational kernel stabilizes. Genomics processing uses custom accelerators because long-read assembly contains fixed dynamic programming alignment kernels that dedicated hardware pipelines directly in silicon (Turakhia et al. 2018). Cryptographic workloads produced application-specific integrated circuits (ASICs)¹¹ for the same reason: fixed hashing algorithms justify dedicated silicon pipelines that trade all programmable flexibility for maximum energy efficiency (Bedford Taylor 2017).

These transitions underscore the central principle of modern systems engineering: the era of automatic performance gains from general-purpose scaling is over. For decades, software developers could rely on Moore’s Law to accelerate existing software without modifying code structures. With the breakdown of Dennard scaling, computational bottlenecks can no longer be resolved by waiting for faster general-purpose cores; they require designing the machine to match the algorithm. In D·A·M terms, the accelerator makes the physical machine constraint explicit. Performance is governed by hardware-software co-design: execution efficiency is determined by how effectively an algorithm’s tensor dimensions, memory access patterns, and parallelism map to the specialized physical structures of domain-specific architectures.

Machine learning hardware specialization

Many important neural-network workloads contain regular tensor operations, substantial data parallelism, and opportunities for reduced precision. Dense matrix multiplications and convolutions are prominent examples because their loop structure exposes repeated operations and predictable reuse. These characteristics have driven specialized hardware architectures that would be ineffective for arbitrary code but can provide substantial gains on matching ML kernels. Other ML operations remain irregular, bandwidth bound, sequential, or too small to fill a wide engine, so the same device need not accelerate every part of a model. The hardware built to exploit the regular portion constitutes a class of devices known as ML accelerators, and the economic trigger for specialization appears when that supported workload is important at fleet scale rather than only in a benchmark.

Example 1.2: The TPU capacity cliff

Scenario: Google projected in 2013 that user adoption of voice-search speech recognition (3 minutes/day/user) would require doubling global data center capacity to run CPU-based neural network inference (Jouppi et al. 2017).

Diagnosis: General-purpose CPU server capacity was economically unable to absorb the exponential growth in inference FLOPs, threatening a capital infrastructure bottleneck.

Systems lesson: Custom hardware acceleration becomes mandatory when workload volume crosses fleet-level economic thresholds. Aggregate arithmetic intensity and server total cost of ownership drive custom ASIC deployment decisions.

Machine learning computational requirements reveal limitations in traditional processors. In an illustrative scenario where a reference CPU sustains 5 percent–10 percent of its peak on a neural network workload, the upper endpoint delivers approximately 0 GFLOP/s (billions of floating-point operations per second). This inefficiency results from architectural mismatches: CPUs optimize for single-thread performance and irregular memory access, while neural networks require massive parallelism and predictable data streams. The memory bandwidth constraint compounds the problem: a single neural network layer may require accessing gigabytes of parameters, overwhelming CPU cache hierarchies designed for smaller working sets.

The energy economics of data movement influence accelerator design. Per event, accessing a 32-bit value from DRAM can consume on the order of $10^2\times$ more energy than one FP32 multiply (exact values vary by technology node and design), making data movement a primary optimization target (Horowitz 2014; Sze et al. 2017). This disparity helps explain the progression from repurposed graphics processors to purpose-built neural network accelerators. TPUs and other custom accelerators can sustain high utilization on dense kernels by implementing systolic arrays and other architectures that maximize data reuse while minimizing movement.

IEEE Standards Association. 2019. IEEE 754-2019: Standard for Floating-Point Arithmetic. https://doi.org/10.1109/IEEESTD.2019.8766229.

¹² Latency vs. throughput in accelerator design: Training commonly uses throughput-oriented execution and larger batches to amortize work, while latency-sensitive inference must control single-request and tail latency. This is a service objective, not a universal training/inference divide: batched inference can also be throughput oriented, and small or interactive training loops may care about step latency. Pipeline depth, batching delay, kernel setup, and queueing can make a training-oriented path less suitable for an interactive request even when its peak FLOP/s is higher, so the design must match the workload’s actual arrival pattern, batch opportunity, and objective.

Training and inference present different computational profiles that influence accelerator design. Training computes gradients and updates weights: FP32 and FP16 are standardized binary floating-point formats (IEEE Standards Association 2019), while mixed-precision training uses lower-precision tensor operations with higher-precision accumulation when accuracy permits (Micikevicius et al. 2017). Backpropagation also retains or recomputes activations (see Activation memory requirements), increasing memory demand. Inference requires only the forward path and may use INT8 or INT4 when the model and kernels support it. Interactive inference may prioritize tail latency, while offline or batched inference may prioritize throughput, cost, or energy; accelerator choice follows the actual service objective. These profiles often favor high-capacity, throughput-oriented training systems and energy-efficient inference paths, but batch size and service policy can reverse the apparent preference.¹²

Definition 1.3: ML accelerator

Machine Learning Accelerators are processors or specialized units designed primarily for common neural-network tensor operations and dataflows. They pursue high $R_{\text{peak}}$ and data reuse on matching workloads by devoting more resources to arithmetic and local data movement than a general-purpose execution path can.

Significance: An ML accelerator’s defining feature is not raw arithmetic alone but a balanced path that supplies operands and retains reusable values nearby. The A100 provides 2.04 TB/s of device-memory bandwidth, about 10.2× the illustrative 200 GB/s host-DRAM path used here (NVIDIA Corporation 2020a; Choquette et al. 2021). This is a ceiling comparison, not a guarantee: sufficient arithmetic intensity, concurrency, and reuse are still required to approach its 312 TFLOP/s FP16/BF16 peak.
Distinction: The gains are conditional on operation support, parallel work, regular data access, and a software stack that selects the intended path. An ML accelerator can be orders of magnitude faster than a reference CPU on large dense matrix multiplication, yet lose much of that advantage on small, irregular, synchronization-heavy, or control-heavy workloads. Transfers and unsupported operators can dominate even when the accelerated kernel itself is fast.
Common pitfall: A frequent misconception is that ML accelerators always accelerate ML. Approaching peak throughput requires a compatible precision, operation shape, layout, implementation, and enough parallel work to keep the relevant units busy. Utilization also depends on whether memory and interconnect paths can sustain that work. A batch-1 autoregressive request may therefore use only a small fraction of a large training accelerator’s arithmetic capacity even though the same device performs well on batched prefill or training.

Choquette, Jack, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. 2021. “NVIDIA A100 Tensor Core GPU: Performance and Innovation.” IEEE Micro 41 (2): 29–35. https://doi.org/10.1109/mm.2021.3061394.

A DRAM access costs over 100$\times$ one FP32 multiply per event.

Deployment context shapes architectural choices by identifying the binding constraint. In data centers, the constraint is often time-to-result for training massive models. An NVIDIA H100 trades hundreds of watts of power for high throughput (Choquette 2023); whether that trade lowers total cost depends on utilization, rental price, workload scaling, and the electricity rate. Google’s TPUv4 makes a similar architectural trade, prioritizing throughput through systolic arrays and high-bandwidth memory (Jouppi et al. 2023).

Checkpoint 1.2: The accelerator gate

Use the energy hierarchy to decide when specialization pays.

Energy inversion

Data movement cost: can you explain why moving data from DRAM costs 100$\times$ more energy than computing on it?
Architectural response: explain how systolic arrays (TPU) and Tensor Cores (GPU) reduce repeated movement of the same operands.

Selection logic

Training vs. inference: why do training chips need massive HBM bandwidth, while inference chips prioritize low latency and INT8 ops?

At the edge, the priority often shifts toward energy per inference and hard latency or power limits, although throughput remains important for continuous camera, audio, or sensor streams. A smartphone camera or always-on audio path operating inside a few-watt budget cannot simply adopt a data center accelerator’s high-power memory system. Edge architectures instead reduce movement through local scratchpads, tightly integrated accelerators, dynamic voltage and frequency scaling, and event-driven processing when the workload permits. The same memory-wall principle applies in both settings: data center chips invest in high-bandwidth memory (HBM) capacity and bandwidth, while edge chips depend heavily on proximity and reuse.

No single architecture dominates every ML workload. Edge devices favor energy efficiency and bounded latency, while cloud-scale training values throughput, capacity, and interconnect performance. Cloud inference can emphasize cost per request or tail latency, and an edge device may still need sustained throughput for a real-time stream. Specialized architectures therefore reflect their deployment context, yet all remain subject to the energy and latency costs of moving data.

Table 1 summarizes these milestones in hardware specialization. Floating-point coprocessors accelerated arithmetic previously implemented in software, early GPUs increased graphics throughput, and media engines showed how stable pipelines could justify fixed-function blocks. AI accelerators combine dense tensor units with memory hierarchies and software support, making integration between data movement and parallel execution a central challenge.

Modern AI acceleration therefore extends beyond the chip. Useful accelerators need framework, compiler, library, driver, and runtime support for graph transformations, kernel selection, fusion, memory scheduling, and deployment across environments from data centers to edge devices. Unsupported operations or surrounding transfers can erase a kernel-level gain.

Table 1: Hardware Specialization Trends: Successive computing eras progressively integrate specialized hardware to accelerate prevalent workloads, moving from general-purpose CPUs to domain-specific architectures and ultimately to customizable AI accelerators. Tailoring hardware to computational patterns improves performance and energy efficiency, driving innovation in machine learning systems.

Era	Computational pattern	Architecture examples	Characteristics
1980s	Floating-Point & Signal Processing	FPU, DSP	• Single-purpose engines • Focused instruction sets • Coprocessor interfaces
1990s	3D Graphics & Multimedia	GPU, SIMD Units	• Many identical compute units • Regular data patterns • Wide memory interfaces
2000s	Real-time Media Coding	Media Codecs, Network Processors	• Fixed-function pipelines • High throughput processing • Power-performance optimization
2010s	Deep Learning Tensor Operations	TPU, GPU Tensor Cores	• Matrix multiplication units • Massive parallelism • Memory bandwidth optimization
2020s	Application-Specific Acceleration	ML Engines, Smart NICs, Domain Accelerators	• Workload-specific datapaths • Customized memory hierarchies • Application-optimized designs

The integration bottleneck

The evolution from the Intel 8087 numeric coprocessor to modern graphics and AI accelerators reveals a consistent pattern: hardware evolves to fit the algorithm’s dominant bottleneck (Palmer 1980; Goodfellow et al. 2016; Sze et al. 2017; Jouppi et al. 2017). Where the 8087 offloaded floating-point arithmetic that overwhelmed general-purpose CPUs in scientific computing, modern AI accelerators target dense matrix multiplications and convolutions. In this domain, provisioning raw arithmetic units is straightforward compared to feeding them with data. Modern AI architectures must resolve an integration bottleneck: transporting operands and results through the memory hierarchy rapidly and efficiently enough to sustain thousands of parallel compute units. Specialized silicon delivers substantial performance-per-watt gains over general-purpose processors only when the workload’s computational demand concentrates into regular operations that align with physical hardware data paths.

¹³ Scratchpad memory: When a compiler can determine a tensor kernel’s access pattern, it can explicitly stage tiles in fast, software-controlled local memory. This avoids some tag, replacement, and coherence machinery, but transfers, synchronization, banking, and capacity must be scheduled correctly. Scratchpads complement rather than universally replace caches because irregular or shared data may still benefit from hardware management. Google’s TPU v1, for example, used a 24 MB software-managed Unified Buffer for intermediate activations, while weights and instructions used other paths (Jouppi et al. 2017).

Jouppi, Norman P., Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, et al. 2017. “In-Datacenter Performance Analysis of a Tensor Processing Unit.” Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA ’17, 1–12. https://doi.org/10.1145/3079856.3080246.

Three properties of neural-network kernels make this integration bottleneck tractable. First, large matrix multiplications and convolutions expose abundant data parallelism that dense processing-element arrays can exploit concurrently. Second, static or compiled tensor graphs exhibit predictable dataflow, enabling compilers to tile and stage transfers into local scratchpads¹³ rather than relying exclusively on hardware-managed caches. This grants software explicit control over regular working sets while retaining hardware caches for irregular access. Third, many tensor operations exhibit algorithmic tolerance to reduced precision, allowing supported 8-bit or 4-bit paths to increase compute density and reduce bytes per value (Dally et al. 2021; Dally 2023). Conversely, workloads dominated by dynamic control flow, large embedding tables, small tensors, variable shapes, or strict precision requirements violate these assumptions and remain bound by memory latency and irregular access.

Once arithmetic capacity is abundant, the central engineering challenge becomes keeping data close enough to use it. A DRAM access can consume more than 100$\times$ the energy of a low-precision arithmetic operation under the physical constraints analyzed by Horowitz (2014). This steep energy hierarchy dictates why accelerator architectures invest silicon area and power budgets into HBM, on-chip SRAM, and software-managed scratchpads rather than exclusively adding arithmetic logic units.¹⁴

¹⁴ HBM: The A100 and H100 generations provide 2.0–3.4 TB/s of device-memory bandwidth through stacked memory and wide interfaces, compared with 760 GB/s for the GDDR6X reference used here (NVIDIA Corporation 2020a; Choquette 2023). Raising the bandwidth ceiling can move some kernels toward the compute-bound side of the roofline, but only if their access pattern and arithmetic intensity can use it. HBM also increases packaging and system cost, so its value depends on the workload’s bandwidth demand.

Hardware acceleration resolves the integration bottleneck through specialized spatial layout and tiered storage hierarchies. As shown in the architectural blueprint in figure 5, high-bandwidth memory feeds shared on-chip storage, which in turn supplies local scratchpad memory surrounding a grid of execution units.

Figure 5: **Anatomy of a Modern AI Accelerator**: This generic schematic combines common components rather than depicting one product. A host interface connects the accelerator to a CPU, a high-bandwidth device-memory interface feeds shared on-chip storage, and processing elements contain matrix, vector, and special-function units with local memory. Real accelerators differ in which units they provide and whether storage is hardware-managed cache or software-managed scratchpad.

In the generic design in figure 5, an array of processing elements contains units specialized for distinct operation classes: matrix units execute dense matrix multiplication, vector units perform element-wise arithmetic, and special function units approximate transcendental operations such as exponentials and activations. The number, grouping, and vendor nomenclature of these units vary across architectures, but all aim to exploit data-level parallelism across tensor tiles. Some designs arrange units into an explicit 2D systolic grid; others distribute matrix engines among wide parallel compute cores. The schematic represents these functional responsibilities rather than a single vendor’s physical layout.

Complementing the execution units, the memory hierarchy amortizes the latency and energy cost of data transport across multiple tiers. High-bandwidth memory provides off-chip capacity and aggregate throughput, while shared on-chip storage and per-element caches or scratchpads reduce off-chip traffic. While specific implementations differ—some relying on hardware-managed caches, others exposing software-managed SRAM, and many combining both—the physical principle is constant: moving a tensor tile across package boundaries consumes orders of magnitude more energy than keeping it on chip. Compilers and runtime kernels must coordinate tile capacity, reuse, prefetching, and double buffering to keep operands staged in fast local memory. Expanding HBM bandwidth raises the off-chip throughput ceiling, but without temporal and spatial reuse in local SRAM, execution units stall waiting for operands. The machine foundations appendix collects reference specifications for modern accelerators, including H100 and TPU v5, and summarizes the latency hierarchy.

The host interface links the accelerator to the host CPU and broader system memory. A host CPU coordinates execution graphs, asynchronous I/O, and unsupported fallback operations, while the accelerator executes dispatched compute kernels. This partition is not absolute: modern accelerators can schedule work queues autonomously, and CPUs can compute tensor operations. In practice, system throughput depends on where each operator runs and the volume of transfers, kernel launches, and synchronization barriers separating them. Direct-memory-access engines and hardware-managed queues can overlap host-device transfers with active execution, but unsupported operators force costly CPU round-trips. The communication boundary between host memory and accelerator storage therefore establishes the effective speedup envelope; kernel execution time matters little if host-device transfers dominate end-to-end latency.

The specialization evident in processing elements, vector pipelines, and tiered scratchpads reflects the computational profile of neural networks: models repeatedly invoke a compact set of linear-algebraic and element-wise primitives. Within the D·A·M framework, algorithmic modifications produce wall-clock speedups only when their computational structure and memory access patterns map directly onto these specialized hardware primitives. Algorithmic optimizations that reduce theoretical operations without aligning with hardware data paths leave compute units starved, rendering the claimed gains purely theoretical.

Self-Check: Question

What primary physical limitation brought about the end of Dennard scaling in the mid-2000s, necessitating the transition from increasing CPU clock frequencies to domain-specific hardware accelerators?
1. Lithography light diffraction preventing further reduction of transistor gate length below $1\,\mu\text{m}$
2. Inability to lower operating voltage proportionally with transistor size, leading to unsustainable power density and heat dissipation limits
3. Quantum tunneling in copper interconnect lines preventing data transmission between arithmetic units
4. Depletion of global silicon substrate supplies requiring migration to gallium nitride semiconductors
Contrast the architectural trade-offs of software-managed scratchpad memory (such as Google TPUv1’s Unified Buffer) with hardware-managed cache hierarchies when executing large-scale tensor workloads.
Place the following computing milestones in chronological order (from earliest to most recent) as hardware evolved toward modern AI accelerators:

Introduction of dedicated Tensor Cores and TPUs for deep learning matrix operations
Emergence of fixed-function media codecs and network processors for video/packet streaming
Integration of floating-point units (FPUs) and digital signal processors (DSPs) as discrete coprocessors
General-purpose programmable GPUs and SIMD instruction set extensions for 3D graphics and multimedia

True or False: Google’s TPUv1 achieved substantial performance-per-watt improvements over contemporary general-purpose CPUs primarily by operating at significantly higher clock frequencies.
In the context of hardware scaling, what phenomenon does the ‘Systems Gap’ describe since the 2012 deep learning breakthrough?
1. The difference in memory bandwidth between high-end datacenter GPUs and consumer-grade mobile SoCs
2. The latency discrepancy between on-chip SRAM access times and host DRAM access times over PCIe
3. The exponential divergence between model compute demand (growing $\approx 6\times$/year) and single-device hardware supply (growing $\approx 1.7\times$/year)
4. The mismatch between Python framework dispatch overhead and raw GPU kernel execution duration

See Answers →

AI Compute Primitives

Across fully connected, convolutional, and attention layers, learned linear transformations reduce to repeated MAC operations. A large layer can issue millions or billions of MACs, and its regular tensor structure exposes parallel work and reuse, making it a natural specialization target. MACs do not dominate every model or phase—normalization, routing, indexing, communication, and element-wise operations can become limiting—but they account for much of the peak arithmetic capacity in modern ML accelerators. Whether they dominate elapsed time still depends on tensor shape, data movement, and the surrounding operators.

The hardware units that exploit these patterns are AI compute primitives: specialized functional blocks, each optimized for a particular class of operation. Three primitives are especially common in accelerators, each targeting a distinct computational pattern found in neural networks.

At the framework level, a single declarative statement encapsulates thousands of arithmetic operations (listing 1).

Listing 1: Dense Layer Abstraction: High-level framework APIs encapsulate 131,072 MACs (256 inputs times 512 outputs) in a single function call, hiding the computational complexity from developers while enabling automatic hardware optimization.

# Framework abstracts compute-intensive operations
dense = Dense(512)(input_tensor)  # 256x512 MACs per sample

Beneath this API call, an autograd or compilation engine lowers the layer into a computational graph. Listing 2 exposes this decomposition into a bulk matrix multiplication, a bias broadcast, and an element-wise activation.

Listing 2: Matrix Operation Expansion: Each dense layer decomposes into matrix multiplication and element-wise operations, exposing the dominant compute pattern that many neural-network kernels are built around.

# Linear transformation work scales with input_dim x output_dim x
# batch.
output = (
    matmul(input, weights) + bias
)  # Matrix multiply dominates cost
output = activation(
    output
)  # Element-wise: proportional to output_dim x batch

While graph representations group these steps into coarse tensor operators, execution on physical hardware requires scheduling individual scalar operations. Lowering the matrix multiplication to loop nests (listing 3) makes the execution structure explicit: three nested iterations traverse the batch, output feature, and reduction dimensions, establishing $\mathcal{O}(B \times d_{\text{in}} \times d_{\text{out}})$ arithmetic complexity.

Listing 3: Processor-Level Execution: Nested loops reveal the $\mathcal{O}(B \times d_{\text{in}} \times d_{\text{out}})$ multiply-accumulate operations that accelerators must execute, with 4.2M MACs for $B$=32, $d_{\text{in}}$=256, $d_{\text{out}}$=512 configurations.

# Total operations: batch_size × output_size × input_size MACs
for n in range(batch_size):  # Batch dimension: parallelizable
    for m in range(output_size):  # Output neurons: parallelizable
        sum = bias[m]  # Initialize accumulator
        for k in range(input_size):  # Reduction dimension: sequential
            sum += input[n, k] * weights[k, m]  # MAC operation
        output[n, m] = activation(sum)  # Nonlinear transformation
# Example work scales as batch_size × output_size ×
# input_size multiply-accumulate operations

These nested loops expose three recurring computational patterns: element-wise operations along vectors, matrix-level multiply-accumulate reductions, and nonlinear transformations. Their ubiquity justifies dedicated hardware execution paths: vector pipelines stream independent elements and reductions, matrix engines organize multiply-accumulate grids across spatial tiles, and special-function units approximate nonlinear math. The boundaries between these units are architectural rather than mathematical; an optimizing compiler frequently fuses operations across all three paths within a single kernel invocation to eliminate intermediate memory traffic.

Vector operations

Vector operations accelerate computation by executing a single instruction across multiple data elements simultaneously. In the scalar formulation of listing 3, processing a batch of 32 samples through a 256-to-512 dense layer requires 4.2M MACs. A scalar processor issues a distinct instruction to load each input-weight pair, perform the multiplication, and update the accumulator, incurring full instruction-fetch, decode, and branch-prediction overhead on every element. Vector pipelines eliminate this instruction overhead by operating on vector registers spanning multiple parallel execution lanes.

RISC-V,¹⁵ an open instruction-set architecture (ISA) with a standardized vector extension (Waterman et al. 2013), provides a concrete vehicle for this execution model. In RISC-V vector assembly (listing 4), vector instructions express operations over arrays of runtime-configured length, decoupling software from the physical lane width of the processor.

¹⁵ RISC-V (reduced instruction set computer V): The open ISA allows hardware teams to add custom ML instructions, including vector dot products, activation functions, and sparse tensor operations, without changing the base ISA. The trade-off is software ecosystem maturity: custom extensions require corresponding compiler, library, and runtime support, which can limit portability and increase integration work.

Waterman, Andrew, Yunsup Lee, Rimas Avizienis, Henry Cook, David Patterson, and Krste Asanovic. 2013. “The RISC-V Instruction Set.” 2013 IEEE Hot Chips 25 Symposium (HCS), 1–1. https://doi.org/10.1109/hotchips.2013.7478332.

Listing 4: Vectorized Dot-Product Loop: This illustrative loop processes several element pairs per iteration and reduces them into one scalar dot product.

fmv.w.x fa0, zero          # Scalar dot-product accumulator
fmv.w.x ft1, zero          # Zero seed for vector reduction
loop_feature:
    vsetvli t0, feature_cnt, e32, m1, ta, ma
    vle32.v v1, (in_ptr)
    vle32.v v2, (wt_ptr)
    vfmul.vv v3, v1, v2
    vfmv.v.f v0, ft1
    vfredusum.vs v4, v3, v0
    vfmv.f.s ft0, v4
    fadd.s fa0, fa0, ft0
    slli t1, t0, 2         # Four bytes per FP32 element
    add in_ptr, in_ptr, t1
    add wt_ptr, wt_ptr, t1
    sub feature_cnt, feature_cnt, t0
    bnez feature_cnt, loop_feature

The vectorized reduction loop in listing 4 progresses through five hardware-visible stages:

Vector length configuration: vsetvli configures the vector units for 32-bit elements (e32), dynamically computing how many elements (t0) fit into the hardware vector registers (VLEN).
Vector initialization: Clears the scalar accumulator that will hold the accumulated dot product.
Vector loads: vle32.v loads contiguous 32-bit elements into vector registers v1 and v2, dispatching wide memory requests across the memory interface.
Vector multiply and reduction: vfmul.vv multiplies element pairs in parallel across vector lanes, and vfredusum.vs reduces the products into a partial scalar sum.
Pointer arithmetic and loop branch: Advances base addresses, decrements the remaining element count by the processed chunk size t0, and branches if elements remain.

Across iterations, the vector multiply instruction processes multiple data elements concurrently while wide loads amortize instruction decode and memory-request overhead. Because the runtime vector length dynamically adapts to the implementation’s hardware width (VLEN), the identical binary executes correctly on both narrow embedded vector units and wide data-center pipelines. For this 256-feature workload, an 8-lane implementation breaks the reduction into approximately 524,288 vector chunks across the batch.

Beyond dense inner products, deep learning workloads rely on a broader set of vector instructions. Table 2 maps these primitive vector operations to the specific neural network layers that depend on them.

Table 2: Vector Operations: Core vector operations map directly to deep learning primitives: reductions implement pooling layers, gathers enable embedding lookups, scatters update embedding gradients, and masked operations handle attention masks. These mappings show how vector hardware supports common ML primitives.

Vector operation	Description	Neural network application
Reduction	Combines elements across a vector (for example, sum, max)	Pooling layers, attention score computation
Gather	Loads multiple nonconsecutive memory elements	Embedding lookups, sparse operations
Scatter	Writes to multiple nonconsecutive memory locations	Gradient updates for embeddings
Masked operations	Selectively operates on vector elements	Attention masks, padding handling
Vector-scalar broadcast	Applies scalar to all vector elements	Bias addition, scaling operations

The benefit extends beyond instruction count. Contiguous vector loads can use memory interfaces efficiently, and control overhead is amortized across several data elements. Gather, scatter, masking, and short tails may use fewer lanes or require extra memory transactions, so vector width alone does not guarantee proportional speedup. The architectural pattern is not new. The Cray-1¹⁶ used vector registers and pipelined functional units for scientific computing in the 1970s (Jordan 1982); modern ML processors apply related principles at much larger commercial scale.

¹⁶ Cray-1 vector legacy: The Cray-1 (1975) used 64-element vector registers with pipelined functional units, allowing a stream of elements to advance through arithmetic operations without issuing one scalar instruction per element. Modern accelerators extend the same principles of wide registers, pipelined execution, and data reuse to vector and matrix tiles.

Jordan, T. L. 1982. “A Guide to Parallel Computation and Some Cray-1 Experiences.” In Parallel Computations. Elsevier. https://doi.org/10.1016/b978-0-12-592101-5.50006-3.

Vector operations excel at element-wise transformations like activation functions, where each output depends only on its corresponding input. Neural networks, however, also require structured computations where each output depends on all inputs—the weighted sums that define layer transformations. These many-to-many operations naturally express themselves as matrix multiplications, the second compute primitive.

Matrix operations

Matrix multiplication accounts for much of the arithmetic in many dense neural networks, transforming high-dimensional data through structured patterns of weights, activations, and gradients (Goodfellow et al. 2016). While vector operations process elements independently or reduce them, matrix operations organize work across multiple dimensions. Hardware and libraries tile those dimensions so blocks of weights and activations can be reused while partial sums accumulate locally. This regular structure drives important hardware strategies, but small or irregular matrices may leave tile capacity unused.

Matrix operations in neural networks

Neural network computations decompose into hierarchical matrix operations. Listing 5 captures this hierarchy through a linear layer that transforms input features into output neurons over a batch.

Listing 5: Linear-Layer Operation Hierarchy: The example expands a linear layer into a matrix multiply and bias broadcast, then applies element-wise ReLU, separating the matrix and vector work that hardware must execute.

layer = nn.Linear(256, 512)  # Layer transforms 256 inputs to 512 outputs
output = layer(input_batch)  # Process a batch of 32 samples

# Framework Internal: Core operations (column-batch convention)
Z = matmul(weights, input)  # Matrix: transforms [256×32]
# input to [512×32] output
Z = Z + bias  # Vector: adds bias to each
# output independently
output = relu(Z)  # Vector: applies activation to
# each element independently

This computation demonstrates the scale of matrix operations in neural networks. Each output neuron (512 total) must process all input features (256 total) for every sample in the batch (32 samples). The weight matrix alone contains 256 $\times$ 512 = 131,072 parameters that define these transformations, illustrating why efficient matrix multiplication dominates performance considerations.

Neural networks employ matrix operations across architectural patterns beyond simple linear layers. Convolution operations transform into matrix multiplications through the im2col technique,¹⁷ enabling efficient execution on matrix-optimized hardware. Listing 6 illustrates these applications.

¹⁷ Im2col (image-to-column): Transforms convolution into a matrix multiplication by arranging receptive-field values as columns. An explicit materialization can duplicate overlapping values and expand storage by up to roughly the kernel area before edge, stride, and channel effects are considered. Production libraries often use implicit-GEMM or direct-convolution kernels to obtain matrix-friendly execution without materializing the full expanded matrix.

Listing 6: Matrix Patterns Across Architectures: Linear layers, attention mechanisms, and convolutions all reduce key work to matrix multiplications, making matrix hardware the shared primitive across modern neural architectures.

hidden = matmul(weights, inputs)
# weights: [out_dim x in_dim], inputs: [in_dim x batch]
# Result combines all inputs for each output

# Attention Mechanisms - Multiple matrix operations
Q = matmul(Wq, inputs)
# Project inputs to query space [query_dim x batch]
K = matmul(Wk, inputs)
# Project inputs to key space [key_dim x batch]
attention = matmul(Q.T, K)
# Compare all query tokens with all key tokens [batch x batch]

# Convolutions - Matrix multiply after reshaping
patches = im2col(input)
# Convert [H x W x C] image to matrix of patches
output = matmul(kernel, patches)
# Apply kernels to all patches simultaneously

These examples differ in tensor shape and reuse pattern, which determines how efficiently each multiplication occupies a matrix unit. Linear layers reuse one weight matrix across a batch; attention forms projections and then a token-to-token score matrix; convolution reuses each kernel across many image patches. Expressing all three as matrix multiplication exposes a common interface to hardware, but it does not make their execution costs identical. Batch size, sequence length, channel count, and tiling determine whether operands are reused locally or repeatedly fetched from memory.

Matrix operations hardware acceleration

This pervasive pattern of matrix multiplication has direct implications for hardware design: accelerators need specialized units that can handle these computations at scale. Listing 7 demonstrates a representative dedicated matrix unit that processes an entire $16{\times}16$ block at once, illustrating why matrix instructions and Tensor Cores can deliver much higher throughput than scalar or vector-only execution paths (NVIDIA 2017; Intel Corporation 2021a).

Listing 7: Illustrative Matrix-Unit Operation: Representative load, matrix-multiply-accumulate, and store instructions operate on matrix blocks rather than individual scalar elements.

mload mr1, (weight_ptr)     # Load e.g., 16x16 block of
                            # weight matrix
mload mr2, (input_ptr)      # Load corresponding input block
matmul.mm mr3, mr1, mr2     # Multiply and accumulate entire
                            # blocks at once
mstore (output_ptr), mr3    # Store computed output block

The illustrative unit operates on matrix tiles rather than individual elements. A full $16{\times}16{\times}16$ tile product contains $16^3=4{,}096$ multiply-accumulate operations, but the cycles required depend on the actual instruction and architecture. Sustained throughput also depends on the number of matrix units, operand precision, issue rate, pipeline behavior, and whether memory can keep the units fed. Matrix units complement vector execution by accelerating structured many-to-many transformations.

Like vector processing, matrix acceleration has deep historical roots—DSPs and GPUs optimized for matrix computations in the 1980s-1990s for image processing, scientific computing, and 3D rendering (Owens et al. 2008; Hwu 2011). Neural networks have made matrix multiplication commercially dominant, driving the integration of two-dimensional systolic arrays and matrix execution engines (such as NVIDIA Tensor Cores and Google TPUs) directly into processor datapaths.

Owens, John D., Mike Houston, David Luebke, Simon Green, John E. Stone, and James C. Phillips. 2008. “GPU Computing.” Proceedings of the IEEE 96 (5): 879–99. https://doi.org/10.1109/jproc.2008.917757.

Hwu, Wen-mei W. 2011. “Introduction.” In GPU Computing Gems Emerald Edition. Elsevier. https://doi.org/10.1016/b978-0-12-384988-5.00064-4.

Linear algebra constitutes only part of the neural network execution pipeline. Between linear transformations, neural networks introduce nonlinear activation functions and normalization layers. While piecewise-linear functions such as ReLU map to simple vector comparison instructions, transcendental computations (such as exponentials in softmax and GELU, or reciprocal square roots in normalization) cannot be computed directly on multiply-accumulate datapaths without incurring severe cycle penalties. Table 3 summarizes the execution characteristics of the three primary compute primitives and their mapping to neural network operations.

Table 3: Operation Characteristics: Matrix operations excel at many-to-many transformations, while vector units handle element-wise operations and reductions such as activation and normalization. Special-function paths handle transcendental arithmetic through architecture-specific approximation or dedicated logic. The operation structure determines which hardware primitive is appropriate.

Operation type	Best For	Examples	Key characteristic
Matrix Operations	Many-to-many transforms	Layer transformations, attention, convolutions	Each output depends on multiple inputs
Vector Operations	Vector and reduction work	Activation functions, layer normalization, element-wise gradients	Uses element-wise operations and vector reductions
Special-Function Paths	Transcendental arithmetic	Exponential, logarithm, reciprocal square root	Uses architecture-specific approximation or dedicated logic

Special function units

Special Function Units (SFUs) or specialized instruction paths accelerate nonlinear functions and related arithmetic, completing the chapter’s trio of processing primitives. The need is not new: floating-point coprocessors addressed scalar arithmetic bottlenecks (Palmer 1980), and digital signal processors added specialized arithmetic for signal-processing workloads (Smith 1997). An implementation may use a standalone unit, vector approximation instructions, or a library sequence, trading accuracy, latency, and throughput. In neural networks, activation, normalization, and softmax operations can become important between matrix kernels, particularly when repeated memory passes or transcendental functions limit throughput.

Palmer, John F. 1980. “The Intel 8087 Numeric Data Processor.” Proceedings of the 1980 National Computer Conference (AFIPS), 887–93. https://doi.org/10.1145/1500518.1500674.

Smith, Steven W. 1997. “Digital Signal Processing.” In Digital Signal Processing Demystified. Elsevier. https://doi.org/10.1016/b978-187870716-1/50004-4.

Nonlinear functions

A common layer sequence illustrates how non-matrix operators enter the execution pipeline (Goodfellow et al. 2016). Listing 8 chains a linear transformation with ReLU activation and batch normalization—operations that appear trivial in Python framework code but impose distinct arithmetic and memory constraints on hardware.

Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press.

Listing 8: Linear–Nonlinear Layer Sequence: A linear layer, ReLU, and BatchNorm combine matrix, element-wise, and normalization work in one common layer sequence.

layer = nn.Sequential(
    nn.Linear(256, 512), nn.ReLU(), nn.BatchNorm1d(512)
)
output = layer(input_tensor)

Listing 9 expands this sequence into its arithmetic primitives, isolating the element-wise thresholding of ReLU from the cross-batch reductions and reciprocal square root required by batch normalization.

Listing 9: Nonlinear Operation Expansion: Expanding the layer sequence exposes ReLU comparison, BatchNorm reductions, variance computation, and square-root normalization around the matrix multiply.

Z = matmul(weights, input) + bias  # Linear transformation
H = max(0, Z)  # ReLU activation
mean = reduce_mean(H, axis=0)  # BatchNorm statistics
var = reduce_mean((H - mean) ** 2)  # Variance computation
output = gamma * (H - mean) / sqrt(var + eps) + beta  # Normalization

Hardware implementation of nonlinear functions

Translating these mathematical functions to silicon exposes sharp differences in memory traffic, instruction count, and pipeline utilization. For instance, batch normalization (Ioffe and Szegedy 2015) demands multi-pass statistics gathering across memory, variance accumulation, and a reciprocal square root, while softmax requires computing exponentials across an entire normalization dimension. While a rectified linear unit (ReLU) is mathematically simple, a naive scalar implementation incurs branch overhead or redundant memory round-trips unless fused into adjacent kernels. Listing 10 contrasts the memory and dependency overheads of pointwise activation against multi-pass normalization in an unfused loop model.

Ioffe, Sergey, and Christian Szegedy. 2015. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.” Proceedings of the 32nd International Conference on Machine Learning (ICML) 37: 448–56.

The label nonlinear hides three different execution contracts. Pointwise functions such as ReLU leave elements independent and therefore map naturally to vector lanes or parallel threads. Reductions such as the mean and variance in batch normalization introduce cross-element dependencies: partial results must be combined before later stages can proceed. Transcendental functions such as exponential, logarithm, and reciprocal square root add a numerical contract because their implementations trade latency, precision, and valid input range. The dependency pattern, not the mathematical label alone, determines whether vector arithmetic, a reduction tree, or a specialized approximation path is appropriate. A faster special-function path accelerates only the operation it implements; it does not remove surrounding tensor traffic or synchronization.

Listing 10 is intentionally dependency-explicit rather than a model of an optimized kernel. The loops expose three state lifetimes: per-element activations, per-feature accumulators, and the final normalized outputs. The normalization parameters cannot be applied until the required statistics are available, so implementations must either retain intermediate activations or recompute them. Training derives those statistics from the current batch, whereas inference can use stored statistics and reduce normalization to an element-wise affine transform. Compilers may fuse legal stages, but the data dependencies determine which transformations preserve the computation. Reading the listing as a dependency graph therefore reveals the durable hardware questions: how often each tensor crosses a memory interface, which intermediates remain local, and where reductions impose synchronization.

Listing 10: ReLU and BatchNorm Operations: Neural networks process input data through element-wise selections and multiple normalization passes, highlighting efficiency challenges in naive implementations.

for batch in range(32):
    for feature in range(512):
        # ReLU: Naive scalar compare/select; optimized kernels
        # usually implement this branchlessly.
        z = matmul_output[batch, feature]
        h = max(0.0, z)  # Conditional operation

        # BatchNorm: Multiple passes over data
        mean_sum[feature] += h  # First pass for mean
        var_sum[feature] += h * h  # Additional pass for variance

        temp[batch, feature] = h  # Extra memory storage needed


# Normalization requires complex arithmetic
for feature in range(512):
    mean = mean_sum[feature] / batch_size
    var = (var_sum[feature] / batch_size) - mean * mean

    # Square root computation: Multiple iterations
    scale = gamma[feature] / sqrt(var + eps)  # Iterative
    # approximation
    shift = beta[feature] - mean * scale

    # Additional pass over data for final computation
    for batch in range(32):
        output[batch, feature] = temp[batch, feature] * scale + shift

The dependency structure in listing 10 exposes why naive execution collapses performance: arithmetic intensity drops precipitously. The pointwise ReLU and normalization scale-and-shift perform only one or two arithmetic operations per four-byte memory access, plunging the workload deep into the memory-bound regime. Furthermore, because batch statistics cannot be finalized until all samples are processed, the intermediate tensor temp must be written back to memory and reread in a subsequent pass, inflating memory bandwidth consumption and cache pressure. At the same time, computing $1/\sqrt{\mathrm{var} + \epsilon}$ requires an iterative square root and division that can stall pipeline execution for tens of cycles if executed on standard integer or floating-point ALUs. Mitigating these overheads requires two complementary strategies: hardware-level Special Function Units to approximate transcendental arithmetic in single-digit cycles, and compiler-level kernel fusion to keep intermediate values in registers rather than spilling them across the memory bus.

SFU hardware implementation

SFUs address arithmetic bottlenecks by implementing nonlinear and transcendental operations in specialized functional units rather than multi-instruction software routines. Accelerators deploy dedicated datapath circuits—such as quadratic polynomial interpolators and small lookup tables (LUTs)—that evaluate functions like exponential, reciprocal square root, and logarithm within a few pipeline stages. Listing 11 illustrates how an instruction set exposes these operations alongside vector memory accesses.

Listing 11: Illustrative SFU Operations: Pseudocode shows vector nonlinear operations that may map to specialized execution paths; instruction names and latency are implementation dependent.

vld.v v1, (input_ptr)    # Load vector of values (pseudocode)
vrelu.v v2, v1           # Vector ReLU path
vsigm.v v3, v1           # Vector sigmoid path
vtanh.v v4, v1           # Vector tanh path
vrsqrt.v v5, v1          # Vector reciprocal-square-root path

Specialized function paths balance circuit area against mathematical precision. While piecewise linear activations such as ReLU require only single-cycle compare-and-select (maximum) datapath logic, transcendental functions trade off lookup table size and polynomial interpolation degrees against approximation error. Table 4 summarizes representative hardware mechanisms and their qualitative latency characteristics across these primitives.

Table 4: Special Function Units: Dedicated paths reduce instruction overhead for nonlinear and reduction primitives. Implementations and latency are architecture-dependent; the table describes mechanisms rather than universal cycle counts.

Function unit	Operation	Implementation strategy	Illustrative latency
Activation unit	ReLU	Compare-and-select or maximum	Low; architecture-dependent
Statistics unit	Mean, variance	Parallel reduction trees	Grows with reduction depth
Exponential unit	exp, log, sigmoid, tanh	Approximation, table lookup, and interpolation	Architecture-dependent
Root/power unit	sqrt, rsqrt	Iterative or approximation hardware	Architecture-dependent

Vector operations, matrix operations, and special-function paths establish the foundational arithmetic primitives of ML accelerators, but functional units alone do not dictate sustained throughput. These primitives govern what the silicon computes; the execution model and memory hierarchy govern how data feeds the datapath. A matrix unit rated for hundreds of teraFLOP/s will idle if memory bandwidth cannot sustain operand delivery, if tile dimensions mismatch register blocking, or if thread synchronization serializes execution. Peak hardware ratings describe performance under idealized operand availability; the architecture’s execution model dictates how much of that peak is achievable on real neural network workloads.

Self-Check: Question

When executing a Transformer layer containing attention projection ($Q = X W_q$), Softmax ($\text{Softmax}(S)$), Layer Normalization ($\text{LayerNorm}(X)$), and GeLU activation ($\text{GeLU}(Z)$), which execution unit is specifically responsible for computing transcendental functions (exponential and error function approximations)?
1. Systolic 2D Matrix Multiply Units
2. Dense Tensor Cores
3. Vector Load-Store Memory Controllers
4. Special Function Units (SFUs)
Explain how the $\text{im2col}$ (image-to-column) transformation allows standard 2D convolution operations to execute on high-throughput matrix multiplication hardware (GEMM engines), and describe the primary memory overhead associated with this approach.
True or False: In modern AI accelerators, element-wise vector operations such as residual additions ($Y = X_1 + X_2$) achieve higher arithmetic intensity than large matrix-matrix multiplications ($C = A \cdot B$).
The hardware transformation technique that enables convolutional layers to execute as matrix multiplications without physically duplicating overlapping patch data in memory is known as ____ GEMM.
Which of the following operations in a modern deep learning architecture exhibits the highest operational arithmetic reuse, making it most suitable for dense 2D systolic arrays and Tensor Cores?
1. Batched linear layer matrix multiplication ($Y = X W$)
2. Element-wise ReLU activation ($\max(0, x)$)
3. Channel-wise Batch Normalization mean computation
4. Token-wise embedding table lookup

See Answers →

Compute Units and Execution Models

Applying ReLU to a 512-element vector shows why execution models matter: the operation is simple, but throughput depends on whether the hardware treats those 512 comparisons as scalar instructions, SIMD lanes, GPU threads, or tensor-program fragments. Modern AI processors package the three compute primitives into distinct execution units: single instruction, multiple data (SIMD) units, Tensor Cores, and processing elements that define how computations are structured and exposed to programmers. Understanding this organization reveals both the theoretical capabilities and practical performance characteristics that determine real-world throughput.

Mapping primitives to execution units

Mapping computational primitives to execution units is governed by physical wire energy and data reuse. Reading an operand from a register file and moving it across on-chip wires consumes several times more energy than the arithmetic operation itself. Hardware accelerators manage this physical constraint by provisioning specialized execution units matched to the reuse characteristics of each primitive:

Vector operations → SIMD and single instruction, multiple threads (SIMT) units. These execute 1D elementwise primitives—such as additions, scaling, and bias offsets—across parallel ALUs that share instruction fetch and decode logic. Because vector operations exhibit $O(1)$ arithmetic reuse per operand loaded, their throughput remains bounded by register file port bandwidth and memory delivery.
Matrix operations → Tensor Cores¹⁸ and systolic arrays. These map 2D inner-product primitives to dense two-dimensional grids of MAC circuits. By routing activations and weights directly between adjacent processing elements through local wires—such as in weight-stationary or output-stationary dataflows—matrix units reuse data in flight. This spatial reuse decouples arithmetic throughput from main register file bandwidth, sustaining peak compute density within a manageable thermal envelope.
Special functions → Special Function Units (SFUs). These map transcendental and non-linear primitives—such as $\exp$, $\ln$, and reciprocal square roots—to dedicated pipelines using piecewise polynomial approximations and lookup tables. Evaluating transcendentals on general-purpose ALUs requires iterative numerical algorithms that consume dozens of clock cycles; dedicated SFUs evaluate these functions off the primary execution path at lower throughput, preventing activation layers from stalling vector pipelines.

¹⁸ Reduced-precision ML: Halving operand width halves the bytes per stored element and can support denser arithmetic, although realized throughput depends on the available datapath and kernel (Dally et al. 2021; Dally 2023). NVIDIA’s transition from Pascal P100 vector arithmetic to mixed-precision Tensor Cores in Volta V100 increased advertised peak FP16 throughput from about 21.2 to 125 TFLOP/s, roughly 6$\times$ across those products (NVIDIA 2017). Precision selection must preserve numerical behavior and use an efficient supported path; the smallest format is not automatically the best one.

Dally, William J., Stephen W. Keckler, and David B. Kirk. 2021. “Evolution of the Graphics Processing Unit (GPU).” IEEE Micro 41 (6): 42–51. https://doi.org/10.1109/mm.2021.3113475.

Dally, Bill. 2023. “Hardware for Deep Learning.” 2023 IEEE Hot Chips 35 Symposium (HCS), 1–58. https://doi.org/10.1109/hcs59251.2023.10254716.

In the D·A·M taxonomy, this partitioning defines how algorithmic primitives bind to physical machine execution paths. Routing dense matrix multiplications through standard vector pipelines rapidly saturates register file read ports, collapsing throughput to a small fraction of the accelerator’s peak capacity. Conversely, routing low-arithmetic-intensity elementwise operations or irregular activations through matrix arrays leaves processing lanes unutilized. Sustained system performance depends on matching each computational primitive to the execution unit whose dataflow and storage hierarchy match its arithmetic intensity.

Evolution from SIMD to SIMT architectures

In scalar processor architectures, executing an elementwise operation across a 512-element activation vector requires 512 individual loop iterations. Each iteration fetches, decodes, and tracks an instruction through the pipeline, consuming substantially more energy in instruction control logic than the arithmetic logic unit (ALU) expends performing the actual comparison or addition. Under fixed thermal design power (TDP) budgets, scalar execution spends silicon area and power on instruction management rather than arithmetic throughput. Flynn’s SIMD execution model amortizes this front-end overhead by issuing a single decoded instruction across multiple data elements packed into wide vector registers (Flynn 1966). A 512-bit SIMD register, for example, processes sixteen 32-bit floating-point values simultaneously in a single clock cycle, reducing instruction fetch and decode traffic sixteenfold.

Flynn, M. J. 1966. “Very High-Speed Computing Systems.” Proceedings of the IEEE 54 (12): 1901–9. https://doi.org/10.1109/proc.1966.5273.

Traditional fixed-width SIMD architectures bind compiled machine code to specific hardware register widths, such as 128, 256, or 512 bits. When a tensor dimension does not align with the fixed vector width, compilers must generate scalar cleanup loops or insert memory padding, increasing binary footprint and branch complexity. The Arm Scalable Vector Extension (SVE) resolves this rigidity through vector-length-agnostic (VLA) execution and per-lane predication (Stephens et al. 2017). Instead of hardcoding physical register widths into the instruction encoding, SVE relies on predicate registers to dynamically configure active lanes, as shown in the vector arithmetic sequence in listing 12.

Listing 12: Arm SVE Vector Execution: Predicated vector assembly loads, multiplies, and adds floating-point elements using length-agnostic registers.

ptrue p0.s              # Create predicate for vector length
ld1w z0.s, p0/z, [x0]   # Load vector of inputs
fmul z1.s, z0.s, z0.s   # Multiply elements
fadd z2.s, z1.s, z0.s   # Add elements
st1w z2.s, p0, [x1]     # Store results

The ptrue instruction activates all available predicate lanes without encoding a fixed vector length. This vector-length-agnostic programming model allows the same SVE binary to run on implementations from 128 to 2048 bits without recompilation (Stephens et al. 2017). Intel’s Advanced Matrix Extensions (AMX) take specialization a step further: dedicated tile registers and matrix-multiplication accelerators operate directly on two-dimensional data blocks rather than one-dimensional vectors (Intel Corporation 2021a). Together, SVE and AMX represent CPU-side adaptations for ML workloads: portable vector execution and fixed-tile matrix pipelines.

Stephens, Nigel, Stuart Biles, Matthias Boettcher, Jacob Eapen, Mbou Eyole, Giacomo Gabrielli, Matt Horsnell, et al. 2017. “The ARM Scalable Vector Extension.” IEEE Micro 37 (2): 26–39. https://doi.org/10.1109/mm.2017.35.

Despite vector-length agility, CPU SIMD architectures face fundamental physical limitations when scaling to large neural network workloads. CPUs dedicate substantial silicon area to branch predictors, out-of-order execution logic, and multi-level cache hierarchies to minimize latency for a few sequential threads. When streaming through gigabytes of tensor data, cache misses force the execution pipeline to wait hundreds of cycles for main memory fetches. The CPU’s instruction reorder buffer fills rapidly during memory stalls, leaving wide SIMD pipelines starved of work. Furthermore, conditional branching within a vector requires evaluating both paths under bitmasks, reducing useful lane utilization.

Single Instruction, Multiple Threads (SIMT) architectures resolve these bottlenecks by decoupling the logical programming abstraction from physical hardware execution (Lindholm et al. 2008; Nickolls et al. 2008). Instead of requiring the programmer to pack data into explicit vector registers, SIMT exposes an execution model of independent scalar threads. In GPU architectures, each Streaming Multiprocessor (SM)¹⁹ coordinates thousands of concurrent threads. Hardware warp schedulers group these threads into warps,²⁰ which are 32-thread execution units (or wavefronts in AMD architectures) that execute instructions in lockstep across parallel ALUs. Rather than relying on large caches to reduce memory latency, GPUs hide latency through massive thread-level concurrency. An SM statically partitions an on-chip register file across all active threads, eliminating context-saving overhead; when one warp stalls waiting hundreds of cycles for an off-chip memory fetch, the warp scheduler selects and issues instructions from another ready warp on the very next cycle. The baseline matrix multiplication kernel in listing 13 illustrates how this scalar thread model maps onto hardware.

Lindholm, Erik, John Nickolls, Stuart Oberman, and John Montrym. 2008. “NVIDIA Tesla: A Unified Graphics and Computing Architecture.” IEEE Micro 28 (2): 39–55. https://doi.org/10.1109/mm.2008.31.

Nickolls, John, Ian Buck, Michael Garland, and Kevin Skadron. 2008. “Scalable Parallel Programming with CUDA: Is CUDA the Parallel Programming Model That Application Developers Have Been Waiting For?” Queue 6 (2): 40–53. https://doi.org/10.1145/1365490.1365500.

¹⁹ SM (streaming multiprocessor): The physical hardware engine that implements the SIMT model by using warp schedulers to coordinate many parallel threads. Maintaining enough active warps can help hide instruction and memory latency, but occupancy alone does not identify the bottleneck. Low occupancy may result from register use, shared-memory use, block dimensions, or insufficient parallel work, while a memory-bound kernel can still have high occupancy.

²⁰ Warp: On NVIDIA GPUs, a warp is a hardware scheduling group of 32 threads that execute a common instruction when their control paths agree. If threads take different paths, the hardware executes the required paths under different masks, reducing useful lane utilization. The penalty depends on path length and how many lanes take each path, which is why ML kernels often prefer uniform or predicated control flow.

Listing 13: SIMT Execution: Each thread processes a unique output element in parallel, demonstrating how SIMT enables efficient matrix multiplication on GPUs.

__global__ void matrix_multiply(float* C, float* A, float*
                                B, int N) {  // CUDA kernel
    // Each thread processes one output element
    int row = blockIdx.y * blockDim.y + threadIdx.y;
    int col = blockIdx.x * blockDim.x + threadIdx.x;

    float sum = 0.0f;
    for (int k = 0; k < N; k++) {
        // Threads in a warp execute in parallel
        sum += A[row * N + k] * B[k * N + col];
    }
    C[row * N + col] = sum;
}

In listing 13, the CUDA²¹ kernel assigns one output element $C[\text{row}, \text{col}]$ to each thread. Adjacent threads in a warp share the same row index while having consecutive column indices (threadIdx.x). When threads fetch $B[k \cdot N + \text{col}]$, their memory requests target contiguous 4-byte floating-point addresses. The memory controller merges these 32 individual 4-byte loads into a single 128-byte coalesced memory transaction, maximizing DRAM bus efficiency. Concurrently, all threads in the warp read the identical element $A[\text{row} \cdot N + k]$, allowing the hardware to broadcast that single fetched value across all 32 lanes. However, this naive kernel illustrates the iron law of ML systems: arithmetic intensity limits delivered throughput. Because each inner-loop iteration performs only two floating-point operations while reloading operands from global memory without on-chip SRAM staging, the kernel remains severely memory bound. Furthermore, executing dense matrix multiplication on scalar SIMT ALUs requires one instruction fetch, decode, and issue cycle for every multiply-accumulate, saturating register file read ports. Overcoming this machine bottleneck requires moving beyond vector-of-scalars execution to specialized two-dimensional matrix units.

²¹ CUDA (compute unified device architecture): Released by NVIDIA in 2006, CUDA eliminated the need to disguise general-purpose computations as graphics operations, opening GPUs to scientific and ML workloads through a C-like programming model. The ecosystem it created—cuBLAS, cuDNN, TensorRT—constitutes a software moat that can lock the ML training stack to NVIDIA hardware: migrating away requires rewriting or replacing thousands of GPU-optimized kernels, a cost that often exceeds the hardware savings of competing platforms. This software lock-in, not raw silicon performance alone, helps explain why many large ML training stacks remain CUDA-centered.

Tensor Cores

Consider a single transformer attention head computing the $\mathbf{Q} \times \mathbf{K}^T$ product for a 2,048-token sequence with 64-dimensional embeddings. This operation requires multiplying a $2048{\times}64$ matrix by a $64{\times}2048$ matrix: roughly 268.4M MACs, or about 536.9 MFLOP when the multiply and add are counted separately. On a scalar processor executing one FLOP per cycle at 2 GHz, this single attention head would take about 268.4 ms. GPU SIMT execution can distribute this work across many threads, but Tensor Cores go further by processing entire matrix tiles per instruction; under the illustrative assumption used here, the tiled path completes the same operation in 0.5 milliseconds, a roughly 536.9× improvement over scalar execution. This speedup arises not from higher clock frequencies, but from hardwiring tile-level data reuse directly into the execution datapath.

While SIMD and SIMT units execute vector work efficiently, large matrix computations benefit from units organized around multidimensional tiles. Dedicated matrix engines amortize operand loads across many multiply-accumulate operations by reusing tiles on chip. NVIDIA GPUs expose Tensor Cores, whereas Google TPUs use matrix units built around systolic arrays. Both execute tiled matrix multiplication and accumulation.²² The benefit depends on tile reuse, shape, precision, and memory behavior.

²² Tensor Core dimension alignment: NVIDIA Tensor Cores are most efficient when matrix dimensions are aligned to precision- and architecture-specific multiples, such as multiples of 8 or 16 for common FP16/BF16/INT8 paths. Modern cuBLAS and cuDNN can still use Tensor Cores for many nonaligned dimensions, but poorly aligned shapes may trigger less efficient kernels, require padding, or reduce effective throughput (NVIDIA 2024a; NVIDIA Corporation 2021). This is why model architects often choose embedding and channel dimensions such as 512 rather than 500 and why batch-size-1 inference may fail to reach peak Tensor Core utilization: alignment and arithmetic intensity jointly determine whether the hardware’s matrix engines stay full.

NVIDIA Corporation. 2021. NVIDIA cuDNN Developer Guide.

²³ Tensor Core: A single Tensor Core instruction executes a complete matrix-multiply-accumulate operation on a small tile of data using a dedicated hardware block (NVIDIA 2017; NVIDIA Corporation 2020a). This approach bypasses the overhead of fetching and scheduling dozens of individual arithmetic instructions on general-purpose CUDA cores. Because these blocks constitute a large fraction of a modern accelerator’s advertised tensor throughput, failing to use them can leave most of the chip’s theoretical peak unavailable to the workload.

NVIDIA. 2017. Training with Mixed Precision.

NVIDIA Parallel Thread Execution (PTX) exposes this architecture through warp-synchronous matrix instructions.²³ In listing 14, all 32 threads in a warp execute mma.sync collectively to compute a matrix tile product $D = A \times B + C$.

Listing 14: Tensor Core Operation: Matrix multiplications are performed in parallel across entire matrix blocks, optimizing computational efficiency for neural network training.


Tensor Core Operation (example GPU PTX):
mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32
  {d0,d1,d2,d3},     // Destination registers
  {a0,a1,a2,a3},     // Source matrix A
  {b0,b1},           // Source matrix B
  {c0,c1,c2,c3};     // Accumulator

In this instruction, the warp executes an $M=16, N=8, K=16$ matrix multiply-accumulate operation in mixed precision: inputs $A$ ($16{\times}16$) and $B$ ($16{\times}8$) are formatted in FP16, while accumulator $C$ and destination $D$ ($16{\times}8$) use FP32. Rather than scheduling 64 separate scalar fused multiply-add (fma) instructions per thread, the warp scheduler issues a single instruction that evaluates 2,048 multiply-accumulate operations across the Tensor Core datapath. Distributing matrix fragments across thread registers and accumulating intermediate sums within dedicated adder trees relieves two key hardware bottlenecks of conventional SIMT execution: instruction issue bandwidth and register file read-port contention.

Design priorities determine how matrix engines appear in different processor families. GPU Tensor Cores preserve programmability while accelerating general-purpose deep learning kernels. TPU-style designs use large-scale matrix units arranged in systolic arrays to maximize sustained training throughput on dense tensor kernels. Mobile NPUs²⁴ shrink the same idea into low-power inference blocks, while server CPUs add matrix instruction extensions (AMX-class tiles) for inference and mixed workloads. Each version changes the same contract: how much flexibility the hardware keeps while reducing movement around dense matrix operations.

²⁴ NPU (neural processing unit): Mobile NPUs achieve low-power inference by implementing common tensor operations in fixed-function or narrowly programmable hardware rather than as fully general GPU kernels. This architectural commitment can deliver large energy-efficiency gains for supported kernels, but it makes deployment dependent on operator coverage: unsupported functions must fall back to a CPU or GPU path that may be far less efficient for that workload (Sze et al. 2017).

Sze, Vivienne, Yu-Hsin Chen, Tien-Ju Yang, and Joel S. Emer. 2017. “Efficient Processing of Deep Neural Networks: A Tutorial and Survey.” Proceedings of the IEEE 105 (12): 2295–329. https://doi.org/10.1109/jproc.2017.2761740.

Figure 6 shows how specialization, lower precision, and sparsity support changed advertised peak capability. From K20X FP32 to H100 FP8 with structured sparsity, the displayed values rise by roughly three orders of magnitude (NVIDIA Corporation 2017, 2020a, 2024; Choquette 2023). The points do not measure one consistent quantity: they mix precision formats and combine FLOP/s with INT8 operations per second. The curve therefore compares each generation’s most prominent advertised path, not same-precision application performance. Master Accelerator Specification Matrix provides the corresponding precision-specific specifications, bandwidth, and power limits.

Figure 6: **Advertised GPU Peak Capability Across Generations**: Single-chip values are plotted on a logarithmic axis. The precision label at each point is essential: the curve combines architectural gains with different numerical formats and, for some points, sparsity assumptions rather than measuring same-precision application scaling (NVIDIA Corporation 2017, 2020a, 2024; Choquette 2023).

Processing elements

At the next level of execution hierarchy, accelerators integrate multiple Tensor Cores with local memory into modular processing elements (PEs). High-throughput matrix units cannot be fed directly from global memory; off-chip DRAM and on-package HBM lack the bandwidth and low latency required to stream operands into dense arithmetic arrays every cycle. A processing element resolves this bandwidth deficit by co-locating heterogeneous compute pipelines—Tensor Cores for matrix multiplication, vector units for element-wise operations, and special function units for nonlinear transformations—directly alongside high-bandwidth register files and scratchpad SRAM. Staging operands inside the PE keeps data adjacent to the functional units, amortizing the latency and energy cost of external memory accesses across repeated arithmetic reuse.

Processing element design varies because each architecture chooses a different balance between compute density, local memory capacity, and interconnect distance. Graphcore’s Intelligence Processing Unit (IPU) distributes computation across 1,472 tiles, each containing an independent processing element backed by dedicated local SRAM to exploit fine-grained parallelism without memory contention (Graphcore 2020). Cerebras extends this local-storage principle in the CS-2 system, integrating roughly 850,000 AI-optimized cores across a wafer-scale device to bypass off-chip memory interfaces entirely in favor of an on-chip SRAM mesh (Systems 2021). Tesla’s D1 processor dedicates substantial silicon area to local SRAM and high-throughput routing interfaces inside its processing elements, prioritizing low-latency data exchange across tightly coupled compute arrays (Tesla, Inc. 2021).

Graphcore. 2020. The Colossus MK2 IPU Processor.

Tesla, Inc. 2021. Tesla Dojo Technology: A Guide to Tesla’s Configurable Floating Point Formats & Arithmetic. Tesla AI Day, Whitepaper.

Across these designs, the binding physical constraint is compute density versus data locality: silicon area dedicated to arithmetic units cannot simultaneously house SRAM storage or routing wires. Packing more execution units raises peak theoretical FLOP/s, but those units idle if memory bandwidth and routing channels cannot deliver operands. A processing element’s delivered efficiency therefore depends as much on its local memory hierarchy and interconnect strategy as on its raw arithmetic capability.

That same dependence on locality governs which algorithmic optimizations the hardware can actually exploit. A regular grid of processing elements accelerates sparsity only when the surviving nonzero values preserve the predictable access patterns the grid depends on, which is precisely the constraint that N:M structured sparsity is designed to satisfy.

N:M structured sparsity mechanics

While unstructured pruning reduces model parameter counts, it rarely translates to hardware speedup because arbitrary nonzero placements destroy memory locality. Hardware accelerators resolve this with N:M structured sparsity,²⁵ which bounds sparsity to a deterministic, fine-grained pattern. The notation “$N{:}M$” specifies that exactly $N$ values must be nonzero within every contiguous block of $M$ values along the reduction dimension. Restricting nonzeros to fixed local intervals guarantees uniform memory access and predictable indexing across parallel execution units.

²⁵ N:M structured sparsity: The 2:4 ratio (50 percent density) used by NVIDIA’s Ampere Sparse Tensor Cores is a hardware-friendly compromise: every contiguous four-value group retains two nonzero values, preserving regular indexing while halving the dense value payload (NVIDIA Corporation 2020a). At 2:4, the metadata overhead is compact enough to store alongside the weights without overwhelming the memory-traffic savings, which is the constraint that makes the advertised 2$\times$ tensor-math throughput path plausible when kernels and model weights satisfy the pattern.

NVIDIA Corporation. 2020a. NVIDIA A100 Tensor Core GPU Architecture. NVIDIA Whitepaper, V1.0.

Parallel architectures require structural regularity because of how sparse matrices must be encoded in memory. Classical formats such as Compressed Sparse Row (CSR) and Block Compressed Sparse Row (BSR)—treated formally in Sparse matrix formats—rely on auxiliary index arrays to locate nonzero entries. When sparsity is unstructured, adjacent threads in a SIMT warp access non-contiguous memory addresses, generating uncoalesced memory transactions that saturate the memory bus while fetching mostly empty cache lines. In addition, varying numbers of nonzeros across rows cause branch divergence, forcing threads to serialize. While coarse block sparsity groups nonzeros into larger dense tiles to mitigate divergence, it degrades model accuracy when whole blocks must be pruned. Figure 7 compares these occupancy patterns and demonstrates the substantial metadata required to index irregular or blocked nonzeros.

Figure 7: **Sparse Matrix Layouts**: The four panels compare dense, CSR, block-sparse, and BSR matrix occupancy patterns. The separate nonzero-block-index column represents the metadata required to locate retained blocks in the block-sparse cases. Regular block structure keeps this metadata predictable and compact so hardware can skip zero blocks and fetch retained blocks efficiently.

NVIDIA’s Sparse Tensor Cores implement a concrete instance of fine-grained structured sparsity: the 2:4 constraint, which requires that exactly two of every contiguous block of four values along the reduction dimension be nonzero (NVIDIA Corporation 2020a; NVIDIA 2020a). In memory, the weight matrix is stored in compressed form: only the two nonzero values are retained, accompanied by a 4-bit metadata tag per block (2 bits per value) indicating each nonzero element’s original position within the four-element vector. During execution, the memory hierarchy transfers only the compressed weights and their metadata, halving the required weight memory bandwidth relative to dense execution. Inside the Sparse Tensor Core datapath, the 4-bit metadata steers an internal multiplexer network that selects the matching two values from an uncompressed four-element activation vector. These selected activations and the two nonzero weights are then routed into a half-sized dot-product multiplier tree. Because the hardware computes only on the nonzeros while streaming compressed weights, this design doubles the peak arithmetic instruction throughput—yielding an advertised 2$\times$ tensor-math speedup over dense operations when weights are fine-tuned to satisfy the pattern.

NVIDIA. 2020a. Accelerating Matrix Multiplication with Block Sparse Format and NVIDIA Tensor Cores.

The 2:4 pattern reinforces the central reality of the memory wall: hardware accelerators gain efficiency not by computing on zeros faster, but by eliminating their transfer entirely. Halving the weight payload reduces traffic across HBM and on-chip SRAM caches, shifting the workload to a higher operational arithmetic intensity on the roofline curve.

While Sparse Tensor Cores accelerate matrix multiplication by filtering operands through multiplexers within a SIMT datapath, alternative architectures bypass memory bandwidth limits by altering how data moves through space. Rather than repeatedly cycling activations between execution units and centralized register files, systolic arrays forward intermediate values directly between neighboring processing elements across a physical grid.

Systolic arrays

While Tensor Cores package matrix operations into small, specialized instruction tiles, systolic arrays structure processing elements into large two-dimensional grids optimized for continuous data flow and operand reuse. By pulsing inputs horizontally and partial sums vertically through the array, systolic architectures minimize expensive DRAM fetches, enabling low-precision, quantized weights to achieve maximum compute density. The core motivation for systolic architectures stems from the same energy constraint that drives accelerator design: minimizing memory access penalties through physical operand reuse. A simple energy comparison through the array reveals why this architecture has become central to modern AI accelerators.

Napkin Math 1.1: The energy advantage of pulsing data

Scenario: The systolic architecture improves energy efficiency by keeping operands local as work pulses through the array; the “Systolic” (heartbeat) metaphor reflects this benefit of reusing operands locally. This worked model compares systolic dataflow with a naive implementation that streams every operand through DRAM:

DRAM-streaming baseline: Loads $\mathbf{A}$, loads $\mathbf{B}$, computes $\mathbf{A} \times \mathbf{B} + \mathbf{C}$, and writes $\mathbf{C}$ without cache or register reuse.
- Data movement: 3 loads + 1 write = 4 DRAM accesses (per operation).
- Energy: ≈ 4 $\times$ 640 pJ + 1 pJ (compute) = 2561 pJ/op.
Systolic Array (128 $\times$ 128 size): Loads A and B once at the edges. Data “pulses” through 128 processing elements.
- Data movement: 2 loads per 128 operations = 0.016 DRAM accesses (per operation).
- Energy: ≈ 0.016 $\times$ 640 pJ + 1 pJ (compute) ≈ 11 pJ/op.

Systems insight: Under this deliberately naive DRAM-streaming baseline, the worked model gives the systolic array a 232.8× energy advantage. The ratio measures locality, not an inherent gap between vector and systolic arithmetic.

Concretely, a $128{\times}128$ array can sustain 16,384 MACs/cycle with a large energy dividend by pulsing data through processing elements instead of repeatedly loading it from DRAM (Horowitz 2014; Jouppi et al. 2023).
This locality lets dense arrays sustain many simultaneous MAC operations without paying a DRAM access for every operation.
Limitation: The comparison assumes no effective reuse in the baseline and full reuse across the array. Caches, register blocking, array underutilization, and other dataflows change the ratio substantially.

A systolic array arranges processing elements in a grid pattern, where data flows rhythmically between neighboring units in a synchronized manner, enabling each operand to participate in multiple computations as it propagates through the array. This structured movement minimizes external memory accesses by maximizing local data reuse. A single weight value can contribute to dozens of operations as it moves through the processing elements, transforming the energy profile from memory-bound to compute-efficient execution.

Kung and Leiserson²⁶ (Kung and Leiserson 1979) first introduced systolic arrays, formalizing their use in parallel computing architectures for efficient matrix operations (Kung 1982). Unlike general-purpose execution units, systolic arrays exploit spatial and temporal locality by reusing operands as they propagate through the grid. Google’s TPU exemplifies this architectural approach: in the TPUv4, a $128{\times}128$ systolic array of multiply-accumulate units processes matrix operations by streaming data through the array in a pipelined manner (Jouppi et al. 2023). Figure 8 follows these data paths: a control unit feeds input buffers that stream data horizontally into the array, while the partial sums each cell produces flow vertically down to the accumulator chain at the bottom, which collects the finished results. Each processing element performs one multiply-accumulate per cycle and passes its operands to its neighbors, so a value loaded once is reused across an entire row or column rather than refetched from memory.

²⁶ Systolic array: From Greek sustole (“contraction”), borrowed from cardiology to evoke rhythmic pumping. Kung and Leiserson used the term for arrays in which data advances through neighboring processing elements on a regular schedule (Kung and Leiserson 1979). Pipeline fill permits concurrent wavefronts and local reuse. The benefit depends on the stationary operand, tile, buffers, and utilization; it does not guarantee fixed DRAM savings. Irregular workloads and poorly aligned shapes remain a weaker fit because fill, drain, and idle elements consume capacity.

Kung, Hsiang Tsung, and Charles E Leiserson. 1979. “Systolic Arrays (for VLSI).” Sparse Matrix Proceedings 1978 1: 256–82.

Kung, H. T. 1982. “Why Systolic Architectures?” Computer 15 (1): 37–46. https://doi.org/10.1109/mc.1982.1653825.

Figure 8: **Systolic Array Dataflow Architecture**: A pipelined grid of processing elements (PEs) passes operands and partial results between neighbors. The illustrated movement is one possible matrix-multiplication dataflow; designs choose which operand or partial sum remains stationary to match reuse, buffers, and tensor shape. Utilization depends on mapping and dimensions.

The tiling principle: Bridging graph and silicon

A fundamental mismatch exists between the computational graph (which sees a single 4,096 $\times$ 4,096 matrix multiplication) and the physical silicon (which possesses a fixed 128 $\times$ 128 systolic array). Bridging this gap requires tiling: the process of partitioning large tensor operations into “tiles” that fit exactly into the hardware’s fast local memory (SRAM or scratchpad memory).

To process the 4,096-wide worked-example layer on a 128-wide systolic array, the compiler decomposes the result into 1,024 output tiles. Each output tile accumulates across 32 reduction-dimension tiles, so the complete matrix multiplication executes 32,768 $\mathbf{A}$-tile-by-$\mathbf{B}$-tile products. This decomposition is a physical requirement imposed by memory hierarchy limits: operand tiles are fetched from high-capacity, high-latency HBM, staged in fast SRAM, and pulsed through the systolic array.

One extra dimension past the tile width tips utilization off a cliff.

Tile dimensions balance on-chip capacity against arithmetic intensity. A larger tile reuses each loaded byte across more multiply-accumulate operations, raising the kernel’s arithmetic intensity and pushing it toward the compute-bound regime of the roofline model; the capacity ceiling is how much of $\mathbf{A}$, $\mathbf{B}$, and $\mathbf{C}$ fits simultaneously in fast on-chip memory. Tiling sustains high hardware efficiency $(\eta_{\text{hw}})$ by ensuring that for every byte loaded from main memory, the data is reused 128× within the systolic grid. Conversely, violating this alignment breaches the silicon contract: if a layer’s dimensions are not multiples of the tile size (for example, a width of 129 on a 128 array), the system pays a fringe tax in underutilized silicon, where 127 units sit idle while one unit finishes the remainder tile. Algorithm 1 formalizes the resulting loop nest emitted by the compiler to orchestrate this on-chip staging and accumulation.

\begin{algorithm} \caption{Tiled (blocked) matrix multiply} \begin{algorithmic} \Require $\mathbf{A} \in \mathbb{R}^{M\times K}$, $\mathbf{B} \in \mathbb{R}^{K\times N}$; tile sizes $T_M, T_N, T_K$ \Ensure $\mathbf{C} = \mathbf{A}\mathbf{B}$ \For{each row tile $i_0$ (size $T_M$)} \For{each column tile $j_0$ (size $T_N$)} \State initialize the $\mathbf{C}$-tile accumulator to zero on chip \For{each $k_0$ over $K$ in steps of $T_K$} \State load $\mathbf{A}$-tile $[i_0,k_0]$ and $\mathbf{B}$-tile $[k_0,j_0]$ on chip \State accumulate the tile product on chip \Comment{reuse $T_N$/$T_M$ per byte} \EndFor \State write the $\mathbf{C}$-tile back to memory \EndFor \EndFor \end{algorithmic} \end{algorithm}

Once operand tiles reside in fast local SRAM, the hardware executes the tile product without further off-chip memory traffic. Systolic arrays achieve computational efficiency by routing operands through a structured 2D grid of processing elements (PEs) that communicate directly with adjacent neighbors across dedicated registers, bypassing shared-memory bus contention. In a weight-stationary design—the canonical TPU dataflow mapped in figure 8—weights from the $\mathbf{B}$-tile are preloaded into local PE registers and held stationary. Input activations from the $\mathbf{A}$-tile stream horizontally across rows from left to right, while partial sums accumulate vertically down columns. On each clock cycle, every active PE multiplies its incoming activation by its stationary weight, adds the result to the incoming partial sum from above, and passes the activation rightward to its adjacent neighbor for the next cycle.

Because operands propagate one hop per cycle, synchronized execution requires an intentional time skew. Consider a minimal $2{\times}2$ tile product where output element $C_{ij} = A_{i0}B_{0j} + A_{i1}B_{1j}$ requires two products and one accumulation. The weight matrix $\mathbf{B}$ is held stationary across four PEs: $\text{PE}_{00}$ stores $B_{00}$, $\text{PE}_{01}$ stores $B_{01}$, $\text{PE}_{10}$ stores $B_{10}$, and $\text{PE}_{11}$ stores $B_{11}$. If activation $A_{00}$ enters $\text{PE}_{00}$ at cycle 0, its partial product $A_{00}B_{00}$ moves down to $\text{PE}_{10}$ at cycle 1. To combine with this arriving partial sum, activation $A_{01}$ must be injected with a one-cycle delay. Inputs along each successive row and column are staggered so operands meeting at any physical coordinate represent identical reduction indices. Local reuse amortizes the initial fill and drain overhead: each loaded activation is reused across an entire row of $T_N$ processing elements, while each stationary weight serves an entire sequence of input vectors before eviction. Sustained throughput, however, remains bounded by off-chip memory bandwidth whenever operand replenishment cannot keep pace with array retirement (section 1.4.1).

A 128 $\times$ 128 systolic array capable of 16,384 operations per cycle requires a continuous operand stream to maintain utilization. On-chip buffers feed activations and weights to the array edges and are replenished from off-chip memory as needed; reuse within the array avoids fetching every operand from HBM each cycle. The TPU v4’s 1,200 GB/s HBM2 bandwidth helps sustain this stream, but off-chip bandwidth can still limit models whose working sets and reuse patterns exceed on-chip capacity.

The quantization techniques in Quantization and Precision reduce model memory footprint by converting FP32 weights to INT8 representations. Converting 32-bit floating-point weights to 8-bit integers reduces weight traffic by 4$\times$; whether that changes a kernel from bandwidth bound to compute bound depends on the operation’s original arithmetic intensity, the accelerator’s INT8 ridge point (the intensity threshold at which its INT8 compute saturates), and the overhead of quantization and dequantization. Similarly, structured pruning removes entire rows or columns of weight matrices, reducing both the data volume that must traverse memory hierarchies and the computation required. In the D·A·M taxonomy, these algorithmic optimizations prove valuable precisely because they target the memory bottleneck that limits accelerator performance in practice.

Systems Perspective 1.1: Matching architecture to workload

Challenge: Systolic arrays must choose which data to keep stationary (in registers) to minimize movement. This choice hard-codes the hardware’s preference for certain model types. Table 5 previews the three stationary-operand strategies and the workloads each favors; section 1.7 develops each one in full as a general mapping decision.

Table 5: Systolic-Array Dataflow Strategies: Three stationary-operand choices for systolic arrays, the reuse pattern each maximizes, and the workload class that benefits, showing how a fixed dataflow choice hard-codes an accelerator’s affinity for specific models.

Strategy	Stationary item	Optimized for	Example Workload
Weight-stationary	Weights ($\mathbf{W}$)	High Reuse of Weights	CNNs (Conv2D): Filters are small and reused across the entire image.
Output-stationary	Partial Sums ($\mathbf{C}$)	High Reuse of Accumulators	Large Batch MatMul: Accumulating results for many inputs against a large weight matrix.
Input-stationary	Inputs ($\mathbf{A}$)	High Reuse of Activations	Transformers: The same activations feed many weight matrices across attention heads.

No single accelerator architecture suits every workload. A chip optimized for weight-stationary dataflow (like early TPUs) excels at CNNs where filter weights are small and reused across many image patches. That same architecture stalls on large language model (LLM) autoregressive inference at batch size 1, where the entire weight matrix must be read from off-chip memory once per generated token with zero batch reuse, forcing architectures toward output-stationary or hybrid dataflow patterns.

Numerics in AI acceleration

Systolic arrays and Tensor Cores often support reduced-precision arithmetic. FP16 halves the bytes per value relative to FP32, and a target may provision more low-precision multiply-accumulate capacity. A 2$\times$ speedup is not automatic: it requires a supported kernel and sufficient parallel work without another bottleneck. Building on Model Compression, reduced precision is a hardware-software decision. Input and accumulation formats may differ, affecting accuracy, throughput, energy, and movement.

Precision trade-offs

Lower precision is not free. FP16 has a 5-bit exponent and 10 stored fraction bits, giving it more significand precision than BF16 but much less dynamic range than FP32. Its smallest normal value is about $6.1\times10^{-5}$ and its largest finite value is 65,504; subnormals extend lower with reduced precision. BF16 retains FP32’s 8-bit exponent and uses 7 fraction bits, preserving similar dynamic range at lower precision. INT8 has no floating-point exponent and depends on scale, zero-point, and clipping choices. Hardware and software must balance these properties against throughput and bandwidth.

The evolution of AI hardware reflects co-design between numerical methods and hardware capability. Earlier GPU generations lacked the dedicated mixed-precision matrix paths now used for deep learning, even when other units supported several formats. As training and inference methods demonstrated acceptable accuracy with selected lower-precision operations, vendors added native FP16, BF16, and integer tensor paths. Frameworks and libraries expose them through compatible kernels, autocasting, quantization, and calibration. Software gains materialize only when the model selects an efficient path; changing storage alone may reduce bytes without producing the advertised arithmetic speedup.

Precision support is integrated into execution units and memory paths. SIMD and SIMT lanes may support several scalar formats, while Tensor Cores (section 1.3.3) and systolic units (section 1.3.6) expose architecture-specific input and accumulation combinations. Lower precision reduces bytes per operand and may increase operations per cycle. Tiling and dataflow, not precision itself, determine reuse, while conversion, scaling, or fallback overhead can offset part of the gain. A format is useful only when the kernel and numerical policy can employ it safely.

Despite the advantages of reduced precision, deep learning models cannot always rely solely on low-bit representations. To address this challenge, modern AI accelerators implement mixed-precision computing, where different numerical formats are used at different stages of execution. These precision choices affect numerical reliability: matrix multiplications may be performed in FP16 or BF16, while accumulations are maintained in FP32 to prevent precision loss. Similarly, inference engines use INT8 arithmetic while preserving key activations in higher precision when necessary.

Mixed-precision computing

Modern AI accelerators increasingly support mixed-precision execution, allowing different numerical formats to be used at various stages of computation. Training workloads often use FP16 or BF16 for matrix multiplications, while maintaining FP32 accumulations to preserve precision (Micikevicius et al. 2017; Mellempudi et al. 2019). The software implementation of mixed-precision training, including loss scaling techniques and framework support, is covered in Mixed-precision training. Inference workloads, by contrast, optimize for INT8 or even INT4, achieving high efficiency while retaining acceptable accuracy.

Micikevicius, Paulius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, et al. 2017. “Mixed Precision Training.” arXiv Preprint arXiv:1710.03740.

Mellempudi, Naveen, Sudarshan Srinivasan, Dipankar Das, and Bharat Kaul. 2019. “Mixed Precision Training with 8-Bit Floating Point.” arXiv Preprint arXiv:1905.12334.

NVIDIA. 2020b. TensorFloat-32 in the A100 GPU Accelerates AI Training, HPC up to 20x.

NVIDIA Corporation. 2018. NVIDIA Tesla T4 Tensor Core GPU. NVIDIA product documentation.

The shift toward precision diversity is evident in the evolution of AI hardware. Early architectures such as NVIDIA Volta provided limited support for lower precision beyond FP16, whereas later architectures, including Turing and Ampere, expanded the range of supported formats. Table 6 traces this progression: Ampere GPUs introduced TF32 as a hybrid between FP32 and FP16 (NVIDIA 2020b), alongside broader support for BF16, INT8, and INT4 (NVIDIA Corporation 2017, 2018, 2020a).

Table 6: Precision Support Evolution: From Volta through Ampere, Tensor Core support expanded from FP16 to TF32, BF16, and integer formats, broadening the precision choices available to training and inference; the CUDA-core column shows the separate scalar/vector formats each generation supports.

Architecture	Year	Supported Tensor Core precisions	Supported CUDA Core Precisions
Volta	2017	FP16	FP64, FP32, FP16
Turing	2018	FP16, INT8, INT4, INT1	FP64, FP32, FP16, INT8
Ampere	2020	FP64, TF32, BF16, FP16, INT8, INT4	FP64, FP32, FP16, BF16, INT8

Newer architectures incorporate a growing diversity of numerical formats because different workloads bind at different points on the accuracy-throughput-energy trade-off. Precision support is therefore another form of workload matching, not a generic feature checklist.

The precision format used in hardware design has cascading implications across the entire system. Reducing from FP32 to FP16 cuts memory traffic in half, which matters far more than it might seem: because memory access dominates energy consumption, halving memory traffic can substantially reduce energy per inference when data movement is the bottleneck (Horowitz 2014). Simultaneously, Tensor Cores and systolic arrays can pack more lower-precision multiply-accumulate units into the same silicon area, raising peak throughput (Dally et al. 2021; Dally 2023). Lower-precision integer arithmetic also consumes less energy than FP32 in representative technology studies (Horowitz 2014), and the inference-focused TPUv1 was built around 8-bit multiply units (Jouppi et al. 2017). The systems insight is that reduced precision does not merely “save bits”: it simultaneously relieves the memory bandwidth bottleneck and increases compute density, attacking both sides of the roofline at once.

As AI models continue to scale, precision support connects the compute primitive discussion back to the memory wall: lower-bit formats matter when they reduce the bytes moved and keep the hardware’s matrix engines fed. The remaining architectural question is how these execution units, precision formats, and memory paths integrate into complete accelerator systems. Architectural integration determines how efficiently computational primitives become usable accelerator throughput. SIMD lanes, Tensor Cores, and systolic arrays are building blocks, but their full-chip organization varies significantly across AI processors; the choice of execution units, their numerical precision support, and their connectivity shape how effectively hardware can scale for deep learning workloads.

Intra-node interconnects: Scaling the stack

Mastery of the single-machine stack requires understanding how bits move between GPUs and the CPU. In the 1–8 GPU regime, scaling is achieved through high-speed intra-node interconnects such as NVLink and host-to-device PCIe transfers that mitigate the memory wall. These links form a bandwidth taper: data-movement speed falls at each step away from the compute units, from on-package HBM through the GPU-to-GPU NVLink bridge down to the host PCIe link. The PCIe step is much slower than the accelerator-local memory and inter-GPU fabric, so any data path that touches the CPU can become a performance hazard, the “PCIe Wall” that NVLink exists to avoid (NVIDIA Corporation 2020b). Section 1.4.5.1 develops this hierarchy quantitatively, where host-accelerator communication is the operative concern.

Intel Corporation. 2021a. “Intel Advanced Matrix Extensions (Intel AMX).” Intel Architecture Instruction Set Extensions Programming Reference.

Modern AI processors exhibit a range of design trade-offs based on their intended applications, and comparing their configurations reveals how deployment constraints drive architectural divergence. A training-optimized accelerator like the NVIDIA A100 packs many Streaming Multiprocessors with wide SIMD units and FP16 Tensor Cores because training throughput scales with aggregate multiply-accumulate capacity (NVIDIA Corporation 2020a). Google’s TPUv4 makes a radically different bet: just two cores per chip, each containing massive BF16 systolic arrays, a design that trades programmer flexibility for efficiency on dense matrix multiplications (Jouppi et al. 2023). At the inference end, Intel’s Sapphire Rapids dedicates Advanced Matrix Extensions (AMX) tile engines to INT8 and BF16, reflecting the insight from Model Compression that inference models tolerate reduced precision (Intel Corporation 2021a). Mobile neural engines take this further by shrinking matrix engines into low-power SoC blocks, prioritizing energy efficiency per operation over peak throughput. Table 7 compares these architectural configurations.

Table 7: AI Processor Configurations: Modern AI processors prioritize different execution unit characteristics for specific workloads: NVIDIA A100 uses wide SIMD and Tensor Cores for training, Google TPUv4 emphasizes high-throughput BF16 matrix multiplication, Intel Sapphire Rapids focuses on INT8-optimized inference, and mobile NPUs prioritize low-power execution. These variations in SIMD width, Tensor Core size, and processing element count reflect the growing diversity in AI hardware architectures.

Processor	Vector execution	Matrix engine	Processing Elements	Primary workloads
NVIDIA A100	CUDA FP/INT lanes	Tensor Core instruction tiles	108 SMs	Training, HPC
Google TPUv4	Vector unit	$128{\times}128$ BF16 systolic arrays	2 cores/chip	Training
Intel Sapphire	512-bit AVX-512	AMX tiles (up to 16 rows by 64 bytes each)	Up to 60 cores	Inference
Mobile NPU	CPU/GPU/DSP vectors	Small matrix engines	Integrated NPU blocks	Mobile inference

The pattern across these configurations reveals a consistent engineering principle: each design sacrifices generality to optimize for its target workload’s dominant operation and precision. Training chips invest silicon in wide floating-point datapaths; inference chips trade precision for throughput; mobile chips trade throughput for energy efficiency. No single design dominates across all workloads, which is precisely why hardware selection depends on workload analysis rather than headline specifications.

Cost-performance analysis

Architectural specifications define computational potential, but deployment decisions require cost-performance analysis. Raw compute is only one input: delivered performance may be limited by arithmetic throughput, memory bandwidth, interconnect latency, software overhead, or low operational utilization.

The physical energy differential established in section 1.1.5—where moving operands from off-chip DRAM consumes orders of magnitude more energy than performing arithmetic—drives accelerator specialization. Because memory access costs dominate computation, accelerators achieve only a fraction of their peak throughput on memory-bound kernels. Conversely, architectures that maximize data reuse, such as systolic arrays executing dense matrix multiplications, sustain substantially higher utilization by amortizing each memory fetch across multiple arithmetic operations.

Consider an engineering team choosing between deploying a larger cluster of older accelerators or a smaller cluster of newer units. Peak FLOP/s figures mislead whenever dominant kernels are memory-bandwidth or communication bound. In a large model pipeline, dense feed-forward and attention projection layers may remain compute bound, whereas embedding lookups, layer normalization, residual additions, and autoregressive decoding steps are strictly memory bound. The decisive metric is delivered workload throughput per dollar under realistic cluster utilization, memory capacity constraints, available memory bandwidth, and scale-out interconnect efficiency.

These architectural and economic constraints explain the rapid industry adoption of newer accelerators despite substantial increases in nominal unit price. For memory-bound workloads, generational improvements in memory bandwidth and compiler code generation dominate end-to-end execution time. Cloud deployments introduce additional trade-offs: hourly rental rates, dynamic instance utilization, and infrastructure overhead continuously shift the economic break-even point between purchasing on-premises hardware and leasing cloud capacity.

Table 8 provides an illustrative cost worksheet for common accelerators. List prices vary across vendor, region, contract terms, and procurement volume. Furthermore, the throughput rows correspond to distinct numerical precision modes, meaning the price-per-peak-operation metric is meaningful within a single row but invalid as a cross-device benchmark. The table establishes a framework for making economic assumptions explicit rather than offering a universal hardware ranking.

Table 8: Illustrative Accelerator Cost Worksheet: Each row divides an approximate acquisition-price assumption by the representative peak mode shown. Because precision, sparsity assumptions, and price sources differ, the ratios are not comparable as workload performance. Total cost also includes utilization, memory capacity, power, cooling, interconnect, software, and infrastructure; TPU acquisition cost is estimated from cloud pricing.

Accelerator	List price	Representative Peak Throughput (precision shown)	Memory Bandwidth	Price/performance
NVIDIA V100	~$10,000	125 TFLOP/s	900 GB/s	$80/(TFLOP/s)
NVIDIA A100	~$15,000	312 TFLOP/s	2,039 GB/s	$48.1/(TFLOP/s)
NVIDIA H100	~$25,000–30,000	494 TFLOP/s (TF32)	3,350 GB/s	~$50.6/(TFLOP/s)
Google TPUv4	~$8,000*	275 TFLOP/s (BF16)	1,200 GB/s	~$29.1/(TFLOP/s)
Intel Gaudi 2	~$12,000	865 TFLOP/s (FP8)	2,450 GB/s	$13.9/(TFLOP/s)

The worksheet demonstrates why accelerator selection cannot be reduced to a single headline specification. A bandwidth-bound kernel benefits more from the H100’s 3,350 GB/s memory subsystem than from an incremental gain in peak arithmetic, whereas a compute-bound general matrix multiplication (GEMM) shows the opposite sensitivity. Device memory capacity dictates whether working sets fit on-chip without spilling or partitioning, while compiler and driver maturity determines whether the application can actually reach the advertised datapath.

Software framework selection directly impacts these economic trade-offs: graph compilers, operator fusion passes, and runtime dispatchers govern the efficiency with which high-level model descriptions map onto physical execution units. Concrete framework-level optimization strategies are examined in ML Frameworks, while empirical measurement and benchmarking methodologies are developed in Benchmarking.

Vector pipelines, matrix-tile engines, and Tensor Cores provide massive peak arithmetic capability. An NVIDIA A100 advertises 312 TFLOP/s on its FP16 tensor path, and subsequent accelerator designs add narrow-precision FP8 execution modes (NVIDIA Corporation 2020a; Kuzmin et al. 2022; Micikevicius et al. 2022). NVIDIA’s Blackwell (B200) architecture extends this trajectory with an FP4 tensor datapath, rated at up to 9 PFLOP/s dense or 18 PFLOP/s sparse peak throughput per chip (NVIDIA Corporation 2024). Reaching those theoretical ceilings requires numerical representations, kernel implementations, and sparsity patterns that align exactly with the specialized hardware paths. Narrower numerical formats expand the available design space, but numerical stability and dynamic range requirements dictate whether a workload can safely utilize FP4.

Kuzmin, Andrey, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort. 2022. “FP8 Quantization: The Power of the Exponent.” Advances in Neural Information Processing Systems (NeurIPS), 14651–62. https://doi.org/10.52202/068431-1065.

Micikevicius, Paulius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, et al. 2022. “FP8 Formats for Deep Learning.” arXiv Preprint arXiv:2209.05433.

Dividing a model’s total operation count by peak arithmetic throughput produces an idealized lower bound on execution time, not a realistic inference latency. On an A100, dividing the arithmetic operations in ResNet-50 by 312 TFLOP/s yields a microsecond-scale estimate. Real ResNet-50 inference, however, requires milliseconds. This orders-of-magnitude discrepancy exposes the central tension in machine learning systems: arithmetic throughput has far outpaced the memory bandwidth required to feed the execution units. Moving data from memory costs orders of magnitude more energy than arithmetic, and memory bandwidth has improved more slowly than tensor arithmetic throughput. This physical imbalance dictates whether that advertised 312 TFLOP/s delivers high sustained hardware utilization or remains largely idle on a given workload (Horowitz 2014; Gholami et al. 2024).

Resolving this performance gap requires analyzing the memory hierarchies that supply the compute units. The memory subsystem is not passive scaffolding; in conjunction with arithmetic intensity and kernel scheduling, it determines the fraction of peak compute an algorithm can extract from the machine.

Self-Check: Question

In NVIDIA’s Ampere and Hopper architectures, how does 2:4 structured sparsity achieve an up to $2\times$ theoretical speedup in Tensor Core matrix multiplication?
1. Exactly two non-zero values are preserved in every contiguous four-element block, allowing weights to be stored in half the memory with 2-bit index metadata while sparse Tensor Cores perform math only on non-zeros
2. Every alternate row of the weight matrix is dropped completely, allowing the GPU to halve the grid launch dimensions
3. Four separate threads simultaneously execute one scalar multiply-accumulate instruction in a single clock cycle
4. Floating-point numbers are converted to 2-bit integers, quadrupling register file capacity
Contrast the data movement mechanics and primary use cases of Weight-Stationary (WS) and Output-Stationary (OS) systolic array dataflows.
Place the following steps in the correct execution sequence for processing a matrix multiplication on an accelerator with Sparse Tensor Cores using 2:4 structured sparsity:

Fine-tune or prune the weight matrix to ensure exactly two non-zero values exist in every four-element contiguous group
Sparse Tensor Cores load compressed weights and decode metadata to gather matching input activation elements
Multiply non-zero weights by gathered activations and accumulate into output partial sums at $2\times$ dense throughput
Compress the sparse weight matrix by storing only the non-zero values alongside 2-bit per-value selection metadata

What is the primary difference between the FP8 E4M3 and FP8 E5M2 numerical formats used in modern AI accelerators (such as NVIDIA Hopper and Ada Lovelace)?
1. E4M3 uses 4 sign bits and 3 exponent bits, whereas E5M2 uses 5 sign bits and 2 exponent bits
2. E4M3 has 4 exponent bits and 3 mantissa bits providing higher precision for forward-pass activations/weights, whereas E5M2 has 5 exponent bits and 2 mantissa bits providing wider dynamic range for gradients
3. E4M3 is exclusively an integer fixed-point format, whereas E5M2 is a standard IEEE floating-point format
4. E4M3 requires twice as many memory bytes per element as E5M2
True or False: In NVIDIA’s SIMT (Single Instruction, Multiple Threads) execution model, when threads within the same 32-thread warp execute divergent branches of an if-else condition, both paths are executed concurrently in parallel at full hardware throughput.
Describe the Tiling Principle in deep learning hardware mapping and explain why multi-level hierarchical tiling (from global memory down to registers) is necessary for high-throughput GEMM kernels.

See Answers →

AI Memory Systems

ResNet-50 can expose the gap between accelerator arithmetic and memory: convolution weights, activations, and intermediate results still have to arrive on time. Modern accelerators reach hundreds of TFLOP/s or more on selected low-precision paths (NVIDIA Corporation 2020a, 2024; Choquette 2023), but a kernel cannot use that capacity when its required traffic exceeds the relevant memory bandwidth. The AI memory wall names this growing mismatch between arithmetic demand and the memory system that feeds it.

Unlike conventional workloads, ML models require frequent access to large volumes of parameters, activations, and intermediate results, leading to substantial memory bandwidth demands. This challenge intersects with the data management strategies covered in Data Engineering. Modern AI hardware addresses these demands through advanced memory hierarchies, efficient data movement techniques, and compression strategies that promote efficient execution.

Understanding the AI memory wall

Definition 1.4: AI memory wall

The AI memory wall is the ML accelerator performance constraint that arises when arithmetic throughput $(R_{\text{peak}})$ outpaces memory bandwidth $(\text{BW})$. The core issue is whether the memory system can supply operands and absorb results quickly enough to sustain the specialized primitives introduced in section 1.3.

Significance: A workload has reached the memory wall when increasing FLOP/s alone no longer reduces execution time because the $\frac{D_{\text{vol}}}{\text{BW}}$ term dominates the overall runtime.
Distinction: Unlike a general-purpose memory wall, which affects all computing, the AI memory wall is driven by the large parameter volumes and intermediate activation tensors required by deep learning.
Common pitfall: A frequent misconception is that the memory wall is “fixed” by more memory. In reality, it is a bandwidth-latency gap: even with infinite capacity, the speed of moving data between memory and compute remains the fundamental physical bottleneck.

Data access can dominate the energy budget of a memory-intensive kernel.²⁷ Figure 9 compares representative operation costs from one technology study, revealing the multi-order-of-magnitude gap between simple local arithmetic and an off-chip DRAM access.

²⁷ Von Neumann bottleneck: Separating storage from compute requires instructions and data to cross interfaces rather than remain in local registers. In the representative 45 nm estimates from Horowitz (2014), a DRAM access consumes over 20,000$\times$ the energy of an INT8 addition. The exact ratio changes with technology and operation, but locality remains a first-order design concern.

Horowitz, Mark. 2014. “1.1 Computing’s Energy Problem (and What We Can Do about It).” 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 10–14. https://doi.org/10.1109/isscc.2014.6757323.

Figure 9: **The Energy Hierarchy**: Representative 45 nm energy estimates on a logarithmic scale from Horowitz (2014). An off-chip DRAM access costs about 128$\times$ an 8 KB SRAM read and over 20,000$\times$ an INT8 addition in this model. The values are not universal device specifications; they illustrate why reuse in registers and SRAM can matter more than reducing arithmetic count alone.

Quantifying the compute-memory performance gap

The energy disparity that figure 9 captures grows more severe with each hardware generation. Over the past two decades, peak computational capabilities have grown substantially faster than DRAM bandwidth (Gholami et al. 2024). This divergence stems from physical scaling constraints: while transistor scaling and dense systolic arrays multiply arithmetic throughput, memory bandwidth remains bounded by pin counts, package-substrate routing density, and interface signaling power. Representative high-end accelerators can deliver on the order of $10^3$ TFLOP/s of peak tensor throughput (for example, NVIDIA H100 delivering 989 TFLOP/s in FP16 or nearly 2,000 TFLOP/s in dense FP8) while providing approximately 3.35 TB/s of memory bandwidth (Choquette 2023). This implies that on the order of $10^2$ FLOP of work per byte moved is required to fully use the compute, which can exceed the arithmetic intensity of many practical neural network workloads.

Gholami, Amir, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, and Kurt Keutzer. 2024. “AI and Memory Wall.” IEEE Micro 44 (3): 33–39. https://doi.org/10.1109/mm.2024.3373763.

When a workload’s arithmetic intensity falls below this threshold, the execution budget shifts entirely to data transfer ($\frac{D_{\text{vol}}}{\text{BW}}$). Accelerators attempt to tolerate memory latency—typically hundreds of clock cycles for off-chip DRAM—by maintaining thousands of concurrent execution threads and switching between active warps while memory requests remain in flight. However, when memory bandwidth saturates or when register pressure limits occupancy, the scheduler runs out of independent instructions to issue. Compute pipelines then stall, leaving high-throughput tensor cores idle while waiting for operands. Simultaneously, driving wide off-chip memory buses across capacitive printed circuit board traces and PHY interfaces consumes a substantial fraction of the chip’s TDP, directly constraining the energy budget available for arithmetic logic (Horowitz 2014; Sze et al. 2017).

Hardware balance ($I_{\text{ridge}}$): The paradigm partition

Different devices place the compute-bandwidth boundary at different intensities. The hardware balance $(I_{\text{ridge}})$ is the ratio of peak arithmetic throughput to peak memory bandwidth. In the roofline model, it is the ridge point where the compute and bandwidth ceilings intersect: \[ I_{\text{ridge}} = \frac{R_{\text{peak}}}{\text{BW}} \]

The ratio sets a device-specific boundary. Depending on the chosen precision, a high-end accelerator such as H100 may require roughly $150$–$300$ FLOP/byte to reach its arithmetic ceiling, whereas a microcontroller may have a much lower ridge point. The same kernel can therefore be bandwidth bound on a compute-dense accelerator and compute bound on an edge processor. Hardware balance does not label either device as universally efficient; it identifies how much reuse a workload needs on that device.

Peak computational capability has expanded faster than memory bus performance across accelerator generations, and the size of that mismatch is what sets how much data reuse a kernel needs to stay fed. Figure 10 separates the two growth rates that produce the ridge point, each normalized to the V100 baseline, so the widening distance between them can be read directly.

Figure 10: **The Compute-Bandwidth Divergence**: Peak dense FP16 throughput and memory bandwidth across five NVIDIA datacenter generations, each normalized to its V100 baseline on a log scale. Over 2017–2024 compute grows 18× while bandwidth grows 8.9×, so the shaded separation between the curves widens by 2.0×. That separation is the data reuse a kernel must supply to keep the arithmetic units busy. The H200 step is the instructive exception: bandwidth rose while peak compute held flat, narrowing the gap for one generation.

The threshold required for a kernel to become compute bound has shifted across accelerator hardware generations. Across successive data center architectures (figure 11), hardware ridge points climb from V100 through B200, steadily increasing the arithmetic intensity required to saturate peak compute throughput.

Figure 11: **The Rising Ridge**: Hardware arithmetic intensity (FLOP/byte) over time computed as each chip’s dense FP16 tensor peak divided by its memory bandwidth. This is the arithmetic intensity a kernel must reach before compute rather than bandwidth limits it. As compute capability grows faster than memory bandwidth, the ridge point rises from V100 through H100 and remains high on B200. This trend explains why architectures with high data reuse flourish while low-reuse workloads face a growing hardware tax.

Beyond performance limitations, memory access imposes a steep energy cost. Fetching data from off-chip DRAM consumes orders of magnitude more energy than performing arithmetic operations on-chip (Horowitz 2014). An arithmetic unit drives signals over micrometer-scale wires on silicon, consuming tens of femtojoules per operation; fetching an operand from DRAM requires driving macroscopic PCB traces and package interconnects, consuming tens of picojoules. In machine learning models where billions of parameters and activations stream across the bus, data movement quickly dominates the total power envelope. This energy differential drives architectural decisions: Google’s TPUv1 achieved 30–80$\times$ better performance per watt than contemporary CPUs and GPUs on inference benchmarks by minimizing off-chip data movement through systolic arrays and large on-chip SRAM (Jouppi et al. 2017). These design choices demonstrate that energy budgets, as much as cycle budgets, dictate what architectures can execute in production.

Memory access patterns in ML workloads

Beyond raw computational throughput, an accelerator’s sustained efficiency depends on supplying operands to execution units without pipeline stalls. Deep neural networks impose three concurrent demands on this data supply. First, model parameters (weights and biases) may reach hundreds of billions of values (Brown et al. 2020). In autoregressive generation at batch size 1, every parameter must be streamed from memory for each generated token, collapsing arithmetic intensity to roughly 1 FLOP/byte. Second, intermediate activations produced at each layer must be buffered in memory for subsequent operations; during training, deep architectures must retain these activations across the entire forward pass until backpropagation consumes them. Third, training adds gradient and optimizer traffic: backpropagation computes gradients for every parameter, while algorithms such as Adaptive Moment Estimation (Adam) read and write multi-word momentum states, multiplying data movement volume per parameter update (Narayanan et al. 2021; Kwon and Rhu 2018).

Kwon, Youngeun, and Minsoo Rhu. 2018. “TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learning.” Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, 740–53. https://doi.org/10.1145/3352460.3358284.

The physical cost of this data movement depends on where each tensor resides in the memory hierarchy. While an on-chip register access completes in a single clock cycle at sub-picojoule energy, an off-chip DRAM fetch stalls execution for hundreds of cycles and expends orders of magnitude more energy. Tracing a single tensor through the memory tiers during an inference pass makes this disparity concrete, as demonstrated by the keyword spotting (KWS) pipeline in lighthouse 1.2.

Lighthouse 1.2: Life of a tensor: GPU-hosted KWS

Recall the Keyword Spotting lighthouse summarized in Five Lighthouse Models. If the same one-second audio clip is batched on a GPU-hosted inference path, its physical journey through the memory hierarchy looks like this:

DRAM (HBM): The tensor starts here.
- Size: 16,000 samples $\times$ 2 bytes (FP16) = 32 KB.
- Latency: Fetching this from off-chip memory takes ~300 ns (plus queuing delay).
- Energy: Cost is ~20 pJ/bit. High cost.
L2 cache: Device-memory requests may be served here when the requested cache lines are resident.
- Latency: ~4 ns.
- Access: Shared across multiple Streaming Multiprocessors (SMs).
L1 cache/shared memory: A specific SM works on a tile of the audio, using hardware-managed cache or explicitly staged shared memory.
- Latency: ~1 ns.
- Locality: If an L1 request misses, it can still be served by L2 before an HBM access is required.
Registers: The Tensor Core operates here.
- Latency: Typically a small number of cycles, depending on the instruction and architecture.
- Throughput: 312 TFLOP/s.
- Energy: Cost is ~0.1 pJ/bit.

Systems insight: Performance is bounded by the lower of peak compute throughput and bandwidth times arithmetic intensity. Which memory link supplies the relevant bandwidth depends on where the working set resides; HBM-to-L2 is one possible limiting path, not the roofline’s sole determinant.

The boundary between memory-bound and compute-bound execution emerges directly from the balance between data transfer time and arithmetic execution time. For a workload requiring data volume $D_{\text{vol}}$ (bytes) across available memory bandwidth $\text{BW}$ (bytes/s), and executing $O$ floating-point operations on hardware with peak throughput $R_{\text{peak}}$ (FLOP/s) at realized hardware utilization $\eta_{\text{hw}}$, the respective execution times are: \[\begin{gather*} T_{\text{mem}} = \frac{D_{\text{vol}}}{\text{BW}} \\ T_{\text{compute}} = \frac{O}{R_{\text{peak}} \cdot \eta_{\text{hw}}} \end{gather*}\]

In a strictly sequential model without overlap, $T_{\text{mem}} > T_{\text{compute}}$ identifies memory-bound execution. When hardware overlaps memory transfers with computation, the larger term sets the theoretical lower bound on execution time. In practice, however, data dependencies, memory controller queueing, and partial pipeline overlap dictate how much arithmetic hardware remains idle while waiting for operands.

The structural widening between model footprints and available memory bandwidth is what turns $T_{\text{mem}}$ into the binding constraint across generations. Figure 12 quantifies this disparity across public milestone models and accelerator architectures (Krizhevsky et al. 2012; Brown et al. 2020; Chowdhery et al. 2022; Dubey et al. 2024). Even high-bandwidth accelerators such as NVIDIA B200 and AMD MI300X/MI325X cannot close this growth-rate gap through bandwidth scaling alone (NVIDIA Corporation 2024; AMD 2023). Parameter counts for proprietary systems such as GPT-4 and Gemini are not officially disclosed, so they are not plotted as factual data points.

Chowdhery, Aakanksha, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, et al. 2022. “PaLM: Scaling Language Modeling with Pathways.” arXiv Preprint arXiv:2204.02311.

Dubey, Abhimanyu, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783.

NVIDIA Corporation. 2024. NVIDIA Blackwell Architecture. NVIDIA product documentation.

AMD. 2023. AMD Instinct MI300X Accelerators. AMD product documentation.

Figure 12: **Model Size vs. Hardware Bandwidth**: Both series are the base-10 logarithm of growth relative to 2014, so each unit on the vertical axis is a tenfold increase. Over the plotted decade parameter counts peak at roughly 4,900$\times$ against bandwidth’s 17$\times$. The shaded band is a gap between logarithms rather than a difference, so its width reads as a ratio, and the two fitted trends diverge at a constant rate.

Irregular memory access

Production machine learning workloads combine regular dense matrix kernels with irregular memory access patterns that defeat conventional caching and prefetching hardware. While dense operations like GEMM and convolution stream contiguous blocks that map cleanly onto hardware caches and systolic arrays, irregular operations rely on indirect, sparse, or input-dependent memory addresses. Table 9 contrasts these two memory regimes across access geometry, cache locality, and hardware pressure.

Table 9: Regular and Irregular Memory Access: Regular dense kernels expose tileable reuse and predictable addresses, whereas irregular ML components rely more on indirect or input-dependent access, weakening caching and prefetching and adding metadata or load-balance overhead. A single model may contain both regimes.

Feature	Regular dense kernels	Irregular ML components
Access pattern	Contiguous or tiled	Indirect, scattered, or input-dependent
Cache locality	Often high after tiling	Often lower and workload-dependent
Data reuse	Structured across tiles or batches	Sparse or dynamic, depending on the operator
Dependencies	Static loop nests	Input-dependent routing or variable shapes
Examples	GEMM, convolution, dense attention	Embedding lookup, unstructured sparsity, MoE routing
Primary pressure	Compute or bandwidth, depending on arithmetic intensity	Latency, bandwidth, metadata, and load imbalance
Energy consequence	Depends on reuse and memory tier	Extra movement and metadata can increase energy

Within a single neural network, memory access behavior varies sharply across layer types. Convolutional layers exhibit high spatial locality and register reuse as small weight kernels slide across input feature maps. Fully connected layers and dense attention execute regular matrix multiplications, but their large parameter matrices or key-value caches can exceed on-chip SRAM capacity and become bandwidth bound when operational reuse is low. Variable sequence lengths complicate memory buffer allocation, but the attention operations within each sequence remain regular, contiguous matrix multiplies.

True memory irregularity arises from three distinct architectural patterns: unstructured sparsity,²⁸ embedding lookups, and dynamic token routing. In unstructured sparsity, nonzero weights or activations are stored in compressed formats (such as CSR or COO) that require pointer indirection to decode operand coordinates. In embedding tables, sparse categorical IDs select arbitrary rows from tables spanning tens of gigabytes in DRAM, transforming sequential memory transfers into uncoalesced, scattered reads. In Mixture-of-Experts (MoE) models, routing gates direct tokens to different expert networks based on input context; while deterministic for a given input, token assignments vary across batches, disrupting prefetch pipelines and unbalancing execution queues across parallel cores.

²⁸ Sparsity and memory irregularity: Unstructured sparse formats commonly gather nonzero elements through indirect addressing, which can weaken sequential access and make metadata overhead significant. Structured sparsity preserves hardware-friendly blocks, so its access behavior differs from arbitrary pruning. Without suitable kernels or hardware support, an unstructured sparse model can run slower than its dense counterpart despite performing fewer arithmetic operations.

Irregularity degrades performance through distinct mechanisms, and treating them as one memory problem can point to the wrong optimization. Scattered addresses may turn one logical access into several memory transactions, so each fetched cache line carries few useful payload bytes. Indirection can also serialize address generation: the system must fetch an index or pointer before it knows which payload to request, limiting memory-level parallelism even when aggregate bandwidth remains available. Input-dependent routing creates a third failure mode when some processing elements receive longer queues than others; completion time follows the slowest queue while neighboring units idle. Variable-sized allocations may fragment capacity, but fragmentation is neither necessary nor sufficient for irregular access. A sequential stream can saturate bandwidth without being irregular, whereas a dependent lookup chain can stall far below peak bandwidth.

Diagnosis should therefore begin with the mechanism that prevents forward progress. A low ratio of useful payload bytes to transferred bytes indicates poor spatial utilization or coalescing. Long memory stalls alongside unused bandwidth indicate dependent misses or too few concurrent requests to hide latency. Uneven queue depths or per-route work reveal load imbalance rather than a cache-capacity problem. Cache hit rate alone cannot distinguish these cases because a miss can waste bandwidth, expose latency, or both. Each diagnosis implies a different remedy: packing or blocking improves spatial utilization, reordering and prefetching expose concurrent requests, structured sparse formats reduce indexing overhead, and routing or partitioning balances work. The memory hierarchy determines the cost of the remaining accesses, while dataflow and tensor-layout choices determine how much irregularity software can remove before execution reaches the hardware.

The optimization order follows that diagnosis. Representation comes first when the format is under system control because a block-structured sparse format or grouped routing can remove indirection before a kernel executes. Request organization comes next: accesses issued together should target adjacent addresses, and enough independent requests must remain in flight to hide unavoidable latency. Additional cache capacity or bandwidth helps only when profiling shows that the regularized request stream still exhausts that resource. More bandwidth cannot resolve a serial pointer dependency, and more capacity cannot rebalance skewed work. The objective is to expose as much regular structure as possible while confining unavoidable irregularity to the smallest portion of execution.

Software can mitigate irregular access by regularizing tensor layouts, enforcing structured sparsity, and sorting dynamic routes into uniform batches. Yet software restructuring cannot eliminate the physical penalty of data movement: whether a memory request is contiguous or scattered, every transferred byte must cross physical wires governed by finite bandwidth, propagation latency, and energy dissipation. Bounding these physical costs requires organizing storage into cascading hardware tiers. Rather than treating memory as a uniform pool, accelerator architectures deploy multilevel memory hierarchies that trade off capacity against latency and energy at every tier.

Memory hierarchy

No single storage technology can simultaneously satisfy the multi-terabyte-per-second bandwidth required by dense execution units and the tens of gigabytes of capacity required to hold modern model parameters. Accelerators resolve this tension through multilevel memory hierarchies. Rather than treating memory as a monolithic address space, the machine organizes storage into distinct physical tiers, each optimized for a specific balance of capacity, access latency, bandwidth, and energy dissipation.

At the apex of the hierarchy, register files and on-chip SRAM caches supply operands to arithmetic units with sub-nanosecond latency and single-digit picojoule energy costs, but their capacity is strictly limited by silicon die area. At the base, nonvolatile flash storage holds hundreds of gigabytes of model checkpoints and training datasets, but requires milliseconds or microseconds to access. Between these boundaries, software-managed scratchpads stage active tensor tiles, while device-local DRAM (such as HBM or GDDR) provides the primary working memory for parameters and activations.

This organization establishes an inverse relationship across tiers: as storage capacity expands and physical distance from arithmetic units grows, bandwidth falls, latency lengthens by orders of magnitude, and energy consumption per bit escalates. Table 10 compares these tiers across latency, bandwidth, capacity, and functional roles in deep learning workloads.

Table 10: Memory Hierarchy Trade-Offs: AI accelerators use a multilevel memory hierarchy to balance performance and capacity. Each level provides distinct latency, bandwidth, and capacity characteristics that dictate how neural network components (weights, activations, and intermediate results) should be allocated to minimize bottlenecks and maximize throughput. Relative labels describe the typical ordering; exact values depend on architecture. Latencies every programmer should know supplies registry-backed reference latencies.

Memory Level	Relative latency	Bandwidth	Relative capacity	Example Use in Deep Learning
Registers	Lowest	Highest	Tiny	Storing operands for immediate computation
L1/L2 Cache (SRAM)	Very low	High	Small	Caching frequently accessed activations and small weight blocks
Scratchpad memory	Very low	High	Small to medium	Software-managed storage for intermediate computations
Device DRAM (HBM, GDDR, or LPDDR)	High	Moderate–Very High	Large	Storing model parameters and activations that do not fit on-chip
Host DRAM (typically DDR)	High locally; transfer adds latency	Moderate	Large to very large	Staging or offloading data outside accelerator memory
Flash Storage (SSD/NVMe)	Very high	Low	Very large	Storing pretrained models and checkpoints for later loading

A recurring systems question is why accelerators cannot bypass on-chip SRAM entirely and stream all operands directly from high-capacity off-chip DRAM. The physical constraint preventing this design is the latency floor imposed by chip geometry, electrical signal propagation, and DRAM cell access dynamics.

Napkin Math 1.2: The speed of light limit

Problem: Why is on-chip SRAM necessary instead of fetching all data from HBM?

Math:

Distance: On an H100-class 814 mm² die, signals travel ~20 mm.
Propagation latency: At $\approx 0.5c$ in silicon, $20\text{ mm}/(0.5c) \approx 130\text{ ps}$.
Clock cycle: At 2 GHz, $1/(2\text{ GHz}) = 500\text{ ps}$.
DRAM: Off-chip HBM sits millimeters away on the package, but DRAM access latency plus protocol overhead = 100+ cycles.

Systems insight: The 20 mm propagation estimate is about 130 ps, less than one 500 ps clock cycle, so distance alone does not prove a one-cycle access impossible. A real HBM access includes DRAM-array latency, serialization, protocol traversal, interconnect delay, and queuing, producing this 100-plus-cycle access path. Local registers and SRAM are therefore required to feed compute units at 2 GHz. For transformer models, this complete access path is why weights must be staged in SRAM in tiles rather than read on demand from HBM, and why repeatedly streaming an entire KV cache can collapse inference throughput when its memory traffic cannot be hidden behind arithmetic.

On-chip memory

The on-chip memory subsystem integrates registers, Static Random-Access Memory (SRAM) caches, and software-managed scratchpads directly on the accelerator die. Register files provide the immediate operand and accumulator storage needed by execution pipelines. Modern accelerators provision substantial register capacity in aggregate across the chip, but partition this pool dynamically among concurrent threads or warps. Increasing per-thread register allocation allows individual threads to retain larger matrix tiles, but limits the total number of concurrently scheduled warps, reducing the hardware’s ability to hide memory latency. Conversely, underallocating registers forces the compiler to spill values to off-chip memory, converting single-cycle register accesses into multi-cycle stalls. Efficient kernel design balances register allocation to maintain maximum occupancy while keeping hot operands resident.

Hardware-managed L1 and L2 caches provide fast buffering for frequently referenced data using automatic tag checking and hardware eviction policies. However, automatic caches are poorly suited for streaming multi-gigabyte tensor operations: scanning through large activation tensors can evict reusable weight tiles before arithmetic units can reuse them. Furthermore, specialized execution pipelines impose strict layout and data-type constraints that general caches cannot satisfy automatically. As illustrated in example 1.3, hardware matrix engines require specific alignments and memory contracts; violating these requirements causes execution to fall back to unaccelerated paths.

Example 1.3: The Tensor Core contract

Scenario: An engineering team migrates a transformer workload to NVIDIA A100 GPUs expecting automatic Tensor Core acceleration, but profiling reveals Tensor Cores remain idle (NVIDIA Corporation 2020a).

Diagnosis: The workload used shapes and data types that selected less efficient kernels rather than the intended Tensor Core path.

Systems lesson: Specialized paths have operator, precision, layout, and shape requirements. Libraries can often handle nonaligned dimensions through padding or alternative kernels, but reaching high utilization requires verifying which path was actually selected.

To avoid the eviction overhead and unpredictability of automatic caches, AI accelerators incorporate scratchpad memory. Unlike hardware caches, scratchpads are software-managed SRAM structures mapped directly into the core’s address space without tag directories or hardware replacement logic. The compiler or programmer explicitly controls when tensor tiles are loaded from external DRAM, how long they reside on-chip, and when updated values are written back. In convolutional models, scratchpads retain input feature maps and weight kernels across sliding receptive windows, eliminating redundant DRAM reads (Chen et al. 2017). On NVIDIA GPUs, scratchpad memory is exposed as shared memory: a fast, programmer-controlled SRAM bank shared across all threads within a thread block. Custom kernels written in CUDA or Triton manage shared memory explicitly. FlashAttention demonstrates the power of this mechanism: rather than materializing the full $S{\times}S$ attention score matrix in off-chip HBM, the kernel tiles queries, keys, and values through shared memory and updates softmax statistics online (Dao et al. 2022). The speedup stems entirely from eliminating high-latency HBM round trips rather than reducing arithmetic operations; in fact, the algorithm recomputes intermediate attention values during the backward pass to avoid storing them to off-chip memory.

Off-chip memory

When tensor working sets exceed on-chip SRAM capacity, operands and parameters reside in device-local DRAM. Rather than sequential hierarchy levels, High Bandwidth Memory (HBM), Graphics DDR (GDDR), and Low-Power DDR (LPDDR) represent alternative memory architectures engineered for distinct cost, bandwidth, and power envelopes. High-performance accelerators use HBM, which stacks DRAM dies vertically using through-silicon vias (TSVs) and interfaces with the accelerator die across a wide silicon interposer (e.g., 1,024 to 8,192 bits). This wide bus delivers multi-terabyte-per-second bandwidth at lower clock frequencies and lower energy per bit than conventional DRAM, but requires complex 2.5D packaging. Where silicon interposers are cost-prohibitive, accelerators adopt GDDR, which uses narrower buses (e.g., 256 to 384 bits) driven at high pin speeds across standard circuit boards. Mobile and embedded accelerators utilize LPDDR, trading raw bandwidth for low operating voltage and reduced standby power.

When parameter and activation footprints exceed device-local DRAM capacity, the system must offload state to host DRAM (typically standard DDR). Accessing host memory requires crossing an I/O interconnect such as PCIe or a coherent fabric like NVLink or Compute Express Link (CXL). Because PCIe bandwidth (tens of gigabytes per second) is one to two orders of magnitude lower than accelerator DRAM bandwidth, unmanaged offloading stalls arithmetic execution. Efficient runtimes mitigate this penalty by overlapping asynchronous Direct Memory Access (DMA) copies with kernel execution, prefetching upcoming layer parameters while current layers compute.

At the base of the hierarchy, nonvolatile flash storage and solid-state drives (SSDs) provide persistent, multi-terabyte capacity for datasets and training checkpoints (Narayanan et al. 2021). With read latencies measured in tens of microseconds and bus bandwidths capped at single-digit gigabytes per second, flash devices cannot participate directly in kernel execution. Checkpointing and data-loading pipelines treat flash storage as an asynchronous streaming source, staging batches through host DRAM into accelerator memory before computation begins.

The physical tiers of the memory hierarchy establish the latency floors and capacity ceilings of the accelerator. How closely an application approaches hardware peak depends on whether software can sustain high utilization across each tier: advertised memory bandwidth represents only a theoretical limit, while achievable throughput is determined by access regularity, tensor layout, and reuse locality.

Memory bandwidth and architectural trade-offs

Advertised memory bandwidth is only a ceiling; achievable bandwidth depends on access pattern, batching, locality, and the host interface that feeds the accelerator. Representative data center accelerators provide memory bandwidth on the order of a few TB/s, paired with tens of GB of high-bandwidth memory. Peak figures assume continuous, stride-1 coalesced access across all memory channels. Real neural network workloads deviate substantially from this ideal: transformer attention, convolution, and fully connected layers realize distinct fractions of peak bandwidth because their memory access strides, tiling dimensions, and operational reuse differ. Fully connected layers approach peak bandwidth only when batch sizes are large enough to amortize the cost of loading weight matrices—connecting directly to the batch-size sensitivity analyzed under the roofline model in section 1.5. When batch sizes are small, execution is throttled by memory latency and underutilized channels, leaving effective throughput well below advertised ceilings.

On-chip memory access typically consumes energy in the single-digit-to-tens of picojoules per access, while external DRAM can be on the order of hundreds of picojoules per access—an order-of-magnitude energy penalty (section 1.4.1). Because driving off-chip physical interfaces dominates the thermal and power budget, accelerators structure their execution pipelines around spatial data reuse. Rather than repeatedly round-tripping tensors through DRAM, architectures adopt specific stationary dataflows. In a weight-stationary design, model parameters remain fixed inside processing element registers while input activations stream through, amortizing parameter loads across multiple tokens or spatial positions. In an input-stationary design, activations are buffered locally while weights stream through, maximizing reuse across kernel filters. In an output-stationary design, partial sums accumulate directly inside local accumulators until final reductions complete, eliminating intermediate round-trips of partially computed feature maps to external DRAM.

Physical constraints at the silicon and packaging boundaries govern how memory bandwidth scales across accelerator classes. HBM achieves multi-terabyte-per-second throughput by stacking DRAM dies vertically using through-silicon vias and interfacing with the processor over a silicon interposer. This packaging provides wide parallel interfaces—reaching on the order of 1 TB/s in mainstream products and a few TB/s in high-end systems—at substantially higher packaging cost and thermal density. TPU-class designs pair high-bandwidth memory with large on-chip buffers and 2D systolic arrays, trading general-purpose memory flexibility for tight spatial reuse on dense matrix multiplications. At the opposite extreme, mobile SoC designs operate within a few-watt power envelope and cannot accommodate the cost or thermal dissipation of interposer-based stacks. These edge processors rely on unified commodity LPDDR architectures, delivering on the order of hundreds of GB/s of shared bandwidth that must be strictly scheduled across CPU, GPU, and neural accelerator cores to prevent memory bus contention and thermal throttling.

Bandwidth limitations extend beyond the accelerator package to the host-device interface. When an input must cross a 64 GB/s PCIe link before using an accelerator with 2 TB/s of HBM bandwidth, the interface has roughly 30$\times$ less bandwidth and can dominate small, frequent transfers (NVIDIA Corporation 2020a, 2020b). For low-latency inference queries or fine-grained kernel offloading, transfer serialization across the host bus can easily outlast kernel computation. Mitigating this interface bottleneck requires keeping model parameters and intermediate activation states resident in accelerator memory across execution steps, or bypassing the host link entirely through direct peer-to-peer storage and network data paths.

Host-accelerator communication

In discrete accelerator systems, the host CPU and the accelerator operate as separate computational domains joined by a peripheral bus. The host manages storage I/O, network streams, and runtime orchestration, while the accelerator executes dense mathematical kernels over high-bandwidth device memory. Because CPU memory (system DRAM) and accelerator memory (HBM or GDDR) occupy distinct physical address spaces, computation cannot begin until tensors traverse the host-accelerator interconnect. Host-accelerator data movement therefore establishes an operational boundary: if transfers over the peripheral bus cannot feed the accelerator’s compute cores at line rate, the device idles regardless of its raw FLOP/s capacity.

Figure 13 traces the four-phase execution sequence that governs this host-device interaction. Before computation begins, input tensors must be copied from host DRAM across the PCIe bus to accelerator memory (step 1) via DMA.²⁹ The CPU then submits kernel launch commands to the accelerator command queue (step 2), prompting the compute units to process tiles in parallel out of local memory (step 3). Once execution completes, the accelerator stores its output tensors in device memory and copies the results back across PCIe to host DRAM (step 4). In a naive synchronous execution model, every transfer step sits directly on the critical path, stalling either the host or the accelerator until each memory transaction finishes.

²⁹ DMA (direct memory access): A dedicated hardware unit that manages the data copy (step 1) without direct CPU management, freeing the CPU to immediately issue computation commands (step 2). This concurrency is critical: without it, the accelerator can idle between compute batches, especially when host-to-device movement is on the critical path.

Figure 13: **Host-Accelerator Data Transfer Sequence**: Four-step execution flow between CPU host and GPU device: (1) copying input data over PCIe, (2) issuing kernel instructions, (3) GPU compute, and (4) transferring results back to host memory. Low PCIe bandwidth and host dispatch latencies can easily dominate runtime unless kernel compute time exceeds data transfer overhead.

The end-to-end latency of a kernel launch decomposes into transfer time, queue dispatch overhead, and arithmetic execution time. When tensor transfers are small or frequent, the fixed overhead of driver submission and bus round trips dominates runtime. When transfers are large, the low bandwidth of PCIe relative to on-package HBM creates a sustained I/O bottleneck. In D·A·M terms, the Machine boundary between host and accelerator forces the Algorithm to maximize arithmetic reuse on the device once tensors arrive.

Node-level interconnect topology

Optimizing data movement requires understanding the physical topology of the compute node. A typical AI server is not a flat mesh of connected devices but a hierarchy of bandwidths that tapers as data moves farther from the chip.

Bandwidth tapers steeply as data moves farther from the accelerator.

Within a multi-accelerator server node, three interconnect tiers define this bandwidth taper:

Device-device interconnect (NVLink/Infinity Fabric): Modern multi-GPU nodes use specialized high-speed bridges like NVLink³⁰ to connect accelerators directly, bypassing the host CPU. Bandwidth ranges from 600 GB/s to 900 GB/s per GPU (NVIDIA Corporation 2020b; Choquette 2023). This link matters whenever tensors must move between accelerators within one server, including model partitioning, activation exchange, and gradient synchronization during training.³¹ The architectural principle is the boundary: traffic that stays on the accelerator fabric is far cheaper than traffic that falls back through the host.
Host-device interconnect (PCIe): The link between the CPU and the accelerator. Bandwidth ranges from 32 to 64 GB/s (PCIe Gen4/Gen5). Host-staged training data can bottleneck here, although resident data, direct storage or network I/O, and coherent accelerator links can use other paths. Multi-GPU servers may also provide multiple PCIe links rather than a single shared 64 GB/s path.
Network interface card: The link from the host to the outside world, connecting to other nodes. Bandwidth ranges from 25 to 50 GB/s (200 Gb/s to 400 Gb/s Ethernet/InfiniBand³²). This interconnect is the first step from single-node hardware reasoning into the next frontier: scale. Traversing the network moves traffic onto the narrowest, highest-latency path in the system.

³⁰ NVLink (NVIDIA Link): This direct GPU-to-GPU interconnect exists to keep accelerator-to-accelerator traffic off PCIe when tensors must move inside a server. Its 600–900 GB/s aggregate bidirectional bandwidth is roughly 300–450 GB/s per direction under balanced full-duplex traffic, several times the directional bandwidth of a standard PCIe link, so workloads with frequent cross-device tensor movement can remain in the fast part of the bandwidth taper (NVIDIA Corporation 2020b; Choquette 2023). The training algorithms that determine how much tensor traffic must cross this link are developed later; the hardware fact needed here is the bandwidth gap.

NVIDIA Corporation. 2020b. “NVLink: Scalable High-Performance Interconnect.” NVIDIA Technical Report 2020.

Choquette, Jack. 2023. “NVIDIA Hopper H100 GPU: Scaling Performance.” IEEE Micro 43 (3): 9–17. https://doi.org/10.1109/mm.2023.3256796.

³¹ AllReduce: A collective operation from MPI that aggregates values across processes (the “reduce”) and distributes the result back to every process (the “all”). Here, the term identifies why accelerator-to-accelerator bandwidth matters for distributed training workloads. At scale, algorithms, topology choices, and runtime protocols determine how costly this synchronization becomes.

³² InfiniBand: Its key feature for multi-node scaling is RDMA (Remote Direct Memory Access). With a GPU-aware path such as GPUDirect RDMA and registered GPU memory, a network adapter can transfer data directly between GPUs across nodes without staging it through host memory. This reduces host involvement and protocol overhead, so scaling is more often constrained by physical bandwidth, topology, and collective implementation rather than CPU-managed packet processing.

These three levels produce a characteristic bandwidth taper:

\[\begin{aligned} \text{HBM (3350 GB/s)} &\gg \text{NVLink (900 GB/s)} \\ &\gg \text{PCIe (64 GB/s)} \gg \text{Network (50 GB/s)} \end{aligned}\]

System efficiency depends on keeping tensors on the highest-bandwidth links available. Because PCIe and network links offer orders of magnitude less bandwidth and higher latency than on-package memory, placement and scheduling must minimize crossings across these boundaries.

Transfer optimization

Two complementary strategies mitigate transfer overheads across the host-accelerator interface: asynchronous data movement and unified memory abstraction.

In an asynchronous pipeline, software allocates pinned (page-locked) host memory and enqueues non-blocking memory copies onto dedicated DMA engines. Because copy engines operate independently of compute units, the next input batch can transfer across PCIe concurrently while the accelerator processes the current batch in local memory. This double buffering conceals transfer latency behind arithmetic runtime whenever computation time exceeds transfer time ($T_\text{compute} \ge T_\text{transfer}$). When compute duration is too short to amortize the transfer, execution remains bus-bound and compute units inevitably stall.

Unified Memory provides an alternative abstraction by presenting a single virtual address space spanning host DRAM and device HBM. Rather than requiring manual buffer allocation and explicit staging copies, the runtime driver migrates memory pages on demand via hardware page faults when either processor touches an address. While this eliminates boilerplate staging code, on-demand migration introduces severe performance unpredictability. Page faults trigger driver interrupts that stall execution pipelines, and scattered access patterns can cause page thrashing across the PCIe bus. Consequently, high-throughput production workloads rely on explicit, pipelined DMA transfers for deterministic performance, reserving Unified Memory for rapid prototyping or applications where programmer productivity outweighs sustained hardware utilization.

These interconnect and memory bounds dictate how neural network architectures map to physical silicon. Convolution exposes high spatial reuse that amortizes transfer costs through on-chip register and SRAM tiling. Dense self-attention exhibits regular matrix structure, yet quadratic sequence-length scaling creates intense pressure on package memory capacity and bandwidth. Sparse embeddings, mixture-of-experts routing, and autoregressive key-value caches present still other memory access patterns—shifting the binding constraint from arithmetic throughput to memory capacity, bus latency, or interconnect bandwidth.

Model memory pressure

Model architecture determines which memory term binds. Multilayer perceptrons (MLPs), convolutional neural networks (CNNs), and transformers exhibit fundamentally different dataflow profiles: some are bound by weight capacity, others by intermediate activation footprint, device memory bandwidth, or host interconnect throughput. Consequently, each workload demands a distinct accelerator memory hierarchy and execution strategy.

The lighthouse models introduced in Lighthouse Models as Reference Workloads ground this analysis: ResNet-50 represents CNN workloads with high spatial reuse and heavy activation footprint, GPT-2 and Llama exemplify transformer memory pressure spanning both parameter capacity and intermediate attention states, DLRM illustrates sparse embedding lookups that stress random-access memory bandwidth rather than dense compute, and MobileNetV2 demonstrates efficiency-optimized architectures where depthwise convolutions trade arithmetic intensity for lower parameter footprint. Their memory characteristics reveal how computational graph structure dictates physical hardware utilization.

Multilayer perceptrons

An MLP layer is a dense matrix multiplication followed by optional bias, activation, and normalization. Its operational intensity depends directly on the batch size: at batch size $B = 1$, each parameter loaded from off-chip memory serves only a single multiply-accumulate operation, yielding low arithmetic intensity (often $\le 1\text{ FLOP/byte}$) that starves ALUs and leaves execution memory-bandwidth bound. Increasing the batch size reuses each loaded weight matrix across $B$ independent input vectors, scaling arithmetic intensity until the matrix multiplication reaches the accelerator’s compute-bound roofline ceiling.

Dense weight matrices regularly exceed the tens of megabytes available in on-chip SRAM, requiring parameters to reside in device-local DRAM (HBM or GDDR). The host interconnect (such as PCIe) enters the critical path only when streaming dynamic inputs or when parameters and activations exceed device memory capacity and require host offloading.

Because MLP memory access patterns are regular and predictable, accelerators tile matrix operations into blocks that fit within on-chip SRAM and register files. Hardware double-buffering prefetches subsequent weight and activation tiles from DRAM into SRAM concurrently with arithmetic execution on current tiles. Delivered throughput therefore depends on whether the batch size provides sufficient arithmetic intensity and whether matrix dimensions align cleanly with hardware tile boundaries (Chen et al. 2017).

Convolutional neural networks

A convolutional layer processes input feature maps using small filter kernels that slide across spatial dimensions. This localized sliding-window structure yields high arithmetic intensity through three distinct forms of data reuse: convolutional reuse (neighboring spatial windows share input activations), filter reuse (the same kernel weights evaluate across every spatial position), and channel reuse (input channels are evaluated across multiple output filters). Unlike MLPs, where weight matrices dominate memory footprint, early CNN layers feature compact parameter tensors but produce massive intermediate activation maps.

Because intermediate activation tensors frequently exceed on-chip SRAM capacity, unoptimized layer-by-layer execution would continuously spill feature maps to device DRAM, bottlenecking the pipeline on memory bandwidth. Accelerators prevent these round-trips through spatial tiling: dividing activations and filters into tiles dimensioned to fit within on-chip SRAM buffers (Chen et al. 2017). Staging overlapping boundary pixels directly in local memory allows adjacent convolutions to consume activations without refetching them across the device DRAM bus.

Chen, Yu-Hsin, Joel Emer, and Vivienne Sze. 2017. “Using Dataflow to Optimize Energy Efficiency of Deep Neural Network Accelerators.” IEEE Micro 37 (3): 12–21. https://doi.org/10.1109/mm.2017.54.

Specialized CNN accelerators configure their hardware dataflow around these reuse patterns. The Eyeriss architecture introduced a row-stationary dataflow that maps one-dimensional rows of filter weights and activations to processing engine registers, keeping filter rows and accumulating partial sums stationary in local storage to minimize energy-intensive interconnect traffic (Chen et al. 2016). In the broader taxonomy of section 1.7, this design exemplifies a hybrid dataflow that exploits multidimensional convolutional reuse across registers, SRAM buffers, and execution units.

Transformer networks

The transformer architecture introduced in Transformers: Parallel Sequence Processing replaces local sliding windows with global self-attention³³ and dense feed-forward blocks, creating dual memory bottlenecks: quadratic intermediate attention matrices and massive parameter footprints.

³³ Attention mechanism: Bahdanau, Cho, and Bengio introduced attention for sequence-to-sequence models; Transformer self-attention later allowed each token to attend to every token in the sequence (Bahdanau et al. 2015; Vaswani et al. 2017); materializing full attention scores for a sequence of length $S$ requires an $S{\times}S$ matrix, producing quadratic intermediate storage. FlashAttention tiles the computation to avoid materializing that full matrix in HBM (Dao et al. 2022). The inference KV cache is separate: its capacity grows linearly with sequence length per layer, although each decode step reads an increasing cache (see Memory and KV cache).

Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. “Neural Machine Translation by Jointly Learning to Align and Translate.” International Conference on Learning Representations (ICLR).

Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. “Attention Is All You Need.” Advances in Neural Information Processing Systems (NeurIPS) 30: 5998–6008.

Dao, T., D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. 2022. “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.” Advances in Neural Information Processing Systems (NeurIPS) 35: 16344–59. https://doi.org/10.52202/068431-1189.

Global token interaction produces distinct memory challenges during training and inference. In standard multi-head attention, computing pairwise token affinity generates an $S \times S$ score matrix for sequence length $S$. Materializing this intermediate matrix in HBM repeatedly reads and writes $O(S^2)$ data across the memory bus, saturating bandwidth even though matrix multiplication arithmetic units remain underutilized. Tiled attention algorithms, such as FlashAttention (Dao et al. 2022), resolve this bottleneck by staging query, key, and value blocks in on-chip SRAM, computing online softmax reductions locally, and streaming outputs directly to HBM without materializing the quadratic intermediate matrix in device DRAM.

In autoregressive inference, memory pressure shifts to the key-value (KV) cache (see Memory and KV cache). Generating each subsequent token requires reading the accumulated historical KV states from HBM for a single vector-matrix operation, operating at very low arithmetic intensity and binding generation speed directly to memory bandwidth.

At the scale of large language models (Brown et al. 2020), parameter counts exceed the physical HBM capacity of any single accelerator. Distributing these models across multiple accelerators requires partitioning weights and activations across GPUs. Within a server node, high-bandwidth interconnects like NVLink provide the interconnect throughput necessary to sustain all-reduce and all-gather collectives during tensor and sequence parallelism, where standard PCIe links would throttle execution. System runtimes schedule asynchronous transfers to overlap these cross-accelerator communication phases with local matrix multiplication arithmetic (Narayanan et al. 2021).

Narayanan, Deepak, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, et al. 2021. “Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM.” Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 1–15. https://doi.org/10.1145/3458817.3476209.

Accelerator design implications

Workload-specific accelerator design is dictated by the memory and data-flow requirements of target neural network architectures. Within the D·A·M taxonomy, the algorithmic formulation directly shapes data movement patterns across the physical memory hierarchy, making a single rigid hardware architecture inefficient across different model classes. Table 11 compares the resulting tensor dimensions, reuse profiles, access regularities, and primary hardware bottlenecks across foundational model families.

Table 11: ML Model Memory Access: Different machine learning models exhibit distinct memory access patterns and bottlenecks due to variations in weight size, activation reuse, and data movement. Standard dense transformers demand high bandwidth and capacity because large weights, KV caches, and attention traffic dominate memory pressure; sparse MoE or pruned variants add further routing and sparsity considerations. CNNs benefit from spatial locality and high activation reuse, reducing memory pressure.

Model type	Weight size	Activation reuse	Memory access pattern	Primary Bottleneck
MLP (Dense)	Large, dense	Low	Regular, sequential (streamed)	Bandwidth (off-chip)
CNN	Small, reused	High	Spatial locality	Feature map movement
Transformer	Large, usually dense; sparse in MoE/pruned variants	Low-to-medium	Mostly regular GEMM plus KV-cache/attention traffic	Memory capacity + bandwidth

Each model architecture exercises the memory subsystem differently. In small-batch multi-layer perceptron (MLP) inference, matrix-vector multiplications (GEMV) dominate execution; each weight element is transferred from off-chip memory to interact with a single activation vector, producing an arithmetic intensity near one FLOP per byte and saturating off-chip bandwidth while matrix units sit idle. Increasing the batch size transforms the kernel into matrix-matrix multiplication (GEMM), amortizing weight loads across multiple input vectors. In CNNs, compact filter kernels slide over spatial feature maps, enabling high arithmetic reuse; accelerators exploit this spatial locality by staging feature tiles in on-chip static RAM (SRAM) or scratchpad memories to prevent redundant off-chip traffic. Transformers present distinct bottlenecks across execution phases: distributed training is constrained by activation retention for backpropagation and inter-accelerator communication, whereas autoregressive decoding is bound by memory bandwidth as each generated token streams the entire parameter set and an expanding key-value (KV) cache from device memory.

Hardware architects address these divergent demands with multi-tier memory hierarchies. On-chip SRAM (caches and scratchpad memories) provides tens of terabytes per second of bandwidth at low latency, capturing high-reuse activations and tiled matrix blocks. HBM provides the capacity and throughput required to stage massive weight tensors and dynamic sequence states. When model scale exceeds the physical memory capacity of an individual accelerator, high-speed interconnects such as NVLink supply peer-to-peer inter-accelerator bandwidth for distributed tensor and pipeline parallelism, avoiding the narrower host PCIe bus.

Data movement also dominates the energy budget of hardware accelerators. Transferring operands from off-chip dynamic RAM (DRAM) consumes roughly two orders of magnitude more energy than executing an arithmetic operation on local execution units. Consequently, hardware efficiency depends on maximizing computation per byte fetched from external storage. Determining whether a given kernel execution is constrained by compute throughput or memory bandwidth requires an analytical boundary. The roofline model establishes this relationship by comparing operational intensity against hardware performance limits, ensuring optimizations target the true physical bottleneck.

Self-Check: Question

An accelerator delivers $R_{\text{peak}} = 1{,}000\text{ TFLOP/s}$ ($10^{15}\text{ FLOP/s}$) in FP16 and has a High-Bandwidth Memory (HBM) subsystem delivering $\text{BW} = 2.0\text{ TB/s}$ ($2\times 10^{12}\text{ bytes/s}$). What is the hardware balance point (ridge point $I_{\text{ridge}}$) of this system?
1. $50\text{ FLOP/byte}$
2. $200\text{ FLOP/byte}$
3. $500\text{ FLOP/byte}$
4. $2{,}000\text{ FLOP/byte}$
When training a 7-billion parameter model using standard mixed-precision (FP16/BF16) with the Adam optimizer, calculate the minimum memory required purely for model states (weights, gradients, and optimizer states) and explain why optimizer states dominate this footprint.
Arrange the following levels of a modern GPU memory hierarchy in order of access latency, from lowest latency (fastest) to highest latency (slowest):

High-Bandwidth Memory (HBM3)
Register File
Pinned Host System Memory (DDR5 via PCIe)
Shared Memory / L1 Cache
On-Chip L2 Cache

Why does High-Bandwidth Memory (HBM) achieve significantly higher bandwidth (e.g., $>2\text{ TB/s}$) than traditional GDDR6X memory (e.g., $\approx 760\text{ GB/s}$) while maintaining comparable or lower power per bit?
1. HBM operates at a $10\times$ higher clock frequency than GDDR6X on standard PCB traces
2. HBM uses optical photonic signaling to transmit data across the motherboard
3. HBM eliminates all error correction codes and row buffer precharge cycles
4. HBM vertically stacks DRAM dies using Through-Silicon Vias (TSVs) and connects to the GPU via a wide 1024-bit per stack silicon interposer bus at lower clock speeds
Memory allocated on the host CPU that is locked into physical RAM and prevents operating system paging, enabling direct DMA transfers over PCIe to the GPU, is called ____ host memory.
True or False: The AI Memory Wall refers solely to the limited physical capacity (GBs) of GPU DRAM, meaning that if an accelerator has sufficient gigabytes to store model weights, memory bandwidth will never bottleneck execution.

See Answers →

Roofline Model

Low arithmetic intensity pins the workload in the memory-bound regime.

³⁴ Roofline model: Introduced by Williams et al. (2009) at UC Berkeley, building on earlier I/O complexity work from the 1980s. Their specific contribution was making the compute vs. bandwidth trade-off visual and actionable: the characteristic roofline plot immediately reveals whether a kernel is compute bound (hitting the flat ceiling) or memory bound (hitting the sloped bandwidth line) and quantifies the gap to hardware limits. A kernel operating at only 50 percent of its ceiling has a clear 2$\times$ utilization gap to close, making this the standard diagnostic tool for accelerator optimization.

Williams, Samuel, Andrew Waterman, and David Patterson. 2009. “Roofline: An Insightful Visual Performance Model for Multicore Architectures.” Communications of the ACM 52 (4): 65–76. https://doi.org/10.1145/1498765.1498785.

Peak FLOP/s indicates what execution units can deliver when operands are already on chip, but sustained accelerator throughput is ultimately bounded by the rate at which the memory hierarchy delivers data. The Roofline Model³⁴ (Williams et al. 2009) formalizes this relationship by plotting attainable performance against arithmetic intensity, establishing whether an operation is compute bound or memory bound. Rather than evaluating compute capacity in isolation, the model intersects hardware arithmetic throughput with memory bandwidth, revealing which physical ceiling binds execution before an engineer commits optimization effort.

Performance is bounded by two ceilings, as equation 2 formalizes. Here, attainable performance $R_{\text{attain}}$ and peak compute $R_{\text{peak}}$ are in FLOP/s (often reported as TFLOP/s), peak bandwidth $\text{BW}$ is in bytes/s (often TB/s), and arithmetic intensity $I$ is in FLOP/byte: \[R_{\text{attain}} = \min(R_{\text{peak}}, \text{BW} \times I) \tag{2}\]

Definition 1.5: Arithmetic intensity

Arithmetic intensity is the ratio of operations to bytes transferred across the memory interface being modeled ($\text{FLOP}/\text{byte}$). Together with that level’s bandwidth and the relevant compute peak, it determines which idealized roofline ceiling is lower.

Significance: The threshold between the two ceilings is the ridge point $R_{\text{peak}} / \text{BW}$. For an A100 (312 TFLOP/s FP16/BF16, 2.04 TB/s), it is roughly 153 FLOP/byte. Under the stated traffic models, a large well-tiled 1024 $\times$ 1024 multiply reaches about 341.3 FLOP/byte, while standalone ReLU is about 0.125 FLOP/byte. They therefore meet different ceilings on the same hardware.
Distinction: Unlike total FLOPs (a count of operations), arithmetic intensity is a ratio that characterizes the shape of a workload’s hardware demand. Two kernels with identical FLOPs but different memory access patterns have different arithmetic intensities and will be bottlenecked by different hardware resources.
Common pitfall: A frequent misconception is that arithmetic intensity is a fixed property of an operation. In practice, it depends on implementation details: a naive matrix multiplication that reloads operands from DRAM for each output element has low arithmetic intensity; a blocked (tiled) implementation that reuses data from fast SRAM achieves high arithmetic intensity—the same mathematical operation, orders of magnitude apart in hardware efficiency.

With $O$ as the operation count and $D_{\text{vol}}$ as data volume in bytes, equation 3 makes the definition operational. \[I = \frac{O}{D_{\text{vol}}} \tag{3}\]

The roofline visualization shows performance (TFLOP/s) on the vertical axis and arithmetic intensity (FLOP/byte) on the horizontal axis. At low arithmetic intensity, performance increases linearly along the memory bandwidth slope ($\text{BW} \times I$), where execution is memory bound. Above the ridge point, performance saturates at peak compute ($R_{\text{peak}}$), where execution is compute bound. Bottleneck diagnostic maps each regime to the optimizations that pay off and the ones that waste effort, so that classifying a workload as memory-bound or compute-bound tells the engineer directly whether a faster accelerator or more bandwidth is the binding investment.

Hardware ridge points

The ridge point, the hardware balance $I_{\text{ridge}}$ established in section 1.4.2, is the arithmetic-intensity threshold at which an accelerator transitions from memory-bound to compute-bound execution. Table 12 quantifies this boundary across three generations of NVIDIA accelerators:

Table 12: Hardware Ridge Points: Vendor-published FP16 peak throughput and memory bandwidth for three NVIDIA generations, and the ridge point each pair implies. The ridge rises from V100 to H100 because compute throughput grows faster than bandwidth; these values are hardware-specific rather than universal generation averages.

NVIDIA Accelerator	Peak FP16	Bandwidth	Ridge Point
V100 (2017)	125 TFLOP/s	0.9 TB/s	138.9 FLOP/byte
A100 (2020)	312 TFLOP/s	2.04 TB/s	153 FLOP/byte
H100 (2022)	989 TFLOP/s	3.35 TB/s	295.2 FLOP/byte

Because arithmetic throughput has grown far faster than memory bandwidth across generations, the hardware ridge point shifts progressively rightward. A workload must therefore exhibit substantially greater data reuse to saturate newer hardware; an algorithm that was compute-bound on an older architecture can easily become memory-bound on a newer one.

Napkin Math 1.3: The utilization gap

Problem: Why is it harder to get 100 percent utilization on an H100 than a V100?

Metric: The ridge point $I_{\text{ridge}} = R_{\text{peak}} / \text{BW}$ (FLOP/byte) from section 1.4.2: how many math operations the hardware must perform for every byte of data loaded to keep the compute units busy.

Evolution:

V100 (2017): 125 TFLOP/s / 0.9 TB/s ≈ 138.9 FLOP/byte.
A100 (2020): 312 TFLOP/s / 2.04 TB/s ≈ 153 FLOP/byte.
H100 (2022): 989 TFLOP/s / 3.35 TB/s ≈ 295.2 FLOP/byte.

Result: The intensity needed to reach the tensor peak has increased. Under this simple HBM roofline, a kernel with $I$ = 200 FLOP/byte lies above the A100 ridge but below the H100 ridge. If its intensity and implementation remain unchanged, its idealized gain is then closer to the 1.6× bandwidth ratio than the 3.2× peak-compute ratio.

Systems insight: A standalone ReLU under a one-read, one-write FP32 traffic model performs about 1 operation per 8 bytes, far below the H100 tensor ridge. A well-tiled 1024 $\times$ 1024 dense multiply reaches about 341.3 FLOP/byte under its idealized traffic model. Fusion is valuable for low-intensity chains because it can remove intermediate round-trips; it is not automatically the best optimization for every kernel.

Operations such as depthwise convolution, embedding lookup, LayerNorm, and softmax serve as low-intensity reference points because their minimal data reuse causes execution time to be dominated by memory transfers. Table 13 maps common neural network operations to the Roofline Model.

Table 13: Operations on the Roofline: Neural network layers span a wide range of arithmetic intensities. Large, well-tiled convolutions and batched GEMMs can be compute-bound, while MobileNet depthwise layers, attention softmax, normalization, and DLRM embeddings are often memory-bound. To see how these intensity values translate into real performance predictions, a transformer layer provides a complete arithmetic-intensity calculation across its major sub-operations.

Operation	Arithmetic Intensity	Classification	Lighthouse example
Conv2D (Dense)	50–200 FLOP/byte	Straddles ridge; high-reuse cases compute-bound	ResNet-50
Dense MatMul (large batch, well-tiled)	64–256+ FLOP/byte	Often compute-bound at large batch	GPT-2 (batched projections)
Depthwise conv	10–20 FLOP/byte	Memory-bound	MobileNet
Attention Softmax	Convention-dependent	Memory-bound	GPT-2 (Generation)
LayerNorm	1–2 FLOP/byte	Memory-bound	GPT-2/Llama
Embedding lookup	$<1$ FLOP/byte	Memory-bound	DLRM

Napkin Math 1.4: Transformer layer analysis

For a transformer with hidden_dim = 768, batch = 32, seq = 512:

Attention QKV projection:

FLOPs: 2 $\times$ 3 $\times$ 32 $\times$ 512 $\times$ 768 $\times$ 768 = 58 GFLOP
Bytes: (input + weights + output) = (32 $\times$ 512 $\times$ 768 + 3 $\times$ 768 $\times$ 768 + 32 $\times$ 512 $\times$ 768 $\times$ 3) $\times$ 2 ≈ 104.2 MB
AI = 58 GFLOP / 104.2 MB = 556.4 FLOP/byte, which is compute bound on A100 (above 153 FLOP/byte threshold)

Softmax:

FLOPs: 32 $\times$ 12 $\times$ 512 $\times$ 512 $\times$ 3 ≈ 302 MFLOP (exp, sum, div)
Bytes: 32 $\times$ 12 $\times$ 512 $\times$ 512 $\times 2 \times 2$ = 402.7 MB
AI = 302 MFLOP / 402.7 MB = 0.75 FLOP/byte under this simple operation count, which is memory-bound. Counts that weight exponentials and division as multiple operations produce a larger numerical intensity without changing the bottleneck classification.

Systems insight: This analysis explains why FlashAttention focuses on reducing memory traffic in attention rather than reducing FLOPs.

The roofline classification dictates which physical subsystem limits performance and which optimizations can relieve it. For memory-bound operations, execution time is dominated by data transfers across the off-chip memory bus; ALU pipelines spend idle cycles waiting for operands. Optimization must therefore focus on conserving memory bandwidth: fusing adjacent operators keeps intermediate activations in on-chip registers or shared memory (SRAM), lower-precision numeric formats reduce the physical byte count of memory transfers, and algorithmic refactorings like FlashAttention restructure access patterns to compute within SRAM tiles. For compute-bound operations, off-chip bandwidth is no longer the bottleneck because high data reuse keeps execution units supplied. Performance here depends on saturating arithmetic execution pipelines: structuring matrix tiles to sustain Tensor Core throughput, hiding instruction latencies through software pipelining, and scaling batch sizes to distribute work across all available streaming multiprocessors.

Calculating memory bandwidth bounds

The memory-bound region of the Roofline Model is governed by the accelerator’s off-chip memory bandwidth. To sustain an operational throughput $R_{\text{ops}}$ (FLOP/s) for a workload with arithmetic intensity $I$ (FLOP/byte), the memory subsystem must transfer data at the rate defined by equation 4: \[\text{BW}_{\text{req}} = \frac{R_{\text{ops}}}{I} \text{ bytes/s} \tag{4}\]

When this required bandwidth exceeds the hardware’s peak memory bandwidth $\text{BW}$, memory transfers cap attainable throughput according to equation 5: \[R_{\text{attain}} = \text{BW} \times I \tag{5}\]

Evaluating an operation against the hardware’s ridge point determines whether execution is bound by memory bandwidth or arithmetic throughput. When data reuse is extensive, arithmetic intensity clears the ridge point, placing the kernel in the compute-bound regime. A 2D convolution demonstrates this behavior.

Napkin Math 1.5: Convolutional layer analysis

Consider a Conv2D layer with input shape (batch = 32, channels = 128, height = 56, width = 56), output channels = 256, and kernel size $3{\times}3$ on an A100 GPU:

Computational requirements:

Output size: $32 \times 256 \times 56 \times 56$ = 25.7M elements
FLOPs per output: $128 \times 3 \times 3 \times 2$ = 2,304 (multiply-add)
Total FLOPs: 25.7M $\times$ 2,304 = 59.2 GFLOP

Memory traffic analysis:

Input: $32 \times 128 \times 56 \times 56 \times 2$ = 25.7 MB (FP16)
Weights: $256 \times 128 \times 3 \times 3 \times 2$ ≈ 0.6 MB (FP16)
Output: $32 \times 256 \times 56 \times 56 \times 2$ = 51.4 MB (FP16)
Total: 77.7 MB

Arithmetic intensity: $I$ = 59.2 GFLOP / 77.7 MB = 762.2 FLOP/byte

Systems insight: This is well above A100’s ridge point of 153 FLOP/byte, so the simple Roofline Model places the operation in the compute-limited region. Peak throughput of ~312 TFLOP/s (FP16 with Tensor Cores) is therefore the ceiling, but realized performance still depends on tiling, occupancy, instruction mix, and library efficiency.

The convolutional layer’s high arithmetic intensity arises from spatial weight reuse: the same $3{\times}3$ kernel is applied across every spatial location in the feature map, amortizing the cost of loading weights from off-chip memory across millions of multiply-accumulate operations. Because intermediate values and kernel weights remain in fast on-chip registers and shared memory, the off-chip memory bus remains idle while arithmetic pipelines stay saturated.

Not all neural network layers exhibit such extensive data reuse. Fully connected (dense) layers—such as classification heads, feed-forward networks, and linear projections in transformer attention blocks—lack spatial dimensions. When evaluated at small batch sizes, their arithmetic intensity drops substantially, shifting execution into the memory-bound regime:

Napkin Math 1.6: Dense layer analysis

Consider a fully connected layer: input (batch = 32, features = 2048) → output (batch = 32, features = 2048) on the same A100:

Computational requirements:

Matrix multiply: $(32 \times 2048) \times (2048 \times 2048)$
Total FLOPs: $2 \times 32 \times 2048 \times 2048$ = 268.4 MFLOP

Memory traffic analysis:

Input: $32 \times 2048 \times 2$ = 131.1 KB (FP16)
Weights: $2048 \times 2048 \times 2$ = 8.4 MB (FP16)
Output: $32 \times 2048 \times 2$ = 131.1 KB (FP16)
Total: 8.7 MB

Arithmetic intensity: $I$ = 268.4 MFLOP / 8.7 MB = 31 FLOP/byte

This intensity sits below A100’s ridge point of 153 FLOP/byte, making this operation memory-bound. Attainable performance: $R_{\text{attain}}$ = 2,039 GB/s $\times$ 31 FLOP/byte = 63.3 TFLOP/s

Systems insight: Delivered throughput reaches only 20.3 percent of peak compute capability, demonstrating the memory wall effect for small batch sizes.

The dense layer’s lower arithmetic intensity stems from limited parameter reuse. While a convolutional kernel is reused across every spatial location in an activation map, a dense weight matrix is reused only across the batch dimension. At a batch size of 32, each weight parameter transferred from HBM is multiplied only 32 times before being evicted. Because the off-chip memory bus cannot transfer operands fast enough to keep the tensor cores fed, arithmetic units spend over 80 percent of their cycles stalled, capping throughput at 20.3 percent of peak compute capacity. This structural constraint explains why transformer inference at low batch sizes is dominated by memory bandwidth rather than arithmetic capability.

While small-batch matrix multiplications suffer from restricted parameter reuse, elementwise and reduction operators represent an even more acute memory bottleneck. Operations such as layer normalization, activation functions, and residual additions perform only a few arithmetic operations per element loaded from memory, driving arithmetic intensity to its physical floor:

Napkin Math 1.7: LayerNorm analysis

LayerNorm with input shape (batch = 32, seq = 512, hidden = 768):

Computational requirements:

Elements: $32 \times 512 \times 768$ = 12.6M
Approximate operations per element: mean reduction (1 ADD), variance (2 ADD, 1 MUL), normalization and affine transform (2 ADD, 2 MUL) ≈ 8; reciprocal-square-root overhead is amortized across each hidden vector
Total FLOPs: 12.6M $\times$ 8 = 100.7 MFLOP

Memory traffic:

Input: 12.6M $\times$ 2 = 25.2 MB
Parameters (scale, bias): $768 \times 2 \times 2$ = 3.1 KB (negligible)
Output: 12.6M $\times$ 2 = 25.2 MB
Total: 50.3 MB

Arithmetic intensity: $I$ = 100.7 MFLOP / 50.3 MB = 2.0 FLOP/byte

This arithmetic intensity sits severely below the A100 ridge point (77× below). Performance is limited to: $R_{\text{attain}}$ = 2039 GB/s $\times$ 2.0 FLOP/byte = 4.1 TFLOP/s

Systems insight: The simple HBM roofline bounds this standalone LayerNorm at about 1 percent of A100’s tensor peak. Actual latency also depends on fusion, reduction strategy, launch overhead, and which memory level supplies the data.

Standalone normalization performs minimal arithmetic relative to the memory traffic it generates, as the LayerNorm analysis demonstrates. In an unfused implementation, every activation element is transferred from off-chip HBM across the global bus, reduced, and written back to memory, achieving an arithmetic intensity of approximately 2 FLOP/byte and leaving compute units idling 99 percent of the time. In the D·A·M framework, when the machine constraint is an unyielding memory bandwidth wall, the algorithmic response is operator fusion: collapsing adjacent operations—such as linear projections, bias additions, normalization, and activations—into a single kernel. Fusion keeps intermediate activations resident in fast on-chip SRAM or registers, eliminating round-trips across the memory bus and transforming bandwidth-starved workloads into compute-efficient execution.

Optimization by intensity regime

The roofline analysis directly informs optimization priorities, summarized in table 14.

Table 14: Optimization by Arithmetic Intensity Regime: Roofline position determines whether an optimization should chase compute utilization, memory-traffic reduction, or complete elimination of memory round-trips. The lower the arithmetic intensity, the more valuable fusion and data-movement avoidance become.

Intensity regime	Typical operations	Optimization priority	Common techniques	Expected impact
High AI ($>200$ FLOP/byte)	Large convolutions	Maximize compute utilization.	Tensor Cores, thread-block tuning, and high occupancy.	Raises sustained compute utilization when the kernel already clears the ridge.
Medium AI (20–200 FLOP/byte)	Medium-sized dense layers	Balance compute and memory optimization.	Larger batches, register tiling, and fusion with adjacent operations.	Can move the binding bottleneck between memory and compute.
Low AI ($<20$ FLOP/byte)	Small dense layers and element-wise operations	Reduce memory traffic.	Aggressive operator fusion, reduced precision (FP16 → INT8), and algorithmic changes.	Reduces external-memory traffic when fusion or layout changes are legal.
Very low AI ($<2$ FLOP/byte)	Normalization layers and activation functions	Eliminate memory round-trips.	Fuse with adjacent operations and use in-place computation where legal.	Removes intermediate round-trips when adjacent operations can be fused.

For low-AI operations, operator fusion is often the decisive optimization: LayerNorm combined with the Gaussian Error Linear Unit (GELU), for example, can become a single fused kernel. One of the most accessible levers for moving an operation up and right on the roofline is batching.

Napkin Math 1.8: Batch size and arithmetic intensity

Increasing batch size improves arithmetic intensity for matrix operations by amortizing weight loading. Equation 6 formalizes this relationship for an idealized FP16/BF16 dense layer $(B{\times}M){\times}(M{\times}N)$, counting one read each of the input and weights and one write of the output, where $B$ is batch size and $M$ and $N$ are the input and output dimensions: \[I = \frac{2BMN}{2BM + 2MN + 2BN} \approx \frac{2BMN}{2MN} = B \quad (\text{when } 2MN \gg 2B(M+N)) \tag{6}\]

Example: Dense layer with M=N=2048 (FP16)

Batch = 1: AI ≈ 1 FLOP/byte (memory bound)
Batch = 32: AI ≈ 31 FLOP/byte (memory bound)
Batch = 256: AI ≈ 204.8 FLOP/byte (compute bound on A100)

Systems insight: This explains why batching can produce large throughput improvements in production inference systems, as MLPerf Inference, the standardized benchmark suite covered in Benchmarking, demonstrates by separating bulk-throughput runs from latency-constrained serving runs (Reddi et al. 2019).

Reddi, Vijay Janapa, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, et al. 2019. “MLPerf Inference Benchmark.” 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 446–59. https://doi.org/10.1109/isca45697.2020.00045.

The batch size analysis reveals why inference serving systems are designed around batching: it changes the arithmetic intensity regime of memory-bound workloads. However, batching introduces latency trade-offs, since requests must wait in a queue until a batch forms. This tension between throughput (favoring large batches) and latency (favoring small batches) is a central challenge in ML serving systems, explored in depth in Dynamic batching latency-throughput trade-offs.

When service latency limits batching, batch-1 or small-batch LLM decode often retains low weight reuse and low arithmetic intensity. The next worked model quantifies that particular regime rather than treating all LLM inference as identical.

Napkin Math 1.9: The throughput ceiling

Problem: A batch-1 GPT-2 calculation makes this bandwidth ceiling concrete: what is the maximum possible utilization of an NVIDIA A100 when running GPT-2 inference (batch size 1)?

The hardware constraints (the denominators)

Peak compute: 312 TFLOP/s (FP16 Tensor Core).
Peak bandwidth: 2.04 TB/s (HBM2e).
Ridge point $(R_{\text{peak}}/\text{BW})$: 312 TFLOP/s / 2.04 TB/s = 153 FLOP/byte (for FP16 Tensor Core).

Interpretation: Saturating this chip at FP16 precision requires 153 FLOP/byte operations for every byte loaded. The ridge point varies by precision: FP32 operations (19.5 TFLOP/s peak) have a ridge point of only ~9.6 FLOP/byte.

The workload characteristics (the numerator)

Model: GPT-2 XL (1.5 billion parameters).
Operation: Autoregressive generation (1 token at a time).
Data movement: Must load all weights (3 GB @ FP16) for every token.
Compute: Vector-Matrix multiplication. 2 $\times$ Params ≈ 3 GFLOP.
Arithmetic intensity: 3 GFLOP / 3 GB = 1 FLOP/byte

The prediction (iron law)

Since Actual Intensity (1) $\ll$ Ridge Point (153 FLOP/byte), the system is bandwidth bound.

Maximum throughput: 1 FLOP/byte $\times$ 2.04 TB/s = 2.04 TFLOP/s.
Utilization ceiling: 2.04 TFLOP/s (Actual) / 312 TFLOP/s (Peak) ≈ $0.7\%$

Systems insight: Without batching, a $15,000 GPU runs at less than 1 percent compute utilization in this weight-streaming decode model. Batching and weight quantization address that gap; key-value caching instead avoids recomputing prior attention states.

Through this derivation, the Roofline Model provides a diagnostic framework for identifying whether operations are compute bound or memory bound. Knowing that a workload is memory bound at 0.7 percent utilization is only the first step; the next challenge is translating this diagnosis into efficient execution plans that exploit accelerator architectures.

Self-Check: Question

A developer runs a LayerNorm kernel on an accelerator with $R_{\text{peak}} = 312\text{ TFLOP/s}$ and $\text{BW} = 1.5\text{ TB/s}$ ($I_{\text{ridge}} = 208\text{ FLOP/byte}$). The LayerNorm has an arithmetic intensity of $I = 4\text{ FLOP/byte}$. What is the maximum attainable performance of this kernel, and what is the binding bottleneck?
1. $6.0\text{ TFLOP/s}$, bound by memory bandwidth
2. $312\text{ TFLOP/s}$, bound by peak compute capacity
3. $78\text{ TFLOP/s}$, bound by warp scheduler instruction issue rate
4. $1.5\text{ TFLOP/s}$, bound by PCIe bus transfer limits
Why has the hardware ridge point ($I_{\text{ridge}}$) increased dramatically across successive GPU generations (e.g., from Volta to Ampere to Hopper), and what pressure does this trend place on compiler and kernel developers?
True or False: When a kernel operates in the memory-bound regime of the Roofline model ($I < I_{\text{ridge}}$), doubling the accelerator’s peak tensor compute capability ($R_{\text{peak}}$) without changing memory bandwidth will double the kernel’s execution speed.
In the Roofline model, the transition point on the horizontal axis where the memory-bandwidth ceiling intersects the peak-compute ceiling is known as the hardware ____ point.
An engineer profiles a transformer inference workload and discovers that the attention Softmax kernel is heavily memory bandwidth bound. Which of the following optimization techniques directly increases arithmetic intensity to move the kernel closer to the compute-bound regime?
1. Upgrading host CPU RAM to DDR5 to decrease kernel enqueue latency
2. Fusing the scale, mask, Softmax, and dropout operations into a single kernel to keep intermediate activations in registers/SRAM
3. Increasing the clock frequency of the GPU Tensor Cores by $15\%$
4. Disabling warp scheduler out-of-order instruction issue

See Answers →

Hardware Mapping

Consider a $3{\times}3$ convolution running on an accelerator tile. The mathematical operation is fixed, but the execution plan is not. One schedule can keep a filter in local registers while many output pixels stream past it; another can advance pixel by pixel and reload the same filter values repeatedly from a slower memory tier. Both schedules compute the same tensor. Only one turns the reuse in the convolution into real bandwidth savings.

Definition 1.6: Mapping in AI acceleration

Mapping in AI acceleration is the accelerator-compiler process of binding the logical computation graph to the physical hardware topology by deciding which operations execute on which processing elements, which data resides in which memory tier, and in what temporal order.

Significance: Within the D·A·M taxonomy, mapping is a machine-axis decision that helps determine how closely an operation approaches the roofline bound $\min(R_{\text{peak}},\; \text{BW} \times I)$. A poor tiling choice that forces unnecessary DRAM accesses can reduce effective arithmetic intensity enough to move a nominally compute-bound operation into the bandwidth-bound regime.
Distinction: Traditional compilation often emphasizes instruction selection, register allocation, and scheduling for general-purpose processors. Accelerator mapping gives equal prominence to spatial placement and explicit data movement across a dataflow architecture. The reason is physical: in the 45 nm comparison introduced earlier, off-chip DRAM access consumed roughly 200$\times$ the energy of a local integer multiply-accumulate, although the ratio depends on the technology and operation.
Common pitfall: A frequent misconception is that mapping is automatically handled by frameworks. For general GPU workloads, compilers such as Accelerated Linear Algebra (XLA) can find strong mappings for common kernels; for specialized accelerators (systolic arrays, custom ASICs), compiler-generated mappings may still lag hand-tuned schedules because the compiler’s search space is limited by the time budget at compilation.

The convolution example exposes the three decisions that recur throughout accelerator compilation. Placement assigns the multiply-accumulate work to processing elements so parallelism does not turn into idle time or interconnect congestion. Allocation keeps weights, activations, and partial sums in the memory tier where their next use will occur, rather than letting reuse spill back to DRAM. Scheduling orders loops and kernels so the chosen placement and allocation remain valid over time. A poor choice in any one dimension can collapse a high-arithmetic-intensity operation back into a bandwidth-bound execution. In practice, these choices are too coupled for developers to manage by hand at model scale, which is why systems such as XLA, TVM, and TensorRT lower high-level models and search or select execution plans within their compile-time and hardware budgets. Section 1.8 examines that compiler support in detail.

Placement and allocation

Translating a model’s computational graph into efficient hardware execution requires solving two tightly coupled problems. Computation placement determines which operations run on which processing elements, balancing parallelism against communication costs. Memory allocation determines where data resides within the memory hierarchy, trading capacity against access latency. These two decisions interact: placing operations on distant processing elements increases the memory bandwidth required to shuttle data between them, while allocating data to fast but small on-chip memory limits which operations can execute concurrently. Getting either wrong leaves thousands of processing elements idle or starved for data.

Computation placement

Computation placement is the process of assigning operations to an accelerator’s processing elements (PEs) to expose parallelism, limit idle time, and reduce unnecessary data movement. Modern accelerators contain many such resources: the NVIDIA H100 has more than 16,000 CUDA cores and more than 500 Tensor Cores (Choquette 2023), TPUs organize thousands of multiply-accumulate units into systolic arrays (Jouppi et al. 2017), and wafer-scale processors such as Cerebras’ CS-2 integrate more than 850,000 cores (Systems 2021). At these scales, placement inefficiencies become measurable because idle cores and redundant transfers waste both time and energy.

The difficulty of placement depends on workload regularity. Convolutional networks and dense transformer operations possess regular access patterns that tile predictably, though transformer architectures mix operations with distinct tensor shapes and reuse profiles. Graph neural networks (GNNs) present greater difficulty: sparse, input-dependent neighborhood aggregations introduce load imbalance and non-uniform communication across processing elements. Table 15 summarizes these core placement challenges and their operational impact.

Table 15: Computation Placement Challenges: Effective neural network deployment requires strategic allocation of computations to processing elements, balancing workload distribution, data movement costs, and hardware constraints to maximize execution efficiency. These challenges guide the design of mapping strategies that optimize resource utilization and minimize communication overhead.

Challenge	Impact on execution	Key Considerations for Placement
Workload imbalance	Some processing elements finish early while others remain overloaded, leading to idle compute resources.	Distribute operations evenly to prevent stalls and ensure full utilization of PEs.
Irregular computation patterns	Sparse graphs, routing, and variable shapes can create nonuniform work that complicates static placement.	Use adaptive placement when workload characteristics vary at run time.
Excessive data movement	Frequent memory transfers introduce latency and increase power consumption.	Keep frequently used data close to the compute units and minimize off-chip memory accesses.
Limited interconnect bandwidth	Poorly placed operations can create congestion, slowing data movement between PEs.	Optimize spatial and temporal placement to reduce communication overhead.
Model-specific execution needs	CNNs, transformers, and GNNs require different execution patterns, making a single placement strategy ineffective.	Tailor placement strategies to match the computational structure of each model type.

Static spatial mappings suffice for regular dense operations whose shapes and loop bounds are known ahead of time. Workloads with dynamic shapes, sparsity, or mixture-of-experts routing instead require runtime scheduling to balance execution across processing elements. Yet even an optimal computation placement stalls if the required operands are not staged in time: compute units can execute only as fast as operands arrive from the memory hierarchy.

Memory allocation

While computation placement determines where operations execute, memory allocation defines where data resides and how it flows through the hierarchy. The goal is to keep reused data near the processing elements without exceeding faster-tier capacity. GPUs expose global memory, shared memory, caches, and registers that kernels coordinate through tiling (NVIDIA Corporation 2020a). TPUs use on-chip buffers to stage activations and weights for systolic-array execution (figure 8) (Jouppi et al. 2017), while wafer-scale processors partition memory and computation to control interconnect traffic (Systems 2021). General-purpose processors and accelerators both use memory hierarchies, but many accelerators expose more placement decisions to compilers and kernels. When allocation fails to exploit locality, processing elements stall waiting for operands, driving down utilization and dissipating energy across the memory bus.

Systems, Cerebras. 2021. “Wafer-Scale Deep Learning Acceleration with the Cerebras CS-2.” Cerebras Technical Paper 2021.

The severity of these penalties varies by workload. Convolutional networks rely on structured, localized access patterns and benefit from well-defined memory layouts that facilitate predictable reuse (Chen et al. 2016). Transformers require access to large parameter sets and intermediate activations, making them sensitive to capacity and bandwidth. Graph neural networks add irregular sparse structures that complicate allocation and prefetching.

Capacity is only the first allocation test. A tensor can fit in accelerator memory yet still dominate execution if each kernel rereads it from a distant tier; conversely, tiling can keep a larger working set efficient by preserving reuse locally. The allocation diagnostic is therefore traffic, not capacity alone: compilers must track how frequently values cross each memory boundary and keep high-reuse tensors in the fastest feasible tier. Systems combine static memory plans for known shapes with dynamic buffer management when shapes vary, while device capacity limits which models fit without partitioning or offload. Table 16 summarizes these allocation challenges across the hierarchy.

Table 16: Memory Allocation Challenges: Efficient memory management in AI accelerators balances data access speed with hardware constraints, mitigating performance bottlenecks caused by latency, bandwidth limitations, and irregular data patterns. Complex models such as transformers and graph networks impose variable and demanding memory requirements that amplify these challenges.

Challenge	Impact on Execution	Key Considerations for Allocation
High memory latency	Slow data access delays execution and reduces throughput.	Prioritize placing frequently accessed data in faster memory locations.
Limited on-chip storage	Small local memory constrains the amount of data available near compute units.	Allocate storage efficiently to maximize data availability without exceeding hardware limits.
High off-chip bandwidth demand	Frequent access to external memory increases delays and power consumption.	Reduce unnecessary memory transfers by carefully managing when and how data is moved.
Irregular memory access patterns	Some models require accessing data unpredictably, leading to inefficient memory usage.	Organize memory layout to align with access patterns and minimize unnecessary data movement.
Model-specific memory needs	Different models require different allocation strategies to optimize performance.	Tailor allocation decisions based on the structure and execution characteristics of the workload.

Combinatorial complexity

The small convolution example illustrates why hardware mapping becomes a combinatorial search problem. Keeping a filter resident in local storage improves reuse only if the assigned processing elements have sufficient storage and if the loop schedule revisits that filter before eviction. Similarly, parallelizing across more processing elements improves throughput only until synchronization overhead and interconnect contention erode the benefit. Because changing one decision alters the feasible and efficient choices for the others, compilers must evaluate placement, allocation, and scheduling jointly rather than independently. Table 17 summarizes these recurring trade-offs across the execution stack.

Table 17: Placement-Allocation-Scheduling Trade-Offs: AI accelerator performance depends on mapping computations to hardware, allocating data to memory tiers, and scheduling execution over time. Careful consideration of these interdependent factors is essential for maximizing throughput and minimizing energy consumption.

Dimension	Placement considerations	Allocation and Scheduling Considerations
Computational granularity	Fine-grained placement enables greater parallelism but increases synchronization overhead.	Coarse-grained scheduling reduces synchronization overhead but may limit flexibility.
Spatial vs. Temporal Mapping	Spatial placement enhances parallel execution but can lead to resource contention and memory congestion.	Temporal scheduling balances resource sharing but may reduce overall throughput.
Memory and Data Locality	Placing data closer to compute units minimizes latency but may reduce overall memory availability.	Allocating data across multiple memory levels increases capacity but introduces higher access costs.
Communication and synchronization	Co-locating compute units reduces communication latency but may introduce contention.	Scheduling synchronization mechanisms mitigates stalls but can introduce additional overhead.
Dataflow and Execution Ordering	Static placement simplifies execution but limits adaptability to workload variations.	Dynamic scheduling improves adaptability but adds scheduling complexity.

Evaluating these trade-offs requires navigating an unconstrained search space governed by three structural decisions: loop execution order, spatial parallelization across processing elements, and memory allocation across the hierarchy. To understand why exhaustive search is impractical, consider the combinatorial growth of each dimension independently.

Ordering computation and execution

Machine learning workloads are typically expressed as nested loops that iterate over tensor dimensions. For instance, a matrix multiplication kernel loops over batch size ($B$), input features ($C_{\text{in}}$), and output features ($C_{\text{out}}$). The loop execution order dictates which tensor elements remain resident in registers or SRAM between iterations, directly determining data locality and reuse.

Ignoring dependences and equivalent orderings, the number of ways to arrange $n_{\text{loops}}$ loops has the toy upper bound: \[ N_{\text{order}} = n_{\text{loops}}! \] which scales rapidly. A typical convolutional layer may involve up to seven loop dimensions, leading to: \[ 7! = 5,040 \text{ possible execution orders.} \]

Mapping choices explode combinatorially as loop dimensions grow.

If each memory level could choose an ordering independently, the corresponding toy upper bound would expand as: \[ (n_{\text{loops}}!)^{N_{\text{mem}}} \] where $N_{\text{mem}}$ is the number of memory hierarchy levels. This rapid expansion shows why execution order optimization matters: suboptimal loop ordering evicts values prematurely and multiplies DRAM traffic, whereas an optimized ordering maximizes reuse in local SRAM and caches (Sze et al. 2017).

Parallelization across processing elements

Modern AI accelerators use thousands of processing elements to execute loop iterations concurrently. Parallelizing an inner reduction loop, however, requires cross-core synchronization and collective communication, whereas parallelizing outer independent loops (such as batch or output feature channels) allows each core to execute independently without interconnect traffic. Determining which loops to map spatially requires careful partitioning.

Before legality and capacity constraints are applied, the number of ordered ways to select loops for parallel execution has the upper bound: \[ \mathcal{P}_{\text{parallel}} = \frac{n_{\text{loops}}!}{(n_{\text{loops}}-k_{\text{parallel}})!} \] where $n_{\text{loops}}$ is the number of loops, and $k_{\text{parallel}}$ is the number selected for parallel execution. For a six-loop computation where three loops are selected, this unconstrained count is: \[ \frac{6!}{(6-3)!} = 120. \]

Even for a single layer, the candidate count reaches hundreds before invalid or equivalent strategies are removed. Each candidate schedule imposes different demands on barrier synchronization, interconnect bandwidth, and PE utilization.

Memory placement and data movement

The physical memory hierarchy forces compilers to determine which tensors reside in registers, shared memory, or global DRAM at each stage of execution. Fetching operands from off-chip DRAM consumes orders of magnitude more energy and latency than reading from local SRAM, making data placement the primary determinant of achieved arithmetic intensity.

If each computational dimension had $n$ independent choices at every memory level, the resulting toy upper bound would be: \[ \mathcal{M}_{\text{placement}} = n^{N_{\text{comp}} \times N_{\text{mem}}} \] where:

$n$ = number of placement choices per level,
$N_{\text{comp}}$ = number of computational dimensions,
$N_{\text{mem}}$ = number of memory hierarchy levels.

For a model with:

$N_{\text{comp}} = 5$ computational dimensions,
$N_{\text{mem}} = 3$ memory levels,
$n = 4$ possible placement choices per level,

the number of possible memory allocations is: \[ 4^{5 \times 3} = 4^{15} = 1,073,741,824. \]

Mapping search space

This unconstrained mapping example exceeds a billion combinations, although many are illegal, coupled, or equivalent. Multiplying the toy counts gives an upper-bound illustration of the mapping search space: \[ \mathcal{S}_{\text{mapping}} = \left( n^{N_{\text{comp}}} \times n_{\text{loops}}! \times \frac{n_{\text{loops}}!}{(n_{\text{loops}}-k_{\text{parallel}})!} \right)^{N_{\text{mem}}} \] where:

$n^{N_{\text{comp}}}$ represents memory placement choices,
$n_{\text{loops}}!$ accounts for computation ordering choices,
$\frac{n_{\text{loops}}!}{(n_{\text{loops}}-k_{\text{parallel}})!}$ captures parallelization possibilities,
$N_{\text{mem}}$ is the number of memory hierarchy levels.

This equation is not an exact count of legal schedules because choices interact and hardware constraints prune the space. It illustrates why candidate spaces grow rapidly and why exhaustive search becomes impractical. Even within a small layer, altering a single ordering decision can produce an order-of-magnitude difference in memory traffic.

Example 1.4: Loop ordering in a small convolution

Consider a convolution applying 16 filters of size $3{\times}3$ to an $8{\times}8$ single-channel input. The computation can be expressed as five nested loops iterating over output rows ($H_{\text{out}}$), output columns ($W_{\text{out}}$), filter count ($C_{\text{out}}$), filter height ($F_h$), and filter width ($F_w$). The $5! = 120$ permutations are an unconstrained upper bound: legal orderings preserve the mathematical result, although floating-point reduction order can change rounding, and they generate different memory traffic.

Ordering A (weight-stationary): Place the filter loops ($C_{\text{out}}$, $F_h$, $F_w$) outermost and the spatial loops ($H_{\text{out}}$, $W_{\text{out}}$) innermost. Each $3{\times}3$ filter is loaded into registers once and then applied across all 36 output positions before the next filter is loaded. Total weight loads: $16 \times 9 = 144$ values, each loaded exactly once.

Ordering B (output-stationary): Place the spatial loops outermost and the filter loops innermost. For every output position, all 16 filters must be loaded, applied, and their partial sums accumulated before advancing to the next position. If the register file cannot hold all 16 filters simultaneously, filters are repeatedly fetched from cache or DRAM. In the worst case, each of the 36 output positions reloads all 144 filter weights, producing $36 \times 144 = 5{,}184$ weight reads.

Systems insight: Under these two limiting assumptions, Ordering A reduces modeled weight reads by 36$\times$ relative to the worst case for Ordering B. Real kernels also account for activation and partial-sum traffic, register capacity, cache behavior, and vectorization. The example nevertheless shows why two mathematically equivalent loop orders can place very different demands on the memory system.

This combinatorial explosion presents a central puzzle in ML systems: if the unconstrained search space contains billions of candidates, how do compilers and kernel developers routinely achieve near-peak hardware efficiency? Rather than evaluating arbitrary permutations, production systems restrict their search to a small family of canonical dataflow patterns—spatial and temporal schedules engineered to exploit the specific reuse properties of neural network operators.

Self-Check: Question

In neural network hardware mapping, what distinguishes a spatial mapping decision from a temporal mapping decision?
1. Spatial mapping refers to compiling graph IR, whereas temporal mapping refers to runtime CUDA kernel launches
2. Spatial mapping determines precision formats (FP16 vs INT8), whereas temporal mapping determines memory allocation sizes
3. Spatial mapping assigns computational tasks to specific physical execution units (e.g., PEs or SMs) simultaneously in parallel, whereas temporal mapping determines the execution ordering and loop scheduling over time on those units
4. Spatial mapping operates only on convolutional layers, whereas temporal mapping operates only on transformer attention layers
Explain why finding the optimal hardware mapping (tiling sizes, loop orders, and spatial partitioning) for a deep neural network on a target accelerator is a combinatorially hard optimization problem.
Explain why reordering loop nests in a tensor contraction (e.g., changing from $I \to J \to K$ to $K \to I \to J$) alters memory bandwidth demands and execution speed without changing the total mathematical operation count.
When mapping a tensor computation to a multi-level memory hierarchy, what is the primary objective function optimized by spatial and temporal tiling?
1. Maximizing the total number of intermediate tensors written to host DDR memory
2. Maximizing data reuse in the fastest, closest memory tiers (registers and SRAM) to minimize traffic to slower, energy-expensive DRAM
3. Ensuring every warp thread executes different instruction streams simultaneously
4. Converting all 2D matrix multiplications into 1D scalar operations

See Answers →

Dataflow Optimization

Spatial and temporal mapping (section 1.6) establish where operations execute and where operands reside, but dataflow optimization governs the exact trajectories and schedules of data moving through processing elements and the memory hierarchy. In a systolic array or tiled vector unit, the temporal schedule and spatial routing of weights, inputs, and partial sums dictate how frequently operands must be fetched from energy-expensive DRAM rather than reused from local registers or scratchpads. By controlling off-chip traffic, the dataflow establishes the effective operational intensity of the kernel, determining whether execution achieves the compute-bound plateau or stalls in the memory-bound regime of the Roofline model. Optimizing compilers (section 1.8) and runtime systems (section 1.9) therefore tailor dataflow patterns to match the reuse characteristics of each layer to the hardware constraints of the target accelerator.

Three recurring architectural decisions structure this design space:

Locality: Stationary operand selection (weight-stationary, output-stationary, or input-stationary) dictates which tensor remains pinned in local registers or processing-element scratchpads, trading off memory traffic against register pressure.
Organization: Physical tensor layouts (such as channels-last NHWC versus channels-first NCHW) govern whether memory accesses match hardware memory controller burst sizes and SIMD vector widths, preventing uncoalesced memory transactions.
Combination: Kernel fusion and loop tiling partition iteration spaces and merge adjacent operations to eliminate round-trips to off-chip memory for intermediate tensors.

Together, these patterns provide a compact vocabulary for navigating dataflow choices without exhaustive search across billions of candidate schedules.

Building blocks of mapping strategies

These three decisions translate into four foundational techniques: data movement patterns that fix specific operands near the arithmetic units (weight-stationary, output-stationary, and input-stationary), memory-efficient tensor layouts (channels-last versus channels-first), kernel fusion that keeps intermediate activations in on-chip SRAM between operations, and loop tiling that bounds working set sizes to fit local memory capacity. Rather than rediscovering schedules from scratch, heuristic compilers and auto-tuning frameworks compose these building blocks to construct efficient execution plans across deep neural network layers.

Data movement patterns

On modern accelerators, accessing an operand from off-chip DRAM consumes roughly two to three orders of magnitude more energy than executing an arithmetic operation or fetching from a local register file (Chen et al. 2016). Whether a workload executes dense matrix operations that exceed on-chip cache capacity or irregular sparse lookups, data movement strategy dictates overall execution latency and total power consumption. If data cannot be staged and supplied to processing elements at the rate required by their arithmetic throughput, the execution units stall on memory latency, stranding theoretical compute capacity and inflating memory subsystem energy.

To prevent arithmetic pipelines from stalling, a data movement strategy must capture the inherent algorithmic reuse of tensor operations within local, low-energy storage. Listing 15 illustrates the canonical three-nested-loop implementation of matrix multiplication, exposing the operand reuse across weights, inputs, and partial sums that a hardware mapping must capture in local storage.

Listing 15: Naïve Matrix-Multiplication Dataflow: The loop nest revisits weight rows, input columns, and output accumulators, exposing the reuse that a mapping must capture in local memory.

## Matrix multiplication where:
## weights: [512x256] - model parameters
## input:   [256x32]  - batch of activations
## Z:       [512x32]  - output activations

## Computing each output element Z[i,j]:
for i in range(512):
    for j in range(32):
        for k in range(256):
            Z[i, j] += weights[i, k] * input[k, j]

This computation reveals several critical dataflow challenges. The first challenge is the number of memory accesses required. For each output $Z_{ij}$, the computation must fetch an entire row of weights from the weight matrix and a full column of activations from the input matrix. Since the weight matrix contains 512 rows and the input matrix contains 32 columns, this results in repeated memory accesses that place a heavy burden on memory bandwidth.

The second challenge comes from weight reuse. The same weights are applied to multiple inputs, meaning that an ideal mapping strategy should maximize weight locality to avoid redundant memory fetches. Without proper reuse, the accelerator would waste bandwidth loading the same weights multiple times (Chen et al. 2018).

The third challenge involves the accumulation of intermediate results. Since each element in $Z_{ij}$ requires contributions from 256 different weight-input pairs, partial sums must be stored and retrieved before the final value is computed. If these intermediate values are stored inefficiently, the system will require frequent memory accesses, further increasing bandwidth demands.

One way to mitigate these challenges is to use SIMD and SIMT execution models, which allow multiple values to be fetched in parallel. However, even with these optimizations, data movement remains a bottleneck. The primary bottleneck is not retrieval speed alone, but transfer frequency and placement within the memory hierarchy (Han et al. 2016).

Han, Song, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J. Dally. 2016. “EIE: Efficient Inference Engine on Compressed Deep Neural Network.” 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), 243–54. https://doi.org/10.1109/isca.2016.30.

Because moving data dominates the energy budget that figure 9 charts, the single most important goal of an accelerator is to minimize memory access. Dataflow strategies achieve this by maximizing data reuse. The central decision is which data is most valuable to keep local. Accelerators answer that decision by determining which data remains fixed in memory and which data streams dynamically: weight-stationary keeps model parameters local, input-stationary maintains activation data, and output-stationary preserves intermediate results. Each approach trades off different memory access patterns to maximize data reuse and minimize the energy-intensive transfers that constitute the primary bottleneck in AI acceleration.

Weight stationary

The weight stationary strategy keeps weights fixed in local memory, while input activations and partial sums are streamed through the system. Weight stationary approaches prove particularly beneficial in CNNs and matrix multiplications, where the same set of weights is applied across multiple inputs. By ensuring weights remain stationary, this method reduces redundant memory fetches, which helps alleviate bandwidth bottlenecks and improves energy efficiency.

A key advantage of weight stationary is that it maximizes weight reuse, reducing the frequency of memory accesses to external storage. Since weight parameters are often shared across multiple computations, keeping them in local memory eliminates unnecessary data movement, lowering the overall energy cost of computation. This makes it particularly effective for architectures where weights represent the dominant memory overhead, such as systolic arrays and custom accelerators designed for machine learning. Listing 16 demonstrates how Weight Stationary execution keeps weights fixed in local memory while streaming inputs and accumulating partial sums.

Listing 16: Weight Stationary Dataflow: Weights stay resident in local memory while inputs and partial sums stream through, minimizing parameter read traffic; best for CNNs and matrix multiplications with heavy weight reuse.

## Weight Stationary Matrix Multiplication
## - Weights remain fixed in local memory
## - Input activations stream through
## - Partial sums accumulate for final output

for weight_block in weights:  # Load and keep weights stationary
    load_to_local(weight_block)  # Fixed in local storage
    for input_block in inputs:  # Stream inputs dynamically
        for output_block in outputs:  # Compute results
            output_block += compute(weight_block, input_block)
            # Reuse weights across inputs

In weight stationary execution, weights are loaded once into local memory and remain fixed throughout the computation while inputs stream dynamically, reducing redundant memory accesses. Partial sums accumulate efficiently, minimizing unnecessary data movement. Because weights need not be reloaded for each new computation, bandwidth requirements drop significantly, making this dataflow highly effective for workloads with heavy weight reuse patterns such as CNNs and matrix multiplications.

However, while this strategy reduces weight-related memory traffic, it introduces trade-offs in input and output movement. Since inputs must be streamed dynamically while weights remain fixed, the efficiency of this approach depends on how well input activations can be delivered to the computational units without causing stalls. Partial sums, which represent intermediate results, must also be carefully accumulated to avoid excessive memory traffic. The total performance gain depends on the size of available on-chip memory, as storing larger weight matrices locally can become a constraint in models with millions or billions of parameters.

The weight stationary strategy is well-suited for workloads where weights exhibit high reuse and memory bandwidth is a limiting factor. It is commonly employed in CNNs, systolic arrays, and matrix multiplication kernels, where structured weight reuse leads to measurable performance improvements. However, for models where input or output reuse is more critical, alternative dataflow strategies, such as output stationary or input stationary, may provide better trade-offs.

Output stationary

Weight stationary keeps weights local and streams inputs through the system. The dominant cost shifts, however, when the bottleneck is not weight loading but the frequent writes of partial sums. In fully connected layers and transformer attention mechanisms, each output element accumulates contributions from hundreds or thousands of weight-input pairs. Writing those intermediate partial sums to external memory after every accumulation step would create a write-bandwidth bottleneck far more severe than the read overhead that weight stationary addresses. The output stationary strategy inverts the priority: it keeps partial sums fixed in local memory while streaming both weights and input activations through the system, so that each output element is written to external memory only once, after all its contributions have been accumulated (Chen et al. 2016).

Chen, Yu-Hsin, Joel S. Emer, and Vivienne Sze. 2016. “Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks.” 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), 367–79. https://doi.org/10.1109/ISCA.2016.40.

Listing 17 demonstrates how accumulating partial sums locally minimizes memory writes and enhances efficiency during matrix multiplication. In this implementation, the accumulator buffer stays in local registers or scratchpad throughout the inner loop; weights and inputs stream in, contribute to the running sum, and are discarded. The final result is written out only once per output element, eliminating the repeated write traffic that would otherwise dominate bandwidth.

This approach aligns naturally with systolic arrays, where computation progresses through a grid of processing elements and partial sums can flow along one axis without leaving the chip. The trade-off is that both weights and activations must now be streamed dynamically, so the system must sustain high read bandwidth for two data streams simultaneously. Parallel implementations also require careful synchronization when multiple PEs contribute to the same output element. Output stationary is therefore most effective for workloads where accumulation dominates, such as fully connected layers and attention mechanisms, but less suitable when input reuse is the critical bottleneck.

Listing 17: Output Stationary Dataflow: Partial sums stay resident in local memory while weights and inputs stream through, so each output is written out only once, minimizing accumulation write traffic; best for fully connected layers and attention.

## - Partial sums remain in local memory
## - Weights and input activations stream through dynamically
## - Final outputs are written only once

for output_block in outputs:  # Keep partial sums stationary
    accumulator = 0  # Initialize accumulation buffer
    for weight_block, input_block in zip(weights, inputs):
        accumulator += compute(weight_block, input_block)
        # Accumulate partial sums
    store_output(accumulator)  # Single write to memory

Input stationary

The two strategies examined so far each fix a different operand in local memory: weight stationary fixes weights to reduce read bandwidth for parameters, and output stationary fixes partial sums to reduce write bandwidth for accumulations. The third strategy completes the picture by fixing the remaining operand: input activations. Within a transformer layer, one activation can feed multiple projections or attention heads; the next layer receives a newly computed representation rather than the same token activation. When activation reuse is the dominant memory cost, keeping inputs stationary and streaming weights through the system can reduce energy and bandwidth. Listing 18 illustrates this approach.

Listing 18: Input Stationary Dataflow: Input activations stay resident in local memory while weights stream through, reducing activation read traffic when one activation tile feeds several computations.

## - Input activations remain in local memory
## - Weights stream through dynamically
## - Partial sums accumulate and are written out

for input_block in inputs:  # Keep input activations stationary
    load_to_local(input_block)  # Fixed in local storage
    for weight_block in weights:  # Stream weights dynamically
        for output_block in outputs:  # Compute results
            output_block += compute(weight_block, input_block)
            # Reuse inputs across weights

Here, input activations are loaded once and held fixed in local storage while weights stream through. Partial sums accumulate and are eventually written out, but unlike output-stationary dataflow, the accumulation buffer is not the primary beneficiary of locality; instead, the input activations are.

The trade-off mirrors the other two strategies: weights stream dynamically, requiring sustained read bandwidth from memory, and partial sums require buffering before write-back. Input stationary is most effective when an activation tile feeds multiple projections, attention heads, or gate evaluations before eviction.

Taken together, the three dataflow strategies provide a decision framework rather than a hierarchy of quality. Weight stationary minimizes parameter read traffic, fitting convolutional networks with compact, heavily reused filters. Output stationary minimizes write traffic for accumulations, fitting fully connected layers with wide fan-in. Input stationary minimizes activation read traffic, fitting transformers and large-batch inference with high activation reuse. No single strategy dominates; the optimal choice depends on which tensor dimension provides the highest reuse ratio relative to on-chip storage capacity. Hardware compilers select among these strategies based on workload dimensions and memory hierarchy limits. Convolution-specific designs add a fourth label, row-stationary (as in Eyeriss), but it is a hybrid that keeps rows of inputs and partial sums local when that mapping yields higher reuse than fixing any single operand (section 1.4.6.2).

Memory-efficient tensor layouts

The dataflow strategies in section 1.7.1.1 determine which operands stay close to compute units; tensor layouts determine whether those operands can be accessed efficiently once they arrive. A well-chosen weight-stationary schedule still stalls if weights are laid out in a format that produces strided, non-coalesced memory fetches. Tensor layout represents a kernel contract: the physical ordering of multidimensional data in linear memory must match the memory access stride expected by the accelerator’s execution units, or the processor pays in memory stalls, cache underutilization, and redundant bus transactions.

Hardware accelerators fetch data from device memory in fixed-size transactions—typically 32 to 128 bytes per transaction. When compute units request non-contiguous addresses, each transaction delivers only a fraction of useful payload, degrading effective memory bandwidth. Aligning the tensor layout with the hardware execution model ensures that concurrent threads or SIMD vector lanes access contiguous memory locations in a single unified transaction (NVIDIA Corporation 2021).

Machine learning frameworks and compilers often manage this layout selection automatically. Optimization tools such as cuDNN for NVIDIA GPUs, XLA for TensorFlow and JAX, and MLIR-based compiler stacks transform tensor layouts as they lower operations to backend-specific kernels (NVIDIA Corporation 2021; Google 2025; Lattner et al. 2020). In high-level frameworks, layout transformations are typically applied transparently, but developers authoring custom kernels or targeting low-level APIs such as CUDA, Metal, or OpenCL must manage tensor formats directly.

In PyTorch, for example, view operations such as tensor.permute() adjust tensor strides without moving data in memory, requiring an explicit tensor.contiguous() call to materialize a packed buffer when a downstream kernel requires contiguous storage (PyTorch Contributors 2026). Conversely, TensorFlow defaults to channels-last (NHWC) for convolutional models on NVIDIA GPUs, warning that NCHW inputs introduce automatic transpose overhead when lowered to GPU kernels (TensorFlow Developers 2024). Hardware-optimized libraries such as cuDNN on GPUs and oneDNN on CPUs implement tailored memory formats to maximize cache line utilization and SIMD throughput. The governing principle is to treat memory layout as an intrinsic property of the backend execution path: the fastest tensor format is the one that minimizes layout conversion overhead and makes memory accesses contiguous across hardware execution units.

PyTorch Contributors. 2026. Tensor Views.

TensorFlow Developers. 2024. Optimize TensorFlow Performance Using the Profiler.

Channels-last layout

In an NHWC channels-last layout, the channel index varies fastest in a contiguous tensor, placing the channel values for each spatial coordinate adjacent in memory. NHWC specifies a logical dimension ordering rather than a linear storage convention; both NHWC and NCHW tensors represent linear storage sequences of their respective dimension orders.

Consider an RGB image tensor of shape $(\text{Height}, \text{Width}, \text{Channels}) = (3, 3, 3)$. The elements are laid out in linear memory as follows: \[\begin{gather*} I(0,0,0), I(0,0,1), I(0,0,2), I(0,1,0), I(0,1,1), \\ I(0,1,2), I(0,2,0), I(0,2,1), I(0,2,2), \ldots \end{gather*}\]

All channel values for a given pixel are adjacent, and memory addresses advance across width before advancing down height. This layout supports vectorized execution when a compute kernel processes multiple channels simultaneously: CPU SIMD registers and GPU Tensor Cores can load contiguous channel vectors directly without intra-register shuffling.

The advantage of channels-last storage depends on the target operator and hardware unit. TensorFlow uses NHWC by default, whereas PyTorch provides a dedicated channels-last memory format that decouples logical dimension ordering from physical storage. Modern Tensor Core kernels in cuDNN favor NHWC because matrix-multiply units pack channel dimensions directly into accumulator tiles. When kernels require channel-contiguous inputs, passing an incompatible format forces the framework runtime to insert an explicit memory transpose, incurring a memory bandwidth penalty before computation begins.

Channels-first layout

In a contiguous NCHW channels-first layout, the spatial width index varies fastest, and all spatial values across an entire channel are stored contiguously before the next channel begins. Accelerators process data in parallel across thread warps or SIMD lanes; when consecutive threads in a warp access consecutive memory addresses, the memory controller coalesces those requests into a single memory transaction (memory coalescing).³⁵

³⁵ Memory coalescing: The GPU hardware mechanism that fuses memory requests from threads in a warp into a single transaction when those threads access contiguous memory. Tensor layout affects coalescing, but the sign of the effect depends on the kernel: NCHW can be efficient for some convolution implementations, while NHWC/channels-last is often preferred for modern Tensor Core convolution and fusion paths. Poor layout choices can still create multi-fold performance gaps, but the correct fix is to match the layout to the backend rather than memorize one universal format.

Consider the same RGB image tensor with dimensions reordered in channels-first order as $(\text{Channels}, \text{Height}, \text{Width}) = (3, 3, 3)$. Retaining $I(h,w,c)$ as semantic coordinate notation, the data are stored in channels-first order as follows: \[\begin{gather*} I(0,0,0), I(0,1,0), I(0,2,0), I(1,0,0), I(1,1,0), I(1,2,0), \ldots, \\ I(0,0,1), I(0,1,1), I(0,2,1), \ldots, I(0,0,2), I(0,1,2), I(0,2,2), \ldots \end{gather*}\]

In this format, all red channel values for the entire image are stored first, followed by all green values, and then all blue values. This arrangement allows kernels that parallelize across spatial coordinates $(H, W)$ to issue coalesced reads along image rows, which suited earlier GPU convolution algorithms and SIMD execution pipelines (Chetlur et al. 2014).

Chetlur, Sharan, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. 2014. “cuDNN: Efficient Primitives for Deep Learning.” arXiv Preprint arXiv:1410.0759.

³⁶ NHWC vs. NCHW: NHWC lists dimensions as batch, height, width, channel; NCHW lists batch, channel, height, width. Physical memory format determines whether adjacent threads read adjacent addresses, so the performance effect is backend-specific. Layout-to-hardware mismatch is not a micro-optimization; it can create multi-fold performance gaps, especially when a conversion prevents Tensor Core convolution or fusion paths from being used.

The relative advantage of NCHW versus NHWC depends on how the underlying kernel maps computation to hardware threads. Convolutional layers evaluate filters across spatial and channel dimensions simultaneously. In classic direct or FFT-based convolutions, channels-first ordering enabled efficient spatial sliding without stride jumps. However, modern GEMM-lowered convolutions and Tensor Core pipelines map channel reduction directly to matrix multiply accumulators, favoring channels-last structures. When a model operates in NCHW but invokes a Tensor Core-optimized backend, the runtime must either run an unoptimized fallback kernel or dynamically transpose the tensor into NHWC, burning memory bandwidth in both cases.³⁶

Comparing channels-last and channels-first layouts

Both channels-last (NHWC) and channels-first (NCHW) layouts address specific hardware access patterns, with their efficiency determined by whether a kernel parallelizes over spatial coordinates or vectorizes across channel dimensions. Layout choice directly governs cache line utilization, memory bus transaction count, and compute unit occupancy. Table 18 contrasts the performance trade-offs and hardware compatibility between these two approaches.

Table 18: Data Layout Strategies: Channels-last (NHWC) and channels-first (NCHW) layouts optimize memory access patterns for different backend kernels. NHWC often suits CPU vectorization and modern Tensor Core convolution/fusion paths, while NCHW remains common in PyTorch model code and many GPU kernels. Choosing the appropriate layout directly impacts performance by maximizing cache utilization and memory bandwidth efficiency.

Feature	Channels-Last (NHWC)	Channels-First (NCHW)
Memory storage order	Pixels are stored row-by-row, channel interleaved	All values for a given channel are stored together first
Best for	CPU loops, element-wise operations, many channels-last kernels	Many legacy GPU convolution paths and channel-first model code
Cache efficiency	High cache locality for sequential row/channel-last access	Can improve coalescing for channel-first kernels
Convolution performance	Often preferred by modern Tensor Core convolution/fusion paths	Efficient for many cuDNN and framework kernels
Memory fetching	Good when kernels vectorize over adjacent channel data	Good when kernels process channel-major tiles
Default in frameworks	Common TensorFlow convention; PyTorch channels-last option	Common PyTorch tensor shape convention

While compilers such as XLA, TensorRT, and PyTorch Inductor automate layout propagation and insert layout conversions when beneficial, format conversions are not free: transposing a tensor requires reading every element from memory and writing it back in a different order, consuming memory bandwidth without executing useful arithmetic. In production deployments, matching the model’s memory layout to the accelerator’s native kernel format eliminates these layout transformations entirely.

Even when tensor layout is perfectly aligned with the hardware, executing operations as isolated kernels introduces a separate bottleneck: intermediate values must still travel to and from device memory between operations. Resolving that bottleneck requires changing how operations are scheduled rather than how tensors are laid out.

Kernel fusion

Reducing intermediate data movement between operations is one of the primary mechanisms for mitigating the memory wall. Kernel fusion³⁷ combines multiple successive operations into a single executable kernel, eliminating intermediate memory round trips.

³⁷ Kernel fusion: Separate GPU kernels expose intermediate results through device memory, although caches can absorb some traffic. A fused kernel can keep intermediates in registers or local storage and avoid corresponding external write/read cycles. The resulting speedup depends on cache behavior, occupancy, launch overhead, and whether memory traffic was the bottleneck; it is not necessarily proportional to the byte reduction.

Intermediate memory write

Neural network execution is frequently bound by memory bandwidth rather than arithmetic throughput. When operations execute as isolated kernels, each intermediate result is written to device memory (DRAM or HBM) and subsequently read back by the downstream operator. This round trip introduces memory latency, stalls compute units, and consumes bus bandwidth on data with zero reuse outside that immediate pair of operations.

This hardware reality provides the physical motivation for the operator fusion techniques introduced in Operator fusion. While section 1.4.1 analyzed bandwidth limits in terms of low operational intensity, chained elementwise operations aggravate the memory wall by multiplying global memory transactions (NVIDIA Corporation 2017).

NVIDIA Corporation. 2017. NVIDIA Tesla V100 GPU Architecture. NVIDIA Whitepaper.

In a naïve execution model, each operation executes as a separate GPU kernel, forcing intermediate tensors to round-trip through device memory (listing 19). In a chain of three operations—such as ReLU, batch normalization, and affine scaling—the intermediate outputs $X_1$ and $X_2$ are written to DRAM only to be immediately re-read by the subsequent kernel. For elementwise operators where arithmetic intensity is low ($I \ll 1\text{ FLOP/byte}$), the time spent moving data across the memory bus outweighs the computation time (Jia et al. 2019). Furthermore, holding these un-fused intermediate buffers in memory inflates the live-tensor footprint during execution (table 19).

Listing 19: Naïve Execution: Each step writes intermediate results to memory before processing the next, leading to increased bandwidth usage and reduced efficiency.

import torch
import torch.nn.functional as F

## Input tensor
X = torch.randn(1024, 1024).cuda()
running_mean = torch.zeros(1024, device=X.device)
running_var = torch.ones(1024, device=X.device)

## Step-by-step execution (naïve approach)
X1 = torch.relu(X)  # Intermediate tensor stored in memory
X2 = F.batch_norm(
    X1, running_mean, running_var, training=False
)  # Frozen inference BatchNorm
Y = 2.0 * X2 + 1.0  # Final result

Kernel fusion for memory efficiency

The two intermediate tensors consume memory capacity when they remain live and create external traffic when separate kernels materialize them. Kernel fusion can eliminate those intermediate writes and reloads (Jia et al. 2019) by propagating values through registers or local memory within one kernel.

Table 19: Intermediate Tensor Storage: If all four named tensors remain live in this naïve example, they occupy 16.8 MB; an optimizing runtime may reuse buffers when earlier values are no longer needed. Fusion primarily guarantees removal of the external write and read for each eliminated intermediate rather than a universal reduction in peak allocated memory.

Tensor	Size (MB) for $1024{\times}1024$ Tensor
X	4.2 MB
X’	4.2 MB
X’’	4.2 MB
Y	4.2 MB
Total Memory	16.8 MB

A machine learning inference sequence might apply ReLU, frozen batch normalization, and then an affine scaling with scale $\alpha$ and offset $\beta$. In a naïve implementation, each separate kernel generates an intermediate tensor that is written to memory and read back by the next kernel: \[ \begin{aligned} \mathbf{X}' &= \text{ReLU}(\mathbf{X}) \\ \mathbf{X}'' &= \text{BatchNorm}(\mathbf{X}') \\ \mathbf{Y} &= \alpha \cdot \mathbf{X}'' + \beta \end{aligned} \]

With kernel fusion, these operations are combined into a single computation step, allowing the entire transformation to occur without generating unnecessary intermediate tensors: \[ \mathbf{Y} = \alpha \cdot \text{BatchNorm}\big(\text{ReLU}(\mathbf{X})\big) + \beta \]

Table 20 highlights the impact of operation fusion on memory efficiency. By keeping intermediate results in registers or local memory rather than writing them to main memory, fusion reduces external traffic. For three separate pointwise inference kernels, the idealized naïve path reads and writes one tensor per operation, whereas the fused path reads the input once and writes the output once.

Table 20: Operation Fusion Benefits: In this idealized inference sequence, fusion reduces external tensor traffic from 25.2 MB to 8.4 MB, a 3$\times$ reduction. Actual traffic depends on cache behavior and the compiler’s buffer reuse.

Execution Model	External tensor payloads	Traffic (MB)
Naïve execution	3 reads + 3 writes	25.2 MB
Fused execution	1 read + 1 write	8.4 MB

Performance benefits and constraints

Kernel fusion brings several key advantages that enhance memory efficiency and computation throughput. By reducing memory accesses, fused kernels ensure that intermediate values stay within registers instead of being repeatedly written to and read from memory. This significantly lowers memory traffic, which is one of the primary bottlenecks in machine learning workloads. GPUs and TPUs, in particular, benefit from kernel fusion because high-bandwidth memory is a scarce resource, and reducing memory transactions leads to better utilization of compute units (NVIDIA Corporation 2020a).

However, not all operations can be fused arbitrarily. Element-wise operations and frozen inference normalization are strong candidates because their computations do not require batch-wide reductions. Training-mode batch normalization computes batch statistics and is not purely element-wise. Matrix multiplications and convolutions constrain fusion because they involve reductions and large data movement; they are often fused with element-wise epilogues such as bias or activation, but cannot be freely fused with unrelated global operations.

Another major consideration is register pressure. Fusing multiple operations means all temporary values must be kept in registers rather than memory. While this eliminates redundant memory writes, it also increases register demand. If a fused kernel exceeds the available registers per thread, the compiler can spill excess values to per-thread local memory, which is backed by device memory and may be cached, introducing additional latency and potentially negating the benefits of fusion. On GPUs, where thread occupancy (the number of threads that can run in parallel) is limited by available registers, excessive fusion can reduce parallelism, leading to diminishing returns.

Different AI accelerators and compilers handle fusion in distinct ways. NVIDIA GPUs, for example, favor warp-level parallelism, where element-wise fusion is straightforward (NVIDIA Corporation 2020a). TPUs, on the other hand, prioritize systolic array execution for dense matrix operations (Jouppi et al. 2017). Compiler and inference stacks such as TVM, XLA, TensorRT, and MLIR apply graph rewrites, lowering passes, or engine-building heuristics to balance memory savings against execution constraints (Chen et al. 2018; Google 2025; NVIDIA 2024b; Lattner et al. 2020).

Despite its advantages, fusion is not always beneficial. Some AI frameworks allow developers to disable fusion selectively, especially when debugging performance issues or making frequent model modifications. The decision to fuse operations must consider trade-offs between memory efficiency, register usage, and hardware execution constraints to ensure that fusion leads to tangible performance improvements.

Checkpoint 1.3: Data movement and kernel fusion

The dataflow building blocks are now in place: data locality, tensor layout, and kernel fusion. These fusion decisions are ultimately about data locality, which ties together the chapter’s core data movement strategies:

Locality choice: given weight-stationary, output-stationary, and input-stationary dataflows, which tensor does each hold near the compute units, and how does that choice shift where the memory traffic lands?
Layout match: a kernel runs on a backend expecting NHWC but receives an NCHW tensor. Predict what happens to its memory access pattern and to delivered throughput.
Fusion eligibility: for a Conv2D, a BatchNorm, and a ReLU executed in sequence, decide which pairs are worth fusing and what determines whether fusing them pays off.
Remaining gap: locality, layout, and fusion are all chosen well, yet the operation still stalls on memory. What is the missing transformation, and why do the other three not address it?

Memory-efficient tiling strategies

While modern AI accelerators offer high computational throughput, their performance is often limited by memory bandwidth rather than raw processing power. If data cannot be supplied to processing units fast enough, execution stalls occur, leading to wasted cycles and inefficient hardware utilization.

Tiling³⁸ mitigates this issue by restructuring computations into smaller, memory-friendly subproblems. When memory bandwidth cannot be increased directly, systems must reduce trips to main memory. Instead of processing entire matrices or tensors at once, which leads to excessive memory traffic, tiling partitions computations into smaller blocks (tiles) that fit within fast local memory (for example, caches, shared memory, or registers) (Lam et al. 1991).

³⁸ Tiling (loop blocking): This restructuring partitions a computation into blocks sized for a fast local tier. A naive loop makes repeated references to the same elements; caches may capture some, while blocking deliberately concentrates reuse before eviction (Lam et al. 1991). This reduction in lower-tier traffic is a primary source of the gap between naive matrix multiplication and optimized GEMM routines.

Lam, Monica D., Edward E. Rothberg, and Michael E. Wolf. 1991. “The Cache Performance and Optimizations of Blocked Algorithms.” Proceedings of the Fourth International Conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS-IV, 63–74. https://doi.org/10.1145/106972.106981.

Matrix multiplication, widely used in AI models, demonstrates inefficient memory access when implemented naively. Listing 20 shows how, without tiling, repeated memory accesses for the same data lead to unnecessary bandwidth consumption.

Listing 20: Naïve Matrix Multiplication: Direct implementation makes $\mathcal{O}(N^3)$ array references for $N{\times}N$ matrices. Caches can serve some references, but the loop order does not deliberately block reuse for the memory hierarchy.

for i in range(N):
    for j in range(N):
        for k in range(N):
            C[i, j] += A[i, k] * B[k, j]  # Repeatedly fetching
            # A[i, k] and B[k, j]

The loop repeatedly references elements of $\mathbf{A}$ and $\mathbf{B}$. Some references can hit in cache, but large matrices and an unfavorable traversal order can still cause excessive lower-tier traffic.

Tiling addresses this problem by ensuring that smaller portions of matrices are loaded into fast memory, reused efficiently, and only written back to main memory when necessary. This technique is especially important in AI accelerators, where memory accesses dominate execution time. Figure 14 labels the product $\mathbf{C} = \mathbf{A}\mathbf{B}$ by its three dimensions: $M$ rows and $K$ columns in $\mathbf{A}$, $K$ rows and $N$ columns in $\mathbf{B}$, and $M{\times}N$ in the output $\mathbf{C}$. Each highlighted tile is the working set that fits in fast memory at one moment: a green $M_{\text{tile}}{\times}K_{\text{tile}}$ row band of $\mathbf{A}$ multiplies a pink $K_{\text{tile}}{\times}N_{\text{tile}}$ column band of $\mathbf{B}$ to accumulate one blue $M_{\text{tile}}{\times}N_{\text{tile}}$ output block ($\text{Block}_{m,n}$) of $\mathbf{C}$. Processing all computations for one tile before moving to the next avoids repeatedly paying the DRAM access penalty.

Figure 14: **Matrix Tiling Mechanics**: Partitioning large matrix multiplication ($\mathbf{C}=\mathbf{A}\mathbf{B}$) into $M_{\text{tile}}\times K_{\text{tile}}$ and $K_{\text{tile}}\times N_{\text{tile}}$ blocks aligns memory footprints with fast on-chip SRAM/registers. Maximizing register reuse within each tile amortizes slow off-chip DRAM accesses and keeps compute engines operating at peak throughput.

Tiling fundamentals

Tiling divides a computation into smaller tiles that fit within available fast memory rather than operating on an entire data structure at once. This structure maximizes data reuse, reducing redundant memory accesses and improving efficiency.

Consider matrix multiplication, a key operation in machine learning workloads. The operation computes $\mathbf{C} = \mathbf{A} \times \mathbf{B}$ where each element $C_{ij} = \sum_{k} A_{ik} \times B_{kj}$. Listing 20 exposes the core problem: every iteration of the innermost loop fetches elements from matrices $\mathbf{A}$ and $\mathbf{B}$ from memory, performs a multiplication, and updates matrix $\mathbf{C}$. Because matrices are large, the processor repeatedly reloads the same values from memory, even though they were just used in previous computations.

This data movement overhead is expensive: DRAM access has much higher latency and energy cost than access to on-chip cache or registers (Horowitz 2014; Sze et al. 2017). The solution is tiling.

Performance benefits of tiling

Instead of computing one element at a time and constantly moving data in and out of slow memory, tiling processes submatrices (tiles) at a time, keeping frequently used values in fast memory. The idea is to divide the matrices into smaller blocks that fit within the processor’s cache or shared memory, ensuring that once a block is loaded, it is reused multiple times before moving to the next one. Listing 21 demonstrates cache-friendly loop blocking: the loop bounds partition the matrices into tiles, and the hardware cache hierarchy keeps recently used tile data close to the compute units when access order has locality.

Listing 21: Cache-Blocked Matrix Multiplication: This high-level loop blocking approach divides matrices into smaller index ranges so hardware caches can reuse data within each tile, improving computational efficiency without explicit scratchpad loads.

TILE_SIZE = 32  # Choose a tile size based on hardware constraints

# Cache blocking: partition data via loop bounds.
# Loads are implicit through the hardware cache hierarchy.
for i in range(0, N, TILE_SIZE):
    for j in range(0, N, TILE_SIZE):
        for k in range(0, N, TILE_SIZE):
            # Each tile computed independently
            for ii in range(i, min(i + TILE_SIZE, N)):
                for jj in range(j, min(j + TILE_SIZE, N)):
                    for kk in range(k, min(k + TILE_SIZE, N)):
                        C[ii, jj] += A[ii, kk] * B[kk, jj]

This restructuring significantly improves performance through three reinforcing effects. Memory reuse improves because the approach visits a small tile repeatedly while it is likely to remain in cache before moving on to the next tile, rather than fetching elements from $\mathbf{A}$ and $\mathbf{B}$ repeatedly from slow memory, which minimizes redundant memory accesses. Memory bandwidth usage drops as a direct consequence: since each tile is used multiple times before being evicted, most required data is available in L1/L2 cache or shared memory rather than DRAM, so traffic falls and execution speeds up. Compute efficiency rises in turn, because processors spend less time waiting for data and more time performing useful work; in architectures like GPUs and TPUs, where thousands of parallel processing units operate simultaneously, tiling keeps data read and processed in a structured manner that avoids unnecessary stalls.

This technique is particularly effective in AI accelerators, where machine learning workloads consist of large matrix multiplications and tensor transformations. Without tiling, these workloads quickly become memory bound, meaning performance is constrained by how fast data can be retrieved rather than by the raw computational power of the processor.

Tiling methods

While the general principle of tiling remains the same, which involves partitioning large computations into smaller subproblems to improve memory reuse, there are different ways to apply tiling based on the structure of the computation and hardware constraints. The two primary tiling strategies are spatial tiling and temporal tiling. These strategies optimize different aspects of computation and memory access, and in practice, they are often combined to achieve the best performance.

Spatial tiling partitions data structures into smaller blocks that fit within fast memory. The tiled matrix multiplication in listing 21 demonstrates cache-friendly loop blocking: the code does not issue explicit scratchpad loads, but the tile-shaped access pattern lets hardware caches reuse nearby values before they are evicted. This strategy is particularly beneficial for large tensors that exceed fast memory capacity—by breaking computations into smaller tiles, data movement between memory levels is minimized, keeping operations localized within cache hierarchies.

Temporal tiling complements spatial tiling by explicitly staging data in shared memory or registers and reorganizing the computation order around that staged data. Many ML workloads access the same data repeatedly across iterations—without temporal tiling, this results in redundant memory fetches. Temporal tiling restructures the computation to ensure that frequently used data stays in fast memory for as long as possible before the next computation begins.

A classic example where temporal tiling is beneficial is convolutional operations, where the same set of weights is applied to multiple input regions. Without loop blocking, these weights might be loaded from memory multiple times for each computation. With temporal tiling, the computation is reordered so that the weights remain in fast memory across multiple inputs, reducing unnecessary memory fetches and improving overall efficiency. Listing 22 illustrates explicit tile staging: the code loads blocks of $\mathbf{A}$ and $\mathbf{B}$ into temporary fast storage, then reuses them across multiple inner-loop operations.

Listing 22: Explicit Tile Staging Pseudocode: A backend maps the temporary tile objects to fast storage and reuses them across inner-loop operations.

# Explicit tile staging: load data into fast
# temporary storage before the inner loops.
for i in range(0, N, TILE_SIZE):
    for j in range(0, N, TILE_SIZE):
        for k in range(0, N, TILE_SIZE):
            # Pseudocode: map these tile objects to fast memory
            A_tile = A[i : i + TILE_SIZE, k : k + TILE_SIZE]
            B_tile = B[k : k + TILE_SIZE, j : j + TILE_SIZE]

            # Reuse loaded tiles for all inner iterations
            for ii in range(A_tile.shape[0]):
                for jj in range(B_tile.shape[1]):
                    for kk in range(A_tile.shape[1]):
                        C[i + ii, j + jj] += (
                            A_tile[ii, kk] * B_tile[kk, jj]
                        )

Explicit tile staging improves performance when the backend places the temporary tiles in fast memory and reuses them before eviction. The pseudocode includes shortened boundary tiles when $N$ is not divisible by TILE_SIZE.

This technique is particularly useful in workloads where certain values are used repeatedly, such as convolutions, recurrent neural networks (RNNs), and self-attention mechanisms in transformers. By applying loop blocking, AI accelerators can significantly reduce memory stalls and improve execution throughput.

Tiling challenges and trade-offs

Tiling improves performance only when the tile matches the locality budget of the hardware. If the tile is too small, memory fetches still dominate execution time because reuse is too limited. If the tile is too large, it exceeds fast memory and causes cache thrashing or scratchpad spills. Selecting the right tile size therefore directly determines computational efficiency and memory bandwidth usage.

The tile choice also controls load balance. In architectures such as GPUs and TPUs, computations execute in parallel across thousands of processing units. If tiles are not evenly distributed, some units remain idle while others are overloaded, leading to suboptimal utilization of computational resources. Effective tile scheduling keeps parallel execution balanced and efficient.

Data movement remains the limiting cost even after tiling. Although tiling reduces the number of slow memory accesses, transferring tiles between hierarchy levels still incurs latency and energy cost, especially when data falls from cache or scratchpad back to DRAM. Efficient memory prefetching and scheduling strategies minimize this residual movement and ensure that data is available when needed.

Hybrid tiling combines spatial and temporal strategies when neither dimension alone captures the workload’s reuse pattern. Some AI accelerators use spatial tiling for matrix multiplications while employing temporal tiling for weight reuse in convolutional layers, dynamically adjusting tile sizes or reordering computations based on real-time execution conditions.

Register blocking, double buffering, and hierarchical tiling extend the same locality principle at smaller and larger memory tiers. AI compilers and runtime systems such as TensorFlow XLA, TVM, and MLIR automatically select these tiling strategies based on hardware constraints, enabling fine-tuned performance optimization without manual intervention. Table 21 provides a comparative overview of spatial, temporal, and hybrid tiling approaches, highlighting their respective benefits and trade-offs.

When machine learning models grow in size and complexity, tiling remains a critical tool for improving hardware efficiency, ensuring that AI accelerators operate near their practical potential. While manual tiling strategies can provide substantial benefits, compilers and hardware-aware optimization techniques further enhance performance by automatically selecting effective tiling strategies for a given workload.

Table 21: Tiling Strategies: Spatial, temporal, and hybrid tiling optimize memory access patterns for improved performance. Spatial tiling maximizes data reuse within fast memory, temporal tiling exploits loop structure for reduced accesses, and hybrid tiling combines both approaches. AI compilers and runtime systems use these techniques to automatically optimize model execution on diverse hardware.

Aspect	Spatial Tiling (Data Tiling)	Temporal Tiling (Loop Blocking)	Hybrid tiling
Primary Goal	Reduce memory accesses by keeping data in fast memory longer	Increase data reuse across loop iterations	Adapt dynamically to workload constraints
Optimization Focus	Partitioning data structures into smaller, memory-friendly blocks	Reordering computations to maximize reuse before eviction	Balancing spatial and temporal reuse strategies
Memory usage	Improves cache locality and reduces DRAM access	Keeps frequently used data in fast memory for multiple iterations	Minimizes data movement while ensuring high reuse
Common use cases	Matrix multiplications, CNNs, self-attention in transformers	Convolutions, recurrent neural networks (RNNs), iterative computations	AI accelerators with hierarchical memory, mixed workloads
Performance gains	Reduced memory bandwidth requirements, better cache utilization	Lower memory fetch latency, improved data locality	Maximized efficiency across multiple hardware types
Challenges	Requires careful tile size selection, inefficient for workloads with minimal spatial reuse	Can increase register pressure, requires loop restructuring	Complexity in tuning tile size and execution order dynamically
Best when	Data is large and needs to be partitioned for efficient processing	The same data is accessed multiple times across iterations	Both data partitioning and iteration-based reuse are important

Applying mapping strategies to neural networks

While these foundational mapping techniques apply broadly, their effectiveness varies based on the computational structure, data access patterns, and parallelization opportunities of different neural network architectures. Each architecture imposes distinct constraints on data movement, memory hierarchy, and computation scheduling, requiring tailored mapping strategies to optimize performance.

A structured approach to mapping is required to address the combinatorial explosion of choices that arise when assigning computations to AI accelerators. Rather than treating each model as a separate optimization problem, the same principles apply across different architectures; only their priority shifts based on workload characteristics. The goal is to systematically select and apply mapping strategies that maximize efficiency for different types of machine learning models.

These principles apply to three representative AI workloads, each characterized by distinct computational demands. CNNs benefit from spatial data reuse, making weight-stationary execution and tiling especially effective. Transformer behavior depends on phase and shape: training and prefill matrix multiplications can be compute bound, while low-batch decoding often depends on weight and KV-cache bandwidth. MLPs involve substantial matrix multiplication and benefit from structured tiling, optimized weight layouts, and memory-aware execution.

Despite their differences, each of these models follows a common set of mapping principles, with variations in how optimizations are prioritized. Table 22 summarizes the suitability of different optimization strategies for CNNs, transformers, and MLPs.

Convolutional neural networks (ResNet-50)

For ResNet-50 and similar CNNs, a key mapping opportunity is spatial weight reuse. A small filter is applied across many spatial locations, so its weights participate in many multiply-accumulate operations. Weight-stationary mappings can capture that reuse by retaining filter tiles locally, while row-stationary or output-stationary mappings may better balance activation and partial-sum traffic on other architectures. Favorable convolution shapes can achieve high arithmetic intensity, which makes them well suited to matrix and systolic execution paths. The model family does not determine one universal dataflow: layer shape, batch size, local-storage capacity, and the target’s supported kernels decide which reuse pattern is worth retaining.

Table 22: Architecture-Specific Mapping Strategies: Each neural network architecture benefits from different optimization priorities based on its computational and memory characteristics.

Optimization technique	CNNs	Transformers	MLPs	Rationale
Dataflow strategy	Weight, row, or output stationary	Phase- and kernel-dependent	Weight or output stationary	CNNs expose filter and spatial reuse; transformer attention and MLP blocks impose different traffic; dense MLPs reuse weights across a batch and accumulate outputs.
Memory-aware tensor layouts	Backend-dependent (often NCHW for cuDNN convolutions, channels-last on Tensor Cores)	Backend-dependent (row-major activations typical)	Row-major typical	Layout choice depends on backend kernel path and precision mode (see section 1.7.1.2); the entries name common defaults, not universal prescriptions.
Kernel Fusion	Convolution + Activation	Fused Attention	GEMM Fusion	CNNs optimize convolution+activation fusion; Transformers fuse attention mechanisms; MLPs benefit from fused matrix multiplications.
Tiling for Memory Efficiency	Spatial Tiling	Temporal Tiling	Blocked Tiling	CNNs tile along spatial dimensions; Transformers use loop blocking to improve sequence memory efficiency; MLPs use blocked tiling for large matrix multiplications.

This spatial regularity also enables aggressive fusion and tiling. For inference with frozen BatchNorm statistics, compilers can fold or fuse convolution, normalization, and activation to avoid unnecessary intermediate writes. Training-mode BatchNorm requires reductions over batch statistics and does not have the same element-wise fusion contract. Spatial tiling partitions the feature map into subregions sized to fit within on-chip SRAM, so the fused kernel processes each tile from fast memory before moving to the next.

Transformer architectures (GPT-2/Llama)

Where CNNs are defined by weight reuse, transformers are defined by the memory pressure of the key-value (KV) cache. During attention computation, every query vector must access stored key and value pairs across the entire sequence length. As sequences grow, the full KV cache usually lives in HBM or device DRAM, while attention kernels tile active blocks through SRAM, registers, or shared memory. This access pattern motivates activation stationary execution: keep the currently used KV tiles close to compute while streaming queries through them, rather than repeatedly materializing large attention intermediates in external memory.

The memory traffic created by standard attention implementations explains why fused attention kernels, such as FlashAttention (Dao et al. 2022), can deliver large performance gains: by fusing the query-key dot product, softmax normalization, and value-weighted summation into a single kernel that tiles along the sequence dimension, these implementations avoid materializing the full attention matrix in main memory. This temporal tiling approach processes sequence blocks that fit within on-chip SRAM, substantially reducing HBM traffic while preserving the $\mathcal{O}(S^2)$ attention computation. Transformer mapping depends on the phase and shape: training and prefill GEMMs can be compute-bound, while low-batch autoregressive decode is commonly constrained by weight and KV-cache bandwidth.

Multilayer perceptrons and DLRM

MLPs present the most straightforward mapping problem because their computation reduces largely to dense GEMM.³⁹ Each fully connected layer multiplies an activation matrix by a weight matrix. The weight matrix is fixed across samples, so batching reuses each loaded weight across more activations and raises arithmetic intensity. A batch size of one provides little cross-sample weight reuse and often underutilizes wide matrix engines, while larger batches can move execution toward the compute-bound regime. General matrix multiply (GEMM) derives the scaling that governs this sensitivity.

³⁹ GEMM: The operation $\mathbf{C} = \alpha \mathbf{A}\mathbf{B} + \beta \mathbf{C}$ is the dense linear-algebra primitive that many deep-learning layers lower to. Optimized GEMM libraries such as cuBLAS and oneDNN use register blocking, vectorization, and hierarchical tiling to approach hardware limits on favorable shapes (NVIDIA 2024a; Intel Corporation 2021b). Modern AI accelerators are heavily specialized for GEMM-like tiles: Tensor Cores, systolic arrays, and matrix extensions all exist to accelerate this primitive, which is why GEMM performance is an important predictor of end-to-end throughput across architectures.

NVIDIA. 2024a. cuBLAS: CUDA Basic Linear Algebra Subprograms.

Intel Corporation. 2021b. oneDNN: Intel’s Deep Learning Neural Network Library.

Naumov, Maxim, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, et al. 2019. “Deep Learning Recommendation Model for Personalization and Recommendation Systems.” arXiv Preprint arXiv:1906.00091.

Because MLP layers are typically followed by activation functions and bias additions, GEMM fusion combines these steps into a single kernel, avoiding intermediate memory writes. Blocked tiling partitions the large matrix multiplications into sub-blocks sized for the accelerator’s shared memory, ensuring high cache utilization throughout computation. The simplicity of the MLP mapping, dominated by a single primitive with predictable access patterns, is precisely why hardware vendors optimize GEMM libraries so aggressively: gains in GEMM performance translate directly to MLP throughput. DLRM is not purely an MLP workload, however. Its large sparse embedding tables and feature-interaction stage can dominate capacity and memory traffic, so the dense mapping described here applies to its bottom and top MLP towers rather than to the entire model (Naumov et al. 2019).

Hybrid mapping strategies

The preceding architectural subsections treat each architecture in isolation, but real models rarely consist of a single layer type. A vision transformer, for example, combines a patch embedding stage, self-attention layers, and MLP blocks (Dosovitskiy et al. 2021). Those layers create different reuse patterns: the embedding stage can benefit from weight-stationary mapping, attention emphasizes activation movement and tiling, and MLP blocks demand blocked GEMM tiling and fusion. No single dataflow strategy is optimal across all these layers, so hardware mapping becomes hybrid and layer-specific.

Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, et al. 2021. “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.” International Conference on Learning Representations (ICLR).

Hybrid mapping addresses this heterogeneity by allowing the accelerator to switch strategies at layer boundaries. Each layer presents a different balance of compute intensity, data reuse, and memory access pattern, and the optimal mapping must shift accordingly (Sze et al. 2017). Rather than committing to one dataflow for the entire model, hybrid approaches select weight stationary execution for layers with high weight reuse, activation stationary execution for attention layers with large KV caches, and output stationary execution for layers where minimizing write traffic matters most.

Modern accelerators provide the architectural features needed to realize hybrid mapping in practice. TPU-style systolic arrays, NVIDIA GPUs, and tile-based accelerator designs expose different combinations of local memory, tensor layouts, fusion, and scheduling controls, allowing compilers and runtimes to choose layer-specific strategies rather than one global dataflow for the whole model (Jouppi et al. 2023; NVIDIA Corporation 2020a; Chen et al. 2018). These implementations require programmable memory hierarchies, efficient interconnects, and specialized execution pipelines, reinforcing the hardware-software co-design principle.

However, hybrid mapping remains a design-time optimization. In production workloads, execution conditions change dynamically due to varying input sizes, memory contention, and hardware resource availability. Machine learning compilers and runtime systems extend these static mapping choices by introducing dynamic scheduling, memory optimizations, and automatic tuning, ensuring that deep learning workloads operate efficiently across diverse accelerators and deployment environments.

Self-Check: Question

Why do modern deep learning libraries (e.g., cuDNN, TensorRT) strongly prefer the Channels-Last (NHWC) tensor layout over Channels-First (NCHW) when executing convolutions on NVIDIA Tensor Cores?
1. NHWC eliminates the need for spatial convolutions by flattening images into 1D vectors
2. NCHW requires floating-point numbers to be stored in big-endian byte order
3. NHWC reduces model parameter count by sharing channel weights across batches
4. NHWC places channel values for a given spatial location in contiguous memory, aligning with the packed vector/matrix multiply requirements of Tensor Cores
Explain the mechanism of vertical kernel fusion (e.g., fusing Conv2D $\to$ BatchNorm $\to$ ReLU), and identify both its performance benefit and its primary architectural constraint.
Place the following loop transformation steps in the logical order applied by an optimizing compiler when targeting a matrix multiplication dataflow to an accelerator:

Loop Reordering: Permute loop indices to establish a specific stationary dataflow (e.g., Output-Stationary)
Loop Unrolling: Fully or partially unroll the innermost loop to expose instruction-level parallelism and map to hardware registers
Loop Tiling (Blocking): Partition global loop iterations into sub-tiles that fit into on-chip shared memory / SRAM
Spatial Partitioning: Assign outer tile loops to physical hardware compute clusters (e.g., GPU thread blocks / SMs)

In a multi-head self-attention layer where a single input activation tensor is projected across multiple query, key, and value weight matrices ($W_q, W_k, W_v$), which stationary dataflow strategy provides the highest data reuse in local scratchpad memory?
1. Input-Stationary (Activation-Stationary)
2. Output-Stationary
3. No Local Reuse (NLR)
4. Weight-Stationary
True or False: Row-Stationary (RS) dataflow, as implemented in architectures like Eyeriss, keeps only the final output activation stationary in registers while streaming 2D convolutional filter rows and input rows from DRAM on every clock cycle.

See Answers →

Compiler Support

A single convolution can be implemented with dozens of valid tiling strategies, kernel variants, and memory layouts, most of which perform poorly on a given target. Machine learning compilers navigate this complexity by translating high-level dataflow representations into target-specific executable code. Compiling ResNet-50 for GPU inference illustrates the core compiler stages:

Graph optimization fuses repeated Conv2D-BatchNorm-ReLU patterns into single compound kernels, eliminating intermediate round-trips to HBM.
Kernel selection chooses specialized Tensor Core implementations for compatible matrix multiplications, exploiting the high arithmetic intensity of dense convolutions.
Memory planning determines activation lifetimes to safely overlap and reuse intermediate buffer allocations.
Computation scheduling pipelines memory copies with execution across independent hardware queues to hide data transfer latencies.

In this scenario, inference latency drops from 47 ms to 8 ms, a 5.9× improvement without altering the model’s mathematical output. These numbers reflect how graph rewrites, kernel selection, memory planning, and scheduling systematically apply kernel fusion (section 1.7.1.3) and memory-efficient tiling (section 1.7.1.4) to convert theoretical hardware FLOP/s into delivered throughput (Chen et al. 2018; NVIDIA 2024b).

This pipeline embodies the hardware-software co-design principle introduced in Acceleration Fundamentals. Machine learning compilers bridge high-level model specifications and low-level execution engines by restructuring computations, selecting hardware-specialized kernels, and planning tensor storage lifetimes (Chen et al. 2018). They build on classical compiler foundations while introducing tensor- and graph-level transformations.

ML compiler design

Machine learning compilers specialize conventional compilation techniques for programs expressed as tensor graphs and operator libraries. General-purpose compilers optimize scalar control flow, vector instructions, register allocation, and cache locality; ML compiler stacks operate at a higher semantic level, optimizing graph topology, tensor memory layouts, shape specialization, accelerator kernel dispatch, and inter-device execution planning (Li et al. 2021).

Li, Mingzhen, Yi Liu, Xiaoyan Liu, Qingxiao Sun, Xin You, Hailong Yang, Zhongzhi Luan, Lin Gan, Guangwen Yang, and Depei Qian. 2021. “The Deep Learning Compiler: A Comprehensive Survey.” IEEE Transactions on Parallel and Distributed Systems 32 (3): 708–27. https://doi.org/10.1109/tpds.2020.3030548.

Table 23 contrasts the emphasis of these two compiler layers. The boundary is complementary rather than disjoint: an ML compiler stack lowers high-level tensor operations into intermediate representations that feed into an LLVM, GPU, or vendor-specific backend.

Table 23: Compiler Optimization Priorities: ML compiler stacks add tensor- and graph-level decisions to conventional instruction- and loop-level compilation. In a production toolchain, the two layers commonly cooperate rather than replace one another.

Aspect	Traditional compiler	Machine learning compiler
Input representation	Source or intermediate-representation program	Tensor graph or tensor-level intermediate representation
Execution Model	Control flow, loops, threads, vectors, and tasks	Tensor operators, kernels, graphs, and accelerator streams
Optimization priorities	Instruction selection, loop transforms, vectorization, registers	Graph rewrites, fusion, layouts, shapes, and kernel selection
Memory management	Objects, stack/heap, caches, locality, and prefetching	Tensor lifetimes, buffer reuse, layouts, tiling, and device transfers
Target Hardware	CPUs, GPUs, and other programmable targets	CPUs, GPUs, TPUs, and custom accelerators
Compilation output	Machine code or lower-level intermediate representation	Kernels plus a hardware-specific graph or execution plan

The distinction in table 23 explains why compiler configuration dictates delivered performance even when the underlying neural network code remains unmodified. ML compilers manage the translation layer between high-level tensor operations and hardware execution units; when this mapping fails to optimize data reuse or layout alignment, arithmetic units stall while memory buses saturate.

Systems Perspective 1.2: The hidden optimization layer

Most practitioners never interact directly with ML compilers, yet compiler quality determines what fraction of peak accelerator throughput a model achieves. Calling model.compile() in Keras, torch.compile() in PyTorch, or deploying an engine through TensorRT invokes multi-stage optimization pipelines that execute four transformations:

Fuse operations never combined by the developer (such as merging Conv2D, BatchNorm, and ReLU into a single kernel)
Reorder computations to maximize data reuse in fast local memory (such as tiling matrix multiplications)
Select kernels from libraries containing hundreds of hand-tuned, architecture-specific implementations
Transform tensor layouts between what the framework API expects and what hardware memory controllers require

The same model definition runs at vastly different throughputs depending on the compilation backend, graph lowering choices, kernel selection, and runtime configuration. When execution latency exceeds service targets, compiler configuration and backend lowering are primary optimization levers that require no alterations to model architecture or training parameters.

ML compilation pipeline

Machine learning models defined in modern frameworks are represented as high-level dataflow graphs describing operations on multidimensional tensors. Lowering these representations into executable code for CPUs, GPUs, TPUs, or custom ASICs requires an ML compilation pipeline (Chen et al. 2018; Google 2025; Lattner et al. 2020).

A standard compilation pipeline executes five interconnected responsibilities:

Graph optimization restructures the computation to eliminate redundant work and reduce data traffic.
Kernel selection maps mathematical operations to optimized target-specific implementations.
Memory planning assigns tensor layouts, static buffer offsets, and lifetime bounds.
Computation scheduling orders kernels and memory transfers to maximize parallelism and hide transfer latencies.
Code generation emits target-specific machine instructions, assembly, or interpreted bytecode.

Across these stages, the compiler applies the dataflow strategies developed in section 1.7: kernel fusion, memory tiling, and compute placement. Hardware acceleration delivers its theoretical throughput only when these compiler-driven transformations expose sufficient arithmetic intensity and align memory access with the hardware hierarchy.

Graph optimization

Machine learning frameworks represent models as high-level computation graphs, where nodes represent mathematical operators (such as convolutions, matrix multiplications, and activations) and directed edges represent tensor data dependencies. Executing this graph node-by-node, as defined by framework frontends, incurs severe memory penalties: every edge forces an allocation and an off-chip memory transaction (DRAM or HBM write followed by a read), while every node incurs kernel launch and synchronization overhead. On accelerators where arithmetic throughput outpaces memory bandwidth by an order of magnitude, materializing unneeded intermediate tensors dominates total execution time.

Graph optimization transforms this computation graph into an efficient execution plan before hardware code generation (Chen et al. 2018; Jia et al. 2019). Working primarily at a target-independent intermediate representation (IR) level before lowering (section 1.8.6.2), graph passes apply four core transformations:

Jia, Zhihao, James Thomas, Todd Warszawski, Mingyu Gao, Matei Zaharia, and Alex Aiken. 2019. “Optimizing DNN Computation with Relaxed Graph Substitutions.” Proceedings of Machine Learning and Systems (MLSys).

Operator fusion merges adjacent operations into a single compound kernel to eliminate off-chip memory round-trips and launch overheads. In convolutional networks, fusing Conv2D, batch normalization, and ReLU into one kernel keeps activations in local registers or shared SRAM throughout the sequence (section 1.7.1.3). In transformer architectures, pattern matchers replace multi-operator self-attention subgraphs (matrix multiplication, scaling, masking, softmax, and projection) with monolithic, IO-aware attention kernels that tile computations entirely within fast on-chip memory (Dao et al. 2022).

Redundant computation elimination and constant folding identify and remove unnecessary operations. During inference compilation, parameters from static layers—such as batch normalization scale and offset vectors—are folded directly into the weights and biases of preceding linear or convolutional layers, eliminating normalization operators from runtime execution. Graph passes also prune dead subgraphs and merge identical parallel paths through common subexpression elimination.

Algebraic simplification and layout adjustments reorder operations where mathematically equivalent to minimize data movement. For example, transpositions can be pushed through elementwise operations or cancelled against inverse permutations, reducing runtime reshape overhead.

Memory-aware node reordering evaluates data dependencies across topological sorts of the graph to schedule independent operations in an order that minimizes peak live-tensor footprint. Reordering nodes ensures that intermediate activations are consumed and deallocated promptly, preventing memory pressure from forcing spills to host memory.

Compilers such as XLA, TVM, TensorRT, and MLIR execute these rewrites via pattern-matching rules and equality saturation. By eliminating unnecessary memory round-trips and exposing coarse-grained parallelism, graph optimization produces an execution graph that aligns with the accelerator’s bandwidth constraints. Once the graph is restructured into optimized compound operations, the compiler must bind each node to a concrete, hardware-specific execution routine.

Kernel selection

Kernel selection binds each node in the optimized graph to a concrete hardware implementation. Accelerators provide dozens of valid kernel implementations for a single tensor operation, each parameterized by tile dimensions, thread block configurations, memory staging patterns, and target precision. Choosing the optimal kernel determines whether the execution engine achieves high compute utilization or stalls on uncoalesced memory accesses and pipeline bubbles (Chen et al. 2018; Zheng et al. 2020).

In most production pipelines, compilers draw from vendor-optimized kernel libraries rather than synthesizing kernels from scratch. NVIDIA’s cuDNN and cuBLAS provide hand-tuned routines for GPUs, Intel’s oneDNN targets x86 architectures, the Arm Compute Library (ACL) targets Arm processors, and BLIS provides optimized CPU matrix primitives. These libraries encode microarchitectural details—such as register file capacity, cache hierarchy latency, and vector lane width—allowing the compiler to invoke pre-tuned implementations.

AI compilers employ three primary strategies to select among candidate kernels:

Rule-based selection relies on static heuristics⁴⁰ to map operators based on tensor shapes, data types, and hardware targets. For example, XLA selects Tensor Core-enabled GEMM kernels on NVIDIA GPUs whenever matrix dimensions are multiples of 8 or 16 and precision is set to FP16 or BF16. Heuristics incur zero compilation time overhead, though they can miss non-obvious optimal configurations on irregular tensor shapes.

⁴⁰ Heuristic in kernel selection: A practical rule that chooses a promising implementation without exhaustively benchmarking the full candidate space. Tile sizes, layouts, precision modes, fusion choices, and shape constraints can create many legal GEMM variants. Heuristics reduce tuning cost but may miss a faster candidate, which is why autotuners such as AutoTVM profile selected options on the target hardware.

Lattner, Chris, Mehdi Amini, Uday Bondhugula, Albert Cohen, Andy Davis, Jacques Pienaar, River Riddle, Tatiana Shpeisman, Nicolas Vasilache, and Oleksandr Zinenko. 2020. “MLIR: Scaling Compiler Infrastructure for Domain Specific Computation.” 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), 2–14. https://doi.org/10.1109/cgo51591.2021.9370308.

Cost model-based selection estimates kernel execution latency and memory traffic using analytical models of the hardware architecture. Frameworks such as MLIR provide infrastructure for cost models that predict register pressure, cache hit rates, and arithmetic intensity across parameter spaces, pruning unpromising candidates without running physical benchmarks (Lattner et al. 2020).

Profile-guided selection evaluates candidate kernels empirically on the target hardware. TVM’s AutoTVM and Ansor systems measure execution latency directly on the accelerator, navigating tile sizes, loop unrolling factors, and thread configurations via statistical search to converge on the fastest implementation (Chen et al. 2018; Zheng et al. 2020). Similarly, TensorRT builds hardware-specific execution engines by benchmarking tactic candidates directly against target batch sizes and memory constraints.

Precision-aware selection integrates numerical requirements into the selection process. While training requires FP32 or BF16 to maintain numerical stability during gradient accumulation, inference pipelines leverage FP16 or INT8 to maximize arithmetic throughput and halve memory traffic. A selection error here impacts correctness as well as speed: routing an uncalibrated model to an INT8 kernel degrades model accuracy, while defaulting to FP32 leaves Tensor Core engines underutilized.

Once kernels are selected, memory planning and computation scheduling determine exact buffer allocations, launch sequences, and stream concurrency.

Memory planning

Memory planning assigns physical storage locations, buffer offsets, and tensor lifetimes across the accelerator’s memory hierarchy (Roesch et al. 2018; Chen et al. 2018). High-performance execution plans fail if memory management introduces uncoalesced memory transactions, bank conflicts, or unnecessary memory allocations that exhaust accelerator memory.

The compiler coordinates three core memory management mechanisms:

Tensor layout optimization determines the physical arrangement of tensor dimensions in memory. Different hardware architectures and compute engines require specific memory layouts to enable coalesced global memory access and vectorization (section 1.7.1.2). For example, NVIDIA Tensor Cores achieve peak throughput with the channels-last NHWC layout for FP16 and INT8 convolutions, where channel dimensions align with 16-byte memory transactions, whereas legacy FP32 CUDA cores historically favored channels-first NCHW (NVIDIA Corporation 2021; Google 2025). Compilers perform layout assignment globally across the graph to prevent back-to-back layout transformation kernels that consume memory bandwidth.

Google. 2025. XLA: Optimizing Compiler for Machine Learning.

Roesch, Jared, Steven Lyubomirsky, Logan Weber, Josh Pollock, Marisa Kirisame, Tianqi Chen, and Zachary Tatlock. 2018. “Relay: A New IR for Machine Learning Frameworks.” Proceedings of the 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 58–68. https://doi.org/10.1145/3211346.3211348.

Liveness analysis and buffer reuse prevent out-of-memory errors by sharing physical memory across non-overlapping tensors. Deep neural networks generate dozens of intermediate activation tensors during forward propagation. Allocating a unique memory buffer for every tensor would quickly exceed the capacity of device memory (such as 16 to 80 GB on typical accelerator GPUs). The compiler analyzes the graph to compute the liveness interval of each tensor—from its generation to the completion of its last consumer. By modeling memory allocation as an interval graph coloring or 2D bin-packing problem, the compiler assigns multiple non-overlapping intermediate tensors to the same physical memory offsets (Roesch et al. 2018). This static reuse reduces peak device memory requirements to the working set size of concurrently active tensors, avoiding runtime allocation overhead and fragmentation.

Memory hierarchy placement optimizes the staging of data between external memory (HBM or GDDR) and on-chip scratchpad memories (SRAM or shared memory). External memory bandwidth is a scarce resource; moving intermediate tiles back and forth across the memory bus creates memory-bound stalls. Compilers synthesize tiling parameters that guarantee intermediate blocks remain resident in on-chip SRAM until all local compute operations complete, minimizing round-trip traffic across the memory bus.

With memory layouts fixed and buffer addresses assigned, the compiler schedules operation execution across hardware compute resources.

Computation scheduling

Computation scheduling determines the execution timeline and hardware resource binding for operations in the compiled graph (Zheng et al. 2020). Modern accelerators feature deep hardware pipelines, multiple independent compute engines, and dedicated DMA copy engines. Effective scheduling coordinates these resources to maximize hardware occupancy, preserve data dependencies, and prevent pipeline stalls (Chen et al. 2018).

Implementation in AI compilers

Compilers implement scheduling across three distinct architectural dimensions:

Task partitioning maps computational subgraphs onto the target’s physical execution units. On GPUs, computations are partitioned into grids of thread blocks dispatched across Streaming Multiprocessors (SMs). On TPUs, matrix multiplications are partitioned into tile sequences matched to the physical dimensions of systolic arrays (e.g., $128 \times 128$ or $256 \times 256$ processing elements) (Norrie et al. 2021). On CPUs, tasks are split into thread pools matching physical cores and vectorized to fill SIMD registers.

Norrie, Thomas, Nishant Patil, Doe Hyun Yoon, George Kurian, Sheng Li, James Laudon, Cliff Young, Norman Jouppi, and David Patterson. 2021. “The Design Process for Google’s Training Chips: TPUv2 and TPUv3.” IEEE Micro 41 (2): 56–63. https://doi.org/10.1109/mm.2021.3058217.

Execution ordering resolves dependencies while minimizing live memory pressure. Operations that depend on prior results must wait for producer kernels to complete, but independent branches—such as multi-head attention projections or parallel residual paths—can execute concurrently or interleave to maintain uniform compute utilization. Compilers analyze the dependency graph to schedule operations in an order that keeps compute pipelines full while keeping activation lifetimes short, preventing buffer bloat.

Asynchronous overlap hides memory latency by executing data transfers concurrently with compute kernels. Accelerators feature dedicated DMA engines that operate independently of arithmetic pipelines. Compilers exploit this architectural separation by scheduling memory transfers (such as host-to-device weights or inter-accelerator halo exchanges) in separate hardware queues or CUDA streams, overlapping transfer latency behind long-running compute operations (Chen et al. 2018; NVIDIA 2024b). When compute and transfer overlap completely, data movement costs are hidden behind arithmetic execution.

Chen, Tianqi, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Q. Yan, Haichen Shen, Meghan Cowan, et al. 2018. “TVM: An Automated End-to-End Optimizing Compiler for Deep Learning.” Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’18), 578–94.

NVIDIA. 2024b. NVIDIA TensorRT: Programmable Inference Accelerator.

Zheng, Lianmin, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, et al. 2020. “Ansor: Generating High-Performance Tensor Programs for Deep Learning.” Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 863–79.

Poor scheduling introduces execution bubbles where arithmetic units sit idle waiting for kernel dispatches or serialized data copies. Synchronizing kernels via explicit compiler-generated events and stream schedules ensures the hardware sustains continuous execution without CPU driver intervention (NVIDIA 2024b; Zheng et al. 2020).

Code generation

With scheduling complete, the final compilation stage translates the scheduled execution plan into target-specific machine instructions. Code generation applies traditional backend compiler passes—instruction selection, register allocation, instruction scheduling, and peephole optimization—tailored to accelerator architectures.

Crucially, instruction selection for ML targets must emit instructions that directly engage specialized matrix ISA extensions. On NVIDIA GPUs, code generators emit PTX instructions such as mma.sync.aligned to invoke Tensor Cores directly, as shown in listing 14. On Intel CPUs with Advanced Matrix Extensions (AMX), the backend targets tile-multiply instructions operating on 2D register tiles. On Arm architectures with the Scalable Matrix Extension (SME), the target is outer-product accumulation across scalable vector tiles. A backend that emits generic floating-point vector instructions instead of these matrix extensions leaves the primary compute engines unutilized, reducing effective throughput by an order of magnitude.

For CPUs and GPUs, compilers emit machine code or optimized assembly; for TPUs, field-programmable gate arrays (FPGAs),⁴¹ and proprietary NPUs, the output is often an optimized bytecode or binary execution package executed by a specialized hardware runtime.

⁴¹ Field-programmable gate array: “Field-programmable” means the logic fabric is configurable after manufacturing, contrasting with fixed-function ASICs. FPGAs can improve performance for latency-sensitive data center services by implementing custom pipelines matched to a particular workload (Putnam et al. 2014). This reconfigurability makes FPGAs attractive for rapidly evolving ML architectures where committing to an ASIC risks obsolescence, but the requirement for hardware description languages (Verilog/VHDL) and compilation times measured in hours creates a productivity barrier that limits adoption to deployments where the efficiency benefit justifies the engineering cost.

Putnam, Andrew, Adrian M. Caulfield, Eric S. Chung, Derek Chiou, Kypros Constantinides, John Demme, Hadi Esmaeilzadeh, et al. 2014. “A Reconfigurable Fabric for Accelerating Large-Scale Datacenter Services.” ACM SIGARCH Computer Architecture News 42 (3): 13–24. https://doi.org/10.1145/2678373.2665678.

From compilation to runtime

A compiler transforms a high-level model into an execution plan tailored to target hardware, but that plan embeds static assumptions about runtime conditions: tensor shapes, workspace allocation limits, device capabilities, and execution concurrency. Ahead-of-time (AOT) compilation optimizes graph structures, fusion boundaries, and memory layouts for fixed shapes and known hardware configurations. While just-in-time (JIT) compilation can compile specialized variants dynamically as new shapes arrive, each compiled kernel still executes a fixed execution plan.

Production AI serving environments, however, operate under dynamic constraints that differ from static compiler assumptions. Request arrival rates fluctuate, batch sizes vary dynamically, multiple inference workloads compete for shared accelerator memory, and thermal throttling can reduce clock frequencies below rated specifications. Serving runtimes manage these variations through dynamic request batching, multi-stream concurrency, admission control, and execution engine selection, rather than regenerating kernel fusion or tiling strategies on the fly. The serving chapter addresses dynamic batching, admission control, and service-level objectives as system-level engineering challenges (Model Serving); the immediate systems concern is how the accelerator runtime dispatches compiled execution plans efficiently once a batch reaches the hardware.

Self-Check: Question

What is the primary purpose of lifetime analysis in an ML compiler’s static memory planner?
1. To calculate the physical degradation and failure rate of HBM memory cells over time
2. To determine the precise intervals during which each intermediate activation tensor is needed, allowing disjoint tensors to share the same physical memory buffer
3. To predict the number of training epochs required for a neural network to converge
4. To prevent the compiler from generating out-of-order instruction streams
Explain how double buffering (software pipelining) implemented by an ML compiler hides memory access latency during loop execution on an accelerator.
Place the following compilation stages in the correct order as an end-to-end ML compiler (such as TVM or XLA) transforms a high-level deep learning model into executable machine code:

Target Code Generation: Emit hardware-specific binary (e.g., PTX or machine instructions)
High-Level Graph Optimization: Perform operator fusion, constant folding, and dead code elimination on the computation graph
Front-End Ingestion: Parse framework model (e.g., PyTorch/ONNX) into High-Level Graph IR
Low-Level IR & Auto-Tuning: Lower fused operators to loop-level IR and optimize tile sizes, thread bindings, and unroll factors
Static Memory Planning: Analyze tensor lifetimes and allocate shared physical buffers

How does an auto-tuning ML compiler (such as TVM/Ansor) differ from a traditional handwritten library approach (such as cuDNN) for kernel selection?
1. Auto-tuning compilers execute code only on the host CPU, whereas handwritten libraries run on GPUs
2. Handwritten libraries search an infinite combinatorial loop space at runtime, whereas auto-tuning compilers use static heuristics
3. Auto-tuning compilers explore large parameterized search spaces of loop transformations and tile sizes using cost models to generate custom kernels, whereas handwritten libraries rely on expert-tuned templates for specific fixed shapes
4. Auto-tuning compilers require all tensors to be quantized to 1-bit integers
The optimization where an ML compiler merges two or more independent operators at the same graph depth into a single batched kernel to maximize GPU parallelism is known as ____ fusion.

See Answers →

Runtime Support

An AI runtime bridges static graph compilation and the dynamic realities of accelerator execution. Compilers optimize and lower computational graphs under fixed structural assumptions, but production systems encounter variable batch sizes, fluctuating sequence lengths, and constrained memory budgets. The runtime executes compiled engines while managing input-dependent tensor shapes, scratchpad allocations, asynchronous hardware streams, and precompiled execution profiles. In production inference stacks such as NVIDIA TensorRT, the offline builder optimizes the network and benchmarks candidate tactics to construct an execution engine, while the runtime engine binds input buffers, selects valid optimization profiles, and dispatches kernels to the accelerator (NVIDIA Corporation 2026a, 2026b).

NVIDIA Corporation. 2026a. How TensorRT Works.

NVIDIA Corporation. 2026b. Working with Dynamic Shapes.

Huang, Y., Y. Cheng, A. Bapna, O. Firat, D. Chen, M. X. Chen, H. Lee, et al. 2019. “GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism.” Advances in Neural Information Processing Systems (NeurIPS) 32: 103–12.

Mirhoseini, Azalia, Hieu Pham, Quoc V. Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. 2017. “Device Placement Optimization with Reinforcement Learning.” International Conference on Machine Learning (ICML) 70: 2430–39.

This execution lifecycle spans three core responsibilities. First, the runtime selects and dispatches precompiled kernel variants matched to runtime tensor dimensions and hardware capabilities. Second, it manages device memory through arena allocators and static buffer reuse, eliminating costly host-device synchronization during execution. Third, in distributed environments, framework runtimes coordinate cross-device execution boundaries, as demonstrated in pipeline-parallel frameworks such as GPipe (Huang et al. 2019) and automated device-placement systems (Mirhoseini et al. 2017). While serving schedulers oversee request batching and admission control across clusters, the accelerator runtime governs local execution on the device.

Comparing AI runtimes to traditional software execution environments clarifies why machine learning workloads demand specialized execution strategies.

ML runtime architecture

General-purpose runtimes manage threads, preemptive tasks, heap allocations, and asynchronous I/O across arbitrary control-flow paths. In contrast, an AI runtime operates over dataflow graphs composed of multidimensional tensors, pre-scheduled kernel grids, device memory arenas, and hardware command queues.

The architectural differences between traditional software runtimes and AI execution environments are contrasted in table 24.

Table 24: Runtime Execution Models: AI runtimes add tensor-, graph-, and accelerator-aware resource management to conventional runtime services. The exact boundary among framework, compiler runtime, device library, and serving scheduler varies by stack.

Aspect	Traditional runtime	AI runtime
Execution Model	Functions, threads, tasks, events, and asynchronous I/O	Tensor graphs, kernels, streams, and device events
Task scheduling	Threads and tasks across processor resources	Dependency-aware kernel and transfer dispatch
Memory Management	Objects, stacks, heaps, pools, and virtual memory	Tensor buffers, workspaces, device memory, and buffer reuse
Optimization Priorities	Responsiveness, throughput, locality, and resource sharing	Shape compatibility, transfer overlap, reuse, and utilization
Adaptability	Dynamic scheduling and allocation within program semantics	Runtime choices within compiled graphs and available kernels
Target Hardware	CPUs and heterogeneous devices	CPUs, GPUs, TPUs, and custom accelerators

Tensor memory management represents a primary architectural divergence. In traditional systems, general-purpose allocators manage variable-sized heap objects using runtime metadata. On AI accelerators, dynamic memory allocations through device-driver calls (such as cudaMalloc) invoke host-driver traps that stall the accelerator pipeline. AI runtimes eliminate this overhead by combining offline lifetime analysis with arena allocation. For fixed-shape graphs, the compiler computes static memory offsets, mapping non-overlapping intermediate tensors to shared physical buffers. For variable-shape tensors, the runtime carves memory from preallocated scratchpad pools, bounding peak memory footprint without runtime allocation calls.

Hardware execution environments also impose physical operating boundaries that diverge from offline benchmarking setups.

Production environments introduce physical constraints that differ from offline benchmarks. For example, an NVIDIA A100 SXM accelerator operates with a maximum TDP of 400 W, whereas the A100 PCIe variant is constrained to a 300 W TDP envelope. Because sustained clock frequencies scale with available power and thermal dissipation, a compiled kernel tuned for an SXM module may experience thermal throttling or fail to meet latency bounds on a PCIe card. The runtime cannot re-synthesize loop structures or change thread-block dimensions dynamically in response to thermal constraints; it must execute within precompiled profiles while the serving system handles request batching and admission control.

Dynamic kernel execution

While static compilation generates high-performance machine code for fixed shapes, production inference requires executing requests with dynamic dimensions. In language models, input sequence length $S$ and batch size $B$ vary across requests. Recompiling a computational graph dynamically when an unexpected shape arrives is impractical: exploring loop unrolling, shared-memory tiling, and register allocation requires seconds or minutes, while serving deadlines operate on millisecond timescales.

AI runtimes resolve this mismatch using precompiled optimization profiles. An optimization profile establishes valid dimension bounds (minimum, optimum, and maximum shapes) for dynamic tensor axes. During engine construction, the builder allocates scratchpad workspace sized for the maximum shape and selects kernel tactics that maximize arithmetic intensity across the defined operating range. At execution time:

When incoming tensor shapes match an active profile, the runtime binds tensor pointers to preallocated buffers and launches the corresponding precompiled kernels without JIT overhead.
When input dimensions fall outside all compiled profiles, the runtime cannot synthesize new tiling strategies on the fly; the serving system must either pad inputs to a valid dimension boundary, split the batch, or route the request to a slower, general-purpose fallback kernel.

Dynamic request processing also strains the host-to-accelerator interconnect. If input transfers proceed synchronously, accelerator compute engines sit idle during data movement. To overlap communication with computation, the runtime employs asynchronous streams and double buffering. By staging input tensors in page-locked (pinned) host memory, dedicated DMA copy engines transfer batch $n{+}1$ across PCIe or NVLink concurrently with the execution of batch $n$ on accelerator cores. This pipelining hides transfer latency provided the compute duration exceeds the transfer duration; if memory movement dominates, the accelerator pipeline stalls.

Runtime kernel selection

Within a selected optimization profile, an operation can be executed by multiple precompiled kernel implementations, termed tactics. The runtime selects the optimal tactic based on tensor dimensions, arithmetic intensity, and available scratchpad memory.

Operational batch size directly dictates the optimal execution strategy. For small batch sizes ($B=1$ during autoregressive token generation), matrix-vector multiplication is memory-bandwidth bound, with low arithmetic intensity. The runtime selects memory-streaming kernels (GEMV) optimized for global memory read throughput. For large prompt-processing batches ($B \ge 64$), the workload transitions to compute-bound GEMM. The runtime dispatches tactics designed for Tensor Core systolic arrays, using 2D register and shared-memory tiling to maximize data reuse from on-chip SRAM.

Scratchpad workspace availability further constrains tactic selection. When a GEMM has a small output dimension but a large reduction dimension $K$, standard tiling underutilizes the accelerator because too few thread blocks are launched to occupy all Streaming Multiprocessors (SMs). In this regime, the builder provides a split-$K$ tactic that partitions the reduction dimension across separate thread blocks running concurrently on different SMs. However, split-$K$ execution requires an auxiliary workspace buffer in device memory (HBM) to store partial reduction sums before a final accumulation pass. If device memory is constrained by concurrent requests, the runtime must reject the split-$K$ variant and dispatch a non-split kernel, sacrificing parallel occupancy to respect the memory ceiling.

Runtime selection remains strictly bounded by the numerical formats established during compilation. In mixed-precision training and inference systems such as Megatron-LM, executing matrix multiplications in FP16 or BF16 on Tensor Cores with FP32 accumulation is an architectural invariant validated prior to deployment (Shoeybi et al. 2019). The runtime does not monitor loss values or dynamically alter precision; it only chooses among the validated tactics that implement the specified numerical precision.

Shoeybi, Mohammad, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.” arXiv Preprint arXiv:1909.08053.

Kernel scheduling and utilization

Kernel dispatch represents the final stage of runtime execution, coordinating host-side task queues with device-side execution.

The software runtime manages host-side command queues (such as CUDA streams or TPU command rings) and records hardware events or semaphores to enforce dataflow dependencies asynchronously. When independent operations—such as parallel multi-head attention projections—exhibit no mutual data dependencies, the runtime dispatches them onto separate streams to enable concurrent execution on the accelerator. Once submitted to the device queue, however, execution control transfers entirely to hardware schedulers. On a GPU, the hardware warp scheduler and work distribution engine assign thread blocks to SMs based on register file availability, shared memory capacity, and maximum warp occupancy; the software runtime cannot schedule individual threads or dictate L1/L2 cache replacement policies. Similarly, TPU runtimes submit macro-instructions to on-chip command queues, while hardware sequencers feed systolic matrix units and vector units (Jouppi et al. 2017).

Memory management directly reinforces accelerator utilization. Calling device-driver memory allocators (such as cudaMalloc) during request execution forces global device synchronization, draining the execution pipeline. AI runtimes eliminate this latency by maintaining preallocated memory arenas and managing tensor lifecycles statically. Intermediate activations are allocated from scratchpad memory with overlapping lifetimes mapped to identical physical addresses, ensuring zero-allocation overhead on the inference hot path.

Even with aggressive kernel fusion, optimal tactic selection, and zero-allocation memory reuse, the execution capacity of a single accelerator is physically bounded by its silicon area, thermal dissipation limit, and high-bandwidth memory capacity. Scaling beyond these physical boundaries requires distributing computation across multiple accelerators.

Consider the estimated $3.14 \times 10^{23}$ floating-point operations associated with training GPT-3 (Brown et al. 2020). As a scale comparison, divide that historical operation count by a later H100’s peak FP8 rate of 1.98 PFLOP/s. The arithmetic lower bound is about 5 years on one device, and 8.4–12.6 years under the illustrative 40–60 percent utilization range (Choquette 2023). This is not a reconstruction of GPT-3 training: the precision, hardware, communication, and workload are counterfactual. It simply shows why model-scale compute or service-scale request volume can exceed the useful capacity of one accelerator.

Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. “Language Models Are Few-Shot Learners.” Advances in Neural Information Processing Systems 33: 1877–901. https://doi.org/10.48550/arxiv.2005.14165.

Self-Check: Question

Why do deep learning frameworks implement custom caching memory allocators (such as PyTorch’s caching_allocator) rather than directly invoking cudaMalloc and cudaFree for every intermediate tensor?
1. cudaMalloc operates only in FP32 precision and cannot allocate FP16 memory buffers
2. Direct OS memory allocation encrypts all tensor data, introducing cryptographic decryption latency
3. GPU DRAM can only be allocated once during system boot time
4. cudaMalloc is a synchronous operation that stalls GPU execution and causes expensive driver and OS page-table synchronization overhead
Explain how CUDA Graphs eliminate CPU kernel launch overhead during repeated training or inference iterations on small or low-latency models.
True or False: In an asynchronous GPU runtime model, when a Python script executes y = torch.matmul(a, b), the CPU thread blocks and waits until the GPU hardware finishes computing the matrix multiplication before executing the next line of Python code.
A sequence of asynchronous GPU operations that execute strictly in FIFO order on an accelerator is called a CUDA ____.
An inference serving system processes dynamic batch sizes ranging from 1 to 32 tokens per request. Why does dynamic batching pose a significant challenge to runtime kernel selection and hardware efficiency?
1. Optimal tile sizes, thread block configurations, and memory bandwidth requirements change across batch sizes, making a single static kernel inefficient across all shapes
2. Dynamic batching forces the GPU to switch from FP16 to FP64 precision for even batch sizes
3. Tensor Cores cannot execute matrix multiplication when batch dimensions are not powers of two
4. The GPU memory controller must physically power down DRAM banks when batch size decreases

See Answers →

Multi-Chip Scaling

A single H100 has a large FP8 peak, yet a training-time or throughput target may still require many accelerators. The techniques covered earlier remain the foundation for multi-chip scaling; each device still needs an efficient dataflow. The new lesson is that communication adds further hierarchy boundaries. Moving data through on-chip storage, HBM, an intra-node link, and a cluster fabric generally increases latency and energy, so scaling is not merely a question of adding chips.

When single-accelerator capacity proves insufficient, the design problem shifts from feeding one chip to choosing which communication boundary the workload can tolerate. Practitioners encounter these boundaries in production even when most optimization work remains inside a single accelerator.

Multi-chip scaling approaches

Large AI systems scale beyond individual accelerators by moving the communication boundary outward, and each boundary changes the dominant trade-off. The sequence begins inside the package. Chiplet-based architectures partition large designs into smaller, modular dies interconnected within one package, bypassing manufacturing limits of monolithic chips while preserving relatively low communication latency. The next boundary is the node: multi-accelerator servers connect several chips through board- or server-level interconnects. Each accelerator has dedicated memory and compute resources, so workloads split through data parallelism (each accelerator processes different batches) or model parallelism (either pipeline parallelism, where different accelerators handle sequential network layers, or tensor parallelism, where individual matrix multiplications are sliced across devices within high-bandwidth interconnect domains). High-bandwidth intra-node interconnects can enable efficient gradient synchronization, though realized performance depends on topology and collective communication efficiency.

Beyond the node, the boundary expands into the cluster. Purpose-built data center fabrics coordinate hundreds of accelerators, making topology and collective communication algorithms central determinants of scaling efficiency; near-linear scaling is achievable on some workloads when communication overhead is controlled. Wafer-scale integration is the counter-move: instead of pushing the boundary outward, it collapses more computation back into one large device. Platforms such as Cerebras WSE-class systems integrate extremely large numbers of transistors and cores on a single device, reducing inter-chip communication overhead while introducing their own challenges in thermal dissipation, fault tolerance, and manufacturing yield.

Why scaling introduces new constraints

The transition from single-chip to multi-chip architectures introduces communication overhead and other constraints that reshape optimization. Communication, synchronization, load balance, and parallel work per device can all limit scaling. Amdahl’s Law⁴² provides one first-order model for exposed work that does not shrink with additional devices. For hundred-billion-parameter models, an unsharded full-gradient payload can occupy hundreds of gigabytes per step, and an AllReduce⁴³ must aggregate it; precision, sharding, compression, and the collective algorithm change the bytes each device transfers.

⁴² Amdahl’s law (scaling limit): Amdahl's Law and Gustafson's Law formalizes the bound for a fixed nonaccelerated fraction. If 5 percent of a fixed workload remains exposed and unaffected by device count, that model caps speedup at 20$\times$. Distributed training is more complicated because communication can overlap computation and its cost can itself change with scale, but the example explains why exposed synchronization must be reduced or hidden.

⁴³ AllReduce: A collective operation from MPI that aggregates values across processes (the “reduce”) and distributes the result back to every process (the “all”). Here, the term identifies why accelerator-to-accelerator bandwidth matters for distributed training workloads. At scale, algorithms, topology choices, and runtime protocols determine how costly this synchronization becomes.

The first-order quantity is the gradient payload. For a model with $P$ parameters, that payload is roughly the parameter count multiplied by the bytes stored for each gradient element before optimizer state, padding, and protocol overhead enter. Scaling therefore improves only when the saved compute time exceeds the time to move this payload through the chosen interconnect and collective algorithm.

This overhead explains why large accelerator counts can show diminishing returns unless the system reduces exchanged data, overlaps communication with useful computation, or chooses a better parallelization pattern. The memory model also changes: separate accelerator memories are commonly managed explicitly, and collective operations establish the required synchronization rather than presenting one fully coherent memory image across the cluster. Coherent links exist in some systems, but their scope and cost are architecture-dependent.

Once computation spans many links, chips, and memory stacks, reliability and energy become part of the same scaling story. Large-scale systems must handle component failures gracefully because the probability of at least one failure rises with system size. The architectural principle is fundamental: distributed TPU systems must tolerate component failures at system scale (Jouppi et al. 2023), while Cerebras uses redundant cores and fabric links to replace manufacturing-defective cores and restore the wafer’s logical mesh (Lie 2021). Data movement also grows more expensive with distance, transforming distributed training into a careful balance between computation parallelism and communication efficiency.

Jouppi, Norm, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, et al. 2023. “TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings.” Proceedings of the 50th Annual International Symposium on Computer Architecture, 1–14. https://doi.org/10.1145/3579371.3589350.

Lie, Sean. 2021. Thinking Outside the Die: Architecting the ML Accelerator of the Future.

Data center scaling and edge deployment represent opposite ends of a deployment spectrum, yet they share the same core principles. Data center scaling coordinates many high-throughput accelerators, while edge scaling fits useful AI into a few constrained watts. Both cases require matching workload characteristics to hardware capabilities while minimizing data movement. The principles of compute specialization, memory hierarchy optimization, and workload mapping apply at both scales; only the constraints differ. Data centers optimize for aggregate throughput within power budgets measured in megawatts; edge devices optimize for responsiveness within tight battery and thermal envelopes. The same vision model that runs comfortably in a data center may need a radically different mapping strategy on a smartphone or microcontroller.

Self-Check: Question

Why have accelerator architectures increasingly shifted from monolithic single-die designs toward Multi-Chip Module (MCM) and chiplet architectures?
1. Chiplets eliminate all need for semiconductor fabrication foundries
2. Monolithic dies are constrained by the physical lithography reticle limit ($\approx 858\text{ mm}^2$) and suffer exponential yield loss as die size increases
3. Monolithic dies cannot support high-bandwidth memory (HBM) interfaces
4. Chiplets allow electrical signals to travel faster than the speed of light
Explain how the non-uniform memory access (NUMA) effect and interconnect bandwidth degradation impact performance when scaling a neural network workload across multiple chiplets or accelerator chips.
Describe the architectural rationale behind Cerebras’s Wafer-Scale Engine and explain how fabricating an accelerator across an entire uncut silicon wafer overcomes traditional multi-chip scaling bottlenecks.
What is the primary bottleneck introduced by inter-node scaling (scaling out across separate servers over Ethernet or InfiniBand) compared to intra-node NVLink scaling?
1. Inter-node network adapters cannot transmit FP16 or BF16 floating-point values
2. Inter-node communication requires GPUs to switch to single-threaded CPU emulation mode
3. Inter-node network bandwidth (e.g., $400\text{ Gbps} \approx 50\text{ GB/s}$) is roughly an order of magnitude lower than intra-node NVLink bandwidth ($\approx 900\text{ GB/s}$), increasing collective communication latency
4. Inter-node scaling eliminates the need for gradient synchronization in distributed training

See Answers →

Heterogeneous SoC Design

Edge, mobile, and automotive deployments operate under strict physical constraints: sealed enclosures without active fans limit thermal dissipation to a few watts, while battery capacity caps total energy consumption. To deliver machine learning capabilities within these boundaries, heterogeneous SoC architectures integrate general-purpose CPU cores, GPU shader arrays, digital signal processors (DSPs), and specialized neural processing units (NPUs) onto a single silicon die sharing a unified memory subsystem. The workload mapping problem introduced for discrete accelerators persists, but it expands to include hardware operator coverage, cross-engine transfer overheads, real-time deadlines, and dynamic thermal throttling.

Mobile SoC architecture evolution

Mobile system-on-chip architectures implement heterogeneous computing by coupling distinct execution engines—multicore CPUs, wide SIMD GPU shader cores, fixed-point DSPs, and dedicated neural processing units (NPUs)⁴⁴—over an on-chip network connected to shared low-power DRAM (LPDDR). Workload partitioning routes sensory and perceptual tasks to the compute engine best aligned with their arithmetic structure: continuous audio feature extraction and keyword spotting run on low-power DSPs, dense convolution and matrix-multiplication layers execute on systolic or tensor-array NPUs, parallel graphics and image postprocessing map to GPUs, and sequential control flow runs on the CPU. Meeting real-time frame deadlines without triggering aggressive thermal throttling requires scheduling execution across these engines while respecting their differing power profiles.

⁴⁴ NPU (neural processing unit): The NPU’s specialized matrix engines are optimized for dense tensor operations, providing the hardware basis for the workload distribution described. This specialization creates a critical constraint for the scheduler: any AI operator not mapped to the NPU’s supported data paths must “fall back” to a GPU or CPU. This fallback can erase the NPU’s energy-efficiency advantage, complicate real-time latency budgets, and contribute to thermal pressure on mobile devices.

Example 1.5: Heterogeneous microcontrollers

Scenario: An embedded smart doorbell must run periodic person detection within a tight memory, energy, and latency budget.

Diagnosis: On a supported deployment, a general-purpose Cortex-M path misses the measured energy or latency target. A micro-NPU such as Arm Ethos-U can offload compatible convolution operators, while the CPU retains control flow and unsupported operations.

Systems lesson: Micro-NPU specialization can improve the energy and latency of supported operators, but feasibility must be measured for the whole pipeline, including fallbacks and sensor processing.

Modern mobile SoCs utilize unified memory architectures where all processing units share a common physical address space in LPDDR. Unlike discrete accelerator architectures that incur explicit PCIe transfer latencies, unified memory allows zero-copy buffer passing between the image signal processor (ISP), CPU, GPU, and NPU. However, sharing physical DRAM does not guarantee zero software overhead. If an NPU engine lacks hardware cache snooping, passing an intermediate tensor from the CPU requires explicit cache flushes and invalidations. Furthermore, specialized NPUs often require hardware-specific tensor layouts—such as channels-last ($NHWC$) with 16- or 32-byte channel alignment to feed vector register lanes—forcing memory-bound transpositions when interfacing with CPU layers structured in channels-first ($NCHW$) formats.

SoC architects customize these compute blocks according to domain-specific physical constraints. High-end smartphone SoCs emphasize peak burst performance and low idle leakage for interactive tasks such as real-time computational photography and on-device text generation. In contrast, embedded and automotive SoCs prioritize deterministic memory access, hardware fault detection, and bounded worst-case latency to satisfy real-time control constraints.

Strategies for dynamic workload distribution

Partitioning neural network workloads across heterogeneous SoC cores requires matching the arithmetic intensity and memory regularity of each computational graph operator to the microarchitecture best equipped to execute it. Consider a real-time object detection pipeline executing on a mobile SoC with an integrated CPU, GPU, and NPU. The pipeline comprises three consecutive stages: a MobileNet backbone for feature extraction, nonmaximum suppression (NMS) for postprocessing candidate bounding boxes, and a display overlay to render final detections onto a camera feed. The convolutional backbone exhibits regular, deterministic memory access and dense arithmetic suitable for the NPU’s fixed-function tensor engine, even though depthwise separable layers exhibit lower arithmetic intensity than standard convolutions. NMS, by contrast, relies on score-based sorting and conditional intersection-over-union loops evaluated across variable-length candidate lists. This irregular, branch-heavy control flow maps poorly to an NPU’s spatial array but executes efficiently on out-of-order CPU cores with branch prediction and hardware caching. Finally, the display overlay requires per-pixel rasterization and alpha blending across a high-resolution frame—an embarrassingly parallel, memory-bandwidth-bound task that maps directly to the GPU’s rasterizers and texture units. Splitting the pipeline across the three engines—NPU for feature extraction, CPU for NMS, and GPU for display compositing—minimizes end-to-end latency and total energy consumption compared to executing the entire pipeline on any single processor.

This pipeline partitioning reflects the broader relationship between algorithmic structures and machine architectures. Regular matrix contractions and convolutions map cleanly to systolic or vectorized NPU datapaths, whereas data-dependent control flow maps to CPU execution resources. Transformer attention layers span both paradigms: the projection GEMMs ($Q, K, V, O$) and attention score contractions ($Q K^T$ and $A V$) execute efficiently on NPU matrix units, but intermediate softmax reductions, causal masking, and layer normalizations involve elementwise operations. If the NPU runtime cannot fuse these elementwise operations into on-chip SRAM scratchpads, intermediate tensors must spill to LPDDR, eroding the accelerator’s throughput advantage.

Optimal processor assignment is dynamic rather than static. As device thermal conditions deteriorate or battery reserves drop, the runtime must adapt its scheduling policy. However, migrating an executing model across processor backends carries non-trivial overhead: weights must be re-bound in memory, intermediate buffers may require layout reformatting, and device drivers incur kernel dispatch latency. If the overhead of synchronizing state across processors exceeds the compute savings of offloading, retaining execution on a single suboptimal engine can yield lower total latency and energy than continuous cross-engine migration.

Shared memory subsystems introduce cross-engine interference during simultaneous execution. Because the CPU, GPU, NPU, camera ISP, and display engine share a single LPDDR memory bus, the SoC memory controller enforces Quality of Service (QoS) priorities to satisfy real-time hardware deadlines. The display engine and camera ISP must receive guaranteed memory bandwidth and bounded latency to prevent display flicker and sensor FIFO overflow; the memory controller consequently prioritizes display and camera transactions over NPU and GPU traffic. During memory-intensive phases—such as autoregressive language model decoding where model weights must be read from DRAM for every generated token—high-priority ISP or display traffic reduces effective NPU memory bandwidth, increasing token generation latency.

Power and thermal management

Fanless mobile devices operate within strict continuous power budgets and thermal envelopes, typically dissipating no more than 3 to 5 watts sustained across the entire chassis before touch temperatures exceed safety thresholds. Although transient bursts can draw 10 to 15 watts, continuous inference rapidly saturates the thermal capacitance of the device, heating the silicon junction and requiring active power management across the SoC.

Heterogeneous SoCs manage power across distinct silicon partitions using dynamic voltage and frequency scaling (DVFS). Because dynamic power scales quadratically with supply voltage and linearly with operating frequency ($P_{\text{dyn}} \propto V^2 f$), dropping the clock frequency allows lowering the core voltage, yielding substantial power reductions. On a heterogeneous SoC, CPU clusters, GPU shader cores, and NPU engines occupy separate power and voltage domains. Increasing the voltage and frequency of the NPU to meet a tight latency deadline consumes electrical current and thermal headroom that would otherwise be available to the CPU or GPU. System power governors continuously adjust these operating points based on workload queues, temperature sensor readings, and frame deadline targets.

When sustained computation exceeds the chassis dissipation rate, the platform initiates thermal throttling. Because specialized NPUs achieve higher energy efficiency (TOPS/W) for tensor math than general-purpose cores, offloading matrix workloads to a CPU or GPU during a thermal event often increases total energy dissipation, exacerbating thermal accumulation. Instead, runtimes manage thermal limits by capping clock frequencies, reducing inference duty cycles (such as throttling a perception model from 30 frames per second to 15 frames per second), or dynamically switching to a smaller, pruned model checkpoint. Task migration is applied selectively to move non-tensor preprocessing away from localized silicon hot spots on the die.

Battery state further shapes runtime scheduling. When battery charge drops below critical thresholds or the device enters a low-power mode, the system scheduler trades inferencing fidelity for battery life by quantizing operations to lower bit widths (such as transitioning from 16-bit floating-point to 8-bit integer precision), increasing execution sparsity, or batching asynchronous inference requests to maximize the time cores spend in deep low-power sleep states.

Automotive heterogeneous AI systems

Automotive computing platforms operate under a dual mandate: delivering hundreds of tera-operations per second (TOPS) for multi-camera perception and sensor fusion while satisfying strict functional safety standards (such as ISO 26262). Unlike mobile systems where thermal throttling gracefully degrades user experience, automotive systems operate under hard real-time deadlines where missing a frame deadline can compromise vehicle control. However, not all in-vehicle AI workloads require the highest safety integrity; infotainment and driver-comfort models share compute platforms with safety-critical perception pipelines.

To support this mixed-criticality execution, automotive SoCs enforce freedom from interference between safety-critical workloads (such as automated emergency braking) and non-critical applications (such as natural-language cabin assistants). Hardware-enforced virtualization, input-output memory management units (IOMMUs), and memory bus bandwidth reservations isolate compute cores and LPDDR channels. Without memory bandwidth reservations, an unverified convenience model streaming weights from DRAM could saturate memory channels, delaying frame delivery to an emergency perception kernel with a certified worst-case execution time (WCET). Real-time operating systems schedule critical neural network pipelines using time-triggered dispatching rather than average-throughput priority queues.

Automotive architectures also address end-to-end latency across distributed and centralized sensor topographies. Autonomous platforms ingest high-bandwidth data streams simultaneously from high-resolution cameras, automotive radar, and lidar. Ingesting, packetizing, and transporting this data across automotive Ethernet (such as 10GBASE-T1) or serial deserializer links (SerDes) introduces transport latency and jitter. Sensor fusion algorithms require hardware-timestamped data streams synchronized to a shared microsecond clock (via IEEE 802.1AS). Accelerators cannot begin cross-sensor fusion until the slowest sensor frame in a temporal window arrives, meaning sensor alignment, serialization overheads, and DMA ingest latencies must be budgeted alongside neural network inference time.

External vehicle-to-everything (V2X) inputs introduce non-deterministic communication latencies and variable packet delivery rates. Because wireless signals are subject to packet drops, multi-path fading, and adversarial spoofing, safety-critical perception algorithms cannot treat V2X inputs as synchronous sensor feeds. Instead, planning pipelines incorporate V2X payloads as asynchronous contextual hints, bounding their influence to ensure vehicle safety remains guaranteed by onboard deterministic sensors.

Software stack challenges

Heterogeneity shifts the primary systems challenge from silicon design to software coordination. Standard programming models like OpenCL and Vulkan provide portable low-level execution abstractions across compute devices, but they do not eliminate the need for microarchitecture-specific kernel tuning. Machine learning inference runtimes partition computation graphs through hardware-specific delegates or backends targeting CPUs, GPUs, DSPs, and NPUs. When a model contains an operator unsupported by an NPU delegate—such as a custom activation, dynamic tensor shape, or non-standard reduction—the runtime splits the graph into subgraphs, inserting fallback paths to the CPU or GPU. Each boundary crossing requires synchronizing execution and translating data representations, transforming an algorithmic graph into a fragmented sequence of hardware dispatches.

Shared memory subsystems require explicit cache and memory management across non-coherent execution domains. While CPUs maintain cache coherence across their own cores via hardware snooping protocols, mobile NPUs and DSPs frequently interface with system memory as non-coherent DMA bus masters. When the CPU writes input tensors to cached DRAM and hands control to an NPU, software must explicitly flush the CPU’s dirty cache lines to DRAM before NPU execution begins. Similarly, upon NPU completion, the CPU must invalidate its own cache lines over the destination memory range to prevent reading stale cached data. If unsupported operators cause an inference graph to ping-pong repeatedly between CPU and NPU, the latency of cache maintenance and intermediate DRAM round-trips can exceed the execution time of the operators themselves.

Runtimes navigate this scheduling complexity using ahead-of-time compilation, empirical profile databases, and dynamic dispatch heuristics. Device telemetry—monitoring frame completion times, core temperatures, and battery drain—informs runtime decisions to throttle clock frequencies or swap execution delegates. However, dynamic adaptation cannot replace deterministic graph lowering for latency-sensitive applications; deployment requires validating the compiled, lowered partition on physical target silicon under worst-case thermal and memory contention.

Heterogeneous acceleration demonstrates that system efficiency is fundamentally a coordination problem rather than an inventory of specialized execution blocks. In the D·A·M taxonomy, an SoC represents a composite Machine layer whose components exhibit divergent memory hierarchies, arithmetic strengths, and thermal envelopes. The iron law of ML systems dictates that overall runtime and energy consumption are governed by data movement: when software coordination fails to maintain tensor locality or induces frequent cross-engine fallbacks, energy dissipation is dominated by interconnect and DRAM traffic rather than arithmetic. Maximizing performance on heterogeneous SoCs requires minimizing data movement across domain boundaries while matching each algorithmic operation to the specialized datapath architected to accelerate it.

Self-Check: Question

On a modern mobile heterogeneous System-on-Chip (SoC) featuring big.LITTLE CPUs, a mobile GPU, and a dedicated NPU, which compute engine is most energy-efficient for running continuous, low-latency 8-bit quantized convolutional inference?
1. The high-performance ‘big’ CPU core running single-threaded FP32 instructions
2. The out-of-order system memory controller
3. The dedicated Neural Processing Unit (NPU) optimized for quantized INT8 matrix operations
4. The host operating system virtualization hypervisor
In automotive autonomous driving SoCs, explain why deterministic worst-case execution time and lockstep redundancy are required, even if they reduce peak average-case throughput.
True or False: In mobile SoCs with unified system memory (LPDDR), sharing physical RAM between the CPU, GPU, and NPU eliminates all data movement overhead between heterogeneous processors.
Describe how Dynamic Voltage and Frequency Scaling (DVFS) and thermal throttling constrain sustained AI inference performance on edge and mobile devices.

See Answers →

Hardware Sustainability

At fleet scale, energy per useful inference becomes an important hardware-selection criterion. Operational emissions depend on workload throughput, measured system power, utilization, cooling overhead, and the electricity mix; embodied emissions add another lifecycle cost. Peak performance per watt is therefore a screening metric, not a substitute for end-to-end energy per request. The following scenario shows how assumptions compound when comparing a generic-CPU fleet with specialized accelerators.

Napkin Math 1.10: An illustrative operational-energy comparison

Problem: Under a stated throughput-equivalence assumption, how does operational energy change when a CPU fleet is replaced by specialized NPUs (Neural Processing Units)?

Assumptions: This stylized comparison treats the listed peak rates as useful workload throughput, assumes both fleets run continuously at the listed power, and excludes host power, cooling, embodied carbon, and utilization differences. It is a sensitivity calculation, not a measured procurement result.

CPU inference: 100 W for 1 TFLOP/s (efficiency = 0.01 TFLOP/s/W).
NPU inference: 5 W for 10 TFLOP/s (efficiency = 2 TFLOP/s/W).
The assumed peak-efficiency gap: 200×.

Math:

Workload: 1 billion inferences per day.
CPU fleet energy: 1,000 CPU servers $\times$ 100 W $\times$ 24 h $\approx$ 2,400 kWh/day.
NPU fleet energy: 100 NPU chips $\times$ 5 W $\times$ 24 h $\approx$ 12 kWh/day.
Carbon savings: At 0.429 kg/kWh, switching to NPUs saves ~373.9 t of CO₂ per year.

Systems insight: Specialized accelerators can reduce operational energy when the workload uses their efficient path and fewer devices deliver the required service throughput. Replace these assumptions with measured requests per second and wall power before using the result for procurement or carbon accounting.

The sustainability perspective reinforces a central theme: hardware selection is not determined by peak speed alone. Energy per useful result, carbon intensity, embodied impact, and total cost of ownership belong beside latency, throughput, and memory capacity. The remaining step is to identify the misconceptions that cause teams to choose the wrong hardware path.

Self-Check: Question

In the lifecycle carbon assessment of advanced deep learning accelerators, what constitutes ‘embodied carbon’?
1. The electrical energy consumed by the GPU during model forward and backward passes
2. The carbon emitted by datacenter air conditioning units during peak summer load
3. The carbon credits purchased by cloud providers to offset datacenter energy usage
4. The greenhouse gas emissions generated during raw material extraction, semiconductor silicon manufacturing, packaging, and hardware transportation
Explain why maximizing accelerator utilization (e.g., via multi-tenant sharing or continuous pipeline saturation) reduces the amortized carbon footprint per trained model.
True or False: In a datacenter with a Power Usage Effectiveness (PUE) of 1.1, the cooling and electrical distribution infrastructure consumes more power than the actual computing IT equipment (servers and accelerators).

See Answers →

Fallacies and Pitfalls

Hardware acceleration exhibits counterintuitive performance characteristics because headline specifications frequently mask workload-specific bottlenecks. The fallacies and pitfalls here capture selection, scaling, and optimization errors that leave expensive compute resources underutilized.

Fallacy: More specialized hardware always provides better performance than general-purpose alternatives.

Specialized accelerators achieve high efficiency only when workload access patterns match underlying execution datapaths: the core tenet of algorithm-machine co-design. Under the Roofline Model (section 1.5), an operation must exceed the hardware ridge point to become compute bound. An A100 GPU has a ridge point of 153 FLOP/byte, meaning operations with arithmetic intensity below this threshold remain memory bound regardless of the chip’s 312 TFLOP/s peak compute. A transformer attention softmax with AI = 2 FLOP/byte–5 FLOP/byte achieves only 4.1 TFLOP/s–10.2 TFLOP/s (3.3 percent utilization) on an A100. CPUs with ridge points around 10 FLOP/byte–20 FLOP/byte also execute this kernel in the memory-bound regime, but that same arithmetic intensity saturates 10 percent–50 percent of the CPU’s compute capacity. When workloads exhibit irregular memory access, tiny batch sizes, or dynamic control flow, massively parallel execution pipelines stall on memory latency and thread divergence. General-purpose CPUs with large cache hierarchies, out-of-order execution, and branch predictors often deliver superior latency and resource efficiency under these conditions. Hardware selection requires matching workload arithmetic intensity and dataflow regularity to architectural ridge points rather than assuming specialization guarantees speed.

Pitfall: Ignoring memory bandwidth limitations when selecting acceleration strategies.

Procurement decisions frequently target peak arithmetic throughput while overlooking memory interface constraints. Off-chip data movement imposes severe performance and energy penalties: fetching an operand from DRAM consumes roughly 640 pJ compared to 0.5 pJ for an on-chip L1 SRAM access (section 1.4.1). Consider an accelerator providing 300 TFLOP/s of compute alongside 2 TB/s of memory bandwidth, setting a ridge point of 150 FLOP/byte. A normalization kernel such as LayerNorm with an arithmetic intensity of 1.5 FLOP/byte achieves only 3 TFLOP/s (1 percent compute utilization). Because LayerNorm streams full activation tensors through memory to compute simple reduction statistics, memory bandwidth strictly caps throughput. Adding compute units without expanding memory bandwidth leaves runtime unchanged and wastes silicon area.

Fallacy: Hardware acceleration benefits scale linearly with additional accelerators.

Distributed training does not deliver $8\times$ throughput on eight accelerators because inter-chip communication overhead degrades parallel efficiency. Synchronizing gradients across ranks via ring AllReduce (section 1.10) requires each rank to transfer $2(N-1)/N$ times the model gradient volume: 1.75× payloads across eight devices. With an interconnect such as NVLink providing 600 GB/s bidirectional bandwidth (half that per direction), transmitting a 1 GB gradient payload consumes 5.83 ms. For a model with a 50 ms compute step, this data exchange adds 11.7 percent communication overhead before counting software collective latency. Without compute-communication pipelining, an eight-accelerator node reaches only 7.2× speedup (89.6 percent parallel efficiency). Stragglers, barrier synchronizations, and collective protocol overhead widen the gap between linear expectations and physical cluster scaling.

Pitfall: Planning accelerator capacity from peak FLOP/s specifications.

Peak FLOP/s measures arithmetic unit capacity under ideal instruction mixes and continuous operand availability, not delivered application throughput. Realized performance depends on kernel arithmetic intensity, matrix dimensions, memory coalescing, and runtime launch overhead. The Roofline Model (section 1.5) bounds the gap between peak compute and bandwidth-limited execution. On an A100 GPU with a 312 TFLOP/s peak, dense transformer training sustains 120 TFLOP/s–180 TFLOP/s (40 percent–60 percent model FLOPs utilization). Conversely, a sparse recommender model sustains only 10 TFLOP/s–30 TFLOP/s (3.2 percent–9.6 percent utilization) because sparse embedding table lookups issue uncoalesced memory reads that starve tensor execution units. Sizing cluster capacity from peak FLOP/s creates severe under-provisioning; capacity planning must rely on measured sustained throughput or roofline-bounded kernel projections.

Fallacy: Any FLOP/s rating can estimate a low-precision workload.

Accelerators have separate datapaths for different precisions, and the peak throughput varies dramatically across them. An H100 delivers roughly 989 TFLOP/s in FP16 tensor operations but only about 67 TFLOP/s in FP32 CUDA-core operations: a roughly 15× gap within the same chip (Choquette 2023). Estimating training time with the FP32 number when the workload actually uses BF16 produces utilization figures that appear severely degraded without physical cause, and matching against the wrong roofline misclassifies kernels as compute-bound when they are memory-bound (or vice versa). Always match the peak constant to the precision the workload actually issues, and quote precision explicitly when reporting model FLOPs utilization.

Pitfall: Deploying small-batch inference workloads on high-compute accelerators.

Small-batch inference severely underutilizes high-compute accelerators because single-sample execution lacks parameter reuse. For a dense linear projection with dimension $M=N=2048$, a single request ($B=1$) requires streaming the entire weight tensor from device memory to execute a matrix-vector product, yielding an arithmetic intensity of only 1 FLOP/byte. At batch size $B=256$, weight reuse amortizes memory transfers across tokens, elevating arithmetic intensity to 204.8 FLOP/byte. At $B=1$, memory bandwidth clamps delivered performance to 2.04 TFLOP/s on an A100 and 0.3 TFLOP/s on a T4. Although the T4 provides a lower peak of 65 TFLOP/s with a ridge point of 203.1 FLOP/byte, its delivered throughput at $B=1$ is within a factor of two of the A100 at a fraction of the acquisition cost and thermal footprint. Deploying an A100 for unbatched real-time inference leaves expensive compute units idle; system architects must align accelerator selection with operational batch sizes and latency objectives.

Fallacy: Vendor-specific optimizations have no long-term portability cost.

Optimizing exclusively for a single vendor runtime captures immediate kernel efficiency but introduces severe architectural lock-in (section 1.8). Codebases saturated with proprietary device intrinsics, vendor-specific assembly, and non-standard memory layouts require months of engineering labor to port when newer, more cost-effective accelerators emerge. This technical debt delays hardware transitions and prevents multi-vendor procurement strategies. To preserve flexibility, systems must isolate vendor-tuned micro-kernels behind hardware abstraction layers or target intermediate compiler representations (such as Triton or MLIR). Preserving portable execution paths allows infrastructure to track hardware innovation without sacrificing the optimization benefits of specialized execution units.

Checkpoint 1.4: Feasibility assessment: Can you run it?

Before procuring hardware, validate each physical constraint. These failure modes translate into a concrete feasibility test:

Memory capacity: compute $M_{\text{req}} = \text{Weights} + \text{KV Cache} + \text{Activation Buffer}$ and verify $M_{\text{req}} < M_{\text{device}}$ for a 7-billion-parameter Llama model with 14 GB FP16 weights on a 16 GB GPU.
Bandwidth: compute $T_{\text{token}} = D_{\text{vol}} / \text{BW}$ for a 70-billion-parameter model (140 GB) on 1 TB/s memory bandwidth, then compare the result with a 50 ms latency target.
Compute: compute $T_{\text{process}} = O / (R_{\text{peak}} \cdot \eta_{\text{hw}})$ and compare it with the throughput target. Processing video at 30 FPS requires completing inference within 33.3 ms.
Roofline placement: for an accelerator at roughly 2 TFLOP/s peak and 200 GB/s bandwidth, estimate the ridge-point arithmetic intensity, then place a 10 FLOP/byte kernel against it: is it compute- or memory-bound, and would a faster clock help it at all?

This checklist translates architectural principles into rigorous engineering practice. System evaluation demands a disciplined diagnostic habit: characterize the workload arithmetic intensity, identify the physical bottleneck metric, match execution to the appropriate datapath, and account for interconnect, scaling, and portability costs.

Self-Check: Question

A team prunes $70\%$ of the weights in a large language model using unstructured magnitude pruning, setting those weights to zero. However, when executing the pruned model on standard GPU dense Tensor Cores, inference latency is identical to or slower than the unpruned baseline. What is the primary cause of this pitfall?
1. Standard dense hardware cannot skip individual zero elements without structured patterns (e.g., 2:4) or specialized sparse matrix indexing, so dense matrix units still execute all multiplications while sparse formats add indexing overhead
2. Floating-point units automatically convert zero values into infinite loops
3. Unstructured pruning forces the GPU driver to downclock memory bandwidth to prevent overheating
4. The operating system kernel intercepts every zero multiplication and raises a hardware page fault
Describe the pitfall of ‘micro-offloading’ small tensor operations from CPU to GPU, and explain why a sequence of scalar operations can run slower on an accelerator than on the host CPU.
Place the following diagnostic steps in the recommended sequence when troubleshooting an underperforming neural network training workload on an accelerator cluster:

Roofline & Hardware Counter Analysis: Determine whether individual kernels are compute-bound, memory-bandwidth-bound, or latency-bound
Amdahl & Host-Side Profiling: Identify serial bottlenecks, data loading stalls, and CPU-GPU synchronization delays
Kernel-Level Optimization: Apply operator fusion, hierarchical tiling, or precision reduction targeted to the identified bottleneck
Multi-Device Communication Profiling: Check for gradient all-reduce synchronization delays and interconnect saturation

Which of the following statements represents a classic fallacy regarding peak accelerator specifications?
1. High-bandwidth memory reduces the latency of memory-bound operations compared to standard DDR5
2. An accelerator with $2\times$ higher peak theoretical TFLOP/s will automatically deliver a $2\times$ speedup on any neural network workload
3. Kernel fusion can improve arithmetic intensity by reducing global memory traffic
4. Warp divergence reduces the execution efficiency of SIMT processor lanes

See Answers →

Summary

Hardware acceleration co-designs compute engines, memory hierarchies, numerical formats, and dataflows. For a chosen memory interface, the iron law compares compute time $O/(R_{\text{peak}}\eta_{\text{hw}})$ with movement time $D_{\text{vol}}/\text{BW}$; the roofline restates that competition as $\min(R_{\text{peak}}, \text{BW}I)$. End-to-end latency also includes launch, synchronization, communication, and fixed overheads. Accelerators improve efficiency only when supported paths capture reuse locally. Hardware selection therefore begins with the workload’s measured or modeled bottleneck, not a vendor peak.

Key Takeaways: Moving data costs more than computing it

The Roofline Model identifies bottlenecks: Compare a kernel’s measured intensity with the target’s precision-specific ridge point (153 FLOP/byte here). Below it, reduce memory traffic; above it, improve compute utilization.
Memory bandwidth constrains performance: GPU compute capacity has grown faster than memory bandwidth. Many inference phases are memory bound, especially low-batch decoding and embedding lookup, while large-batch GEMMs can remain compute bound.
Hardware-software co-design compounds performance: Matching algorithm patterns to architectural capabilities, such as dense GEMM on systolic or tensor units and supported sparsity on sparse paths, can produce large improvements.
Tensor Cores require a compatible path: Precision, dimensions, layouts, and library support must match the architecture. Batch and shape affect reuse and utilization, but no single batch threshold guarantees peak performance.
End-to-end speedup stops at the next bottleneck: Accelerating one kernel leaves launch, transfer, synchronization, communication, and unsupported work unchanged. Use Amdahl’s law and profiling to decide whether further silicon, better mapping, or less movement will improve the complete workload.

An accelerator is not a uniformly faster computer; it provides efficient paths for particular operations and dataflows. Tiling, fusion, hierarchy-aware scheduling, and systolic execution reduce modeled-interface traffic only when the workload, numerical format, and software path can use them.

The Roofline Model provides a first diagnostic by using arithmetic intensity and the ridge point to narrow the likely bottleneck; profiling then accounts for latency, occupancy, communication, and implementation overhead. The result is a testable hypothesis, not a prediction.

What’s Next: From optimization to validation

The ML optimization stack rests on three complementary efficiency strategies: data selection (Data Selection) minimizes training sample requirements, model compression (Model Compression) reduces algorithmic parameter and precision complexity, and hardware acceleration maximizes physical machine throughput. Optimization without empirical measurement, however, remains incomplete. Benchmarking moves from theoretical FLOP bounds to measured wall-clock latency, applying standardized benchmark suites and statistical protocols to test optimization claims against operational reality.

Self-Check: Question

What is the central architectural insight of the hardware acceleration chapter regarding the interaction between model architecture and accelerator efficiency?
1. Accelerators will soon eliminate all memory hierarchies in favor of infinite register files
2. Hardware efficiency depends entirely on maximizing clock frequency regardless of memory bandwidth
3. High delivered hardware efficiency requires co-design across computational primitives, dataflow reuse strategies, memory hierarchies, and compiler-runtime systems
4. General-purpose out-of-order CPUs remain superior to specialized TPUs for all deep learning workloads
Summarize how the Roofline model serves as a unified diagnostic bridge connecting high-level neural network operations to low-level hardware architecture choices.
Which combination correctly summarizes the primary function of each layer in the modern AI hardware acceleration software stack?
1. Framework: Silicon manufacturing; Compiler: Host PCIe routing; Runtime: Floating-point unit logic
2. Framework: Direct transistor clocking; Compiler: Operating system page fault handling; Runtime: Mathematical differentiation
3. Compiler: Real-time sensor power regulation; Runtime: Neural network gradient backpropagation; Hardware: Python interpreter dispatch
4. Framework: Graph definition and automatic differentiation; Compiler: Graph optimization, operator fusion, and tiling; Runtime: Memory pooling, stream scheduling, and kernel dispatch; Hardware: Parallel matrix/vector execution

See Answers →

Self-Check Answers

Self-Check: Answer

What primary physical limitation brought about the end of Dennard scaling in the mid-2000s, necessitating the transition from increasing CPU clock frequencies to domain-specific hardware accelerators?
1. Lithography light diffraction preventing further reduction of transistor gate length below $1\,\mu\text{m}$
2. Inability to lower operating voltage proportionally with transistor size, leading to unsustainable power density and heat dissipation limits
3. Quantum tunneling in copper interconnect lines preventing data transmission between arithmetic units
4. Depletion of global silicon substrate supplies requiring migration to gallium nitride semiconductors
Answer: The correct answer is B. Dennard scaling postulated that as transistors shrank, operating voltage could drop in proportion, keeping chip power density constant while operating frequencies increased. When threshold voltage and leakage currents prevented further voltage scaling around 2005, increasing clock frequency resulted in prohibitive power density and thermal dissipation limits (the power wall and ‘dark silicon’), forcing architects to seek energy efficiency through architectural specialization and parallelism. The lithography choice misidentifies lithography limits as the cause of the mid-2000s frequency stall. The quantum tunneling choice confuses gate oxide leakage with bulk copper transmission. The semiconductor depletion choice is physically fabricated.

Learning Objective: Explain the physical breakdown of Dennard scaling that led to domain-specific acceleration.
Contrast the architectural trade-offs of software-managed scratchpad memory (such as Google TPUv1’s Unified Buffer) with hardware-managed cache hierarchies when executing large-scale tensor workloads.

Answer: Software-managed scratchpads eliminate the hardware overhead of tag arrays, cache line state machines, coherence snooping, and dynamic replacement policies, allowing more silicon area and power to be dedicated to storage capacity and dense arithmetic units. When compilers can statically determine regular tensor access patterns, they can deterministically tile and stage data. However, scratchpads increase compiler complexity and software burden, and they perform poorly on irregular, dynamically indexed memory patterns that hardware caches manage automatically.

Learning Objective: Compare software-managed scratchpad memory with hardware-managed caches for deep learning workloads.
**Place the following computing milestones in chronological order (from earliest to most recent) as hardware evolved toward modern AI accelerators:

Introduction of dedicated Tensor Cores and TPUs for deep learning matrix operations
Emergence of fixed-function media codecs and network processors for video/packet streaming
Integration of floating-point units (FPUs) and digital signal processors (DSPs) as discrete coprocessors
General-purpose programmable GPUs and SIMD instruction set extensions for 3D graphics and multimedia**

Answer: The correct order is (3) Integration of floating-point units (FPUs) and digital signal processors (DSPs) as discrete coprocessors -> (4) General-purpose programmable GPUs and SIMD instruction set extensions for 3D graphics and multimedia -> (2) Emergence of fixed-function media codecs and network processors for video/packet streaming -> (1) Introduction of dedicated Tensor Cores and TPUs for deep learning matrix operations. During the 1980s, discrete FPUs and DSPs accelerated arithmetic. The 1990s introduced SIMD extensions and early 3D GPUs. The 2000s expanded fixed-function media engines and network processors. The 2010s ushered in deep learning DSAs like Google TPUs (2015/2017) and NVIDIA Volta Tensor Cores (2017).

Learning Objective: Explain the historical evolution of hardware specialization from FPUs to modern AI accelerators.

True or False: Google’s TPUv1 achieved substantial performance-per-watt improvements over contemporary general-purpose CPUs primarily by operating at significantly higher clock frequencies.

Answer: False. TPUv1 operated at a modest clock frequency (approximately $700\text{ MHz}$) and achieved $30\times\text{--}80\times$ higher performance-per-watt than contemporary CPUs by omitting complex general-purpose features (such as out-of-order execution, branch prediction, and multi-level cache coherence) and dedicating silicon area to a massive $256{\times}256$ INT8 systolic matrix multiplier and a large software-managed Unified Buffer.

Learning Objective: Evaluate the architectural source of energy efficiency in domain-specific accelerators like TPUv1.
In the context of hardware scaling, what phenomenon does the ‘Systems Gap’ describe since the 2012 deep learning breakthrough?
1. The difference in memory bandwidth between high-end datacenter GPUs and consumer-grade mobile SoCs
2. The latency discrepancy between on-chip SRAM access times and host DRAM access times over PCIe
3. The exponential divergence between model compute demand (growing $\approx 6\times$/year) and single-device hardware supply (growing $\approx 1.7\times$/year)
4. The mismatch between Python framework dispatch overhead and raw GPU kernel execution duration
Answer: The correct answer is C. The Systems Gap refers to the widening divergence between the computational demand of frontier AI models (growing at roughly $6\times$ per year according to scaling law trends) and single-accelerator hardware performance growth (growing at roughly $1.7\times$ per year under Huang’s Law / architectural scaling). Closing this massive exponential gap requires distributed parallelism, system co-design, and algorithmic optimizations. The mobile vs datacenter memory bandwidth choice describes edge heterogeneity rather than macro compute trends. The SRAM vs DRAM latency choice describes the physical memory hierarchy gap. The Python dispatch vs kernel duration choice describes host runtime overhead.

Learning Objective: Analyze the Systems Gap and its implications for AI systems co-design.

← Back to Questions

Self-Check: Answer

When executing a Transformer layer containing attention projection ($Q = X W_q$), Softmax ($\text{Softmax}(S)$), Layer Normalization ($\text{LayerNorm}(X)$), and GeLU activation ($\text{GeLU}(Z)$), which execution unit is specifically responsible for computing transcendental functions (exponential and error function approximations)?
1. Systolic 2D Matrix Multiply Units
2. Dense Tensor Cores
3. Vector Load-Store Memory Controllers
4. Special Function Units (SFUs)
Answer: The correct answer is D. Special Function Units (SFUs) are specialized hardware pipelines designed to evaluate transcendental and nonlinear mathematical functions—such as exponentials in Softmax, reciprocal square roots in LayerNorm, and Gaussian error approximations in GeLU—using hardware lookup tables and polynomial approximations. Matrix units and Tensor Cores are dedicated to high-throughput matrix multiply-accumulate operations, not transcendental approximations. Vector Load-Store Memory Controllers handle memory address generation and data movement across caches and DRAM.

Learning Objective: Classify deep learning mathematical operations to their corresponding hardware execution units.
Explain how the $\text{im2col}$ (image-to-column) transformation allows standard 2D convolution operations to execute on high-throughput matrix multiplication hardware (GEMM engines), and describe the primary memory overhead associated with this approach.

Answer: The $\text{im2col}$ transformation extracts overlapping local receptive field patches from an input tensor and flattens each patch into a column (or row) of a 2D matrix, while flattening the convolution filters into a weight matrix. This converts the sliding-window convolution into a standard GEMM ($C = A \cdot B$) that executes efficiently on Tensor Cores or systolic arrays. However, because receptive fields overlap, input pixel values are duplicated across multiple patches, expanding memory footprint by up to roughly the filter area ($K_h \times K_w$) unless implemented via implicit-GEMM address calculation without physical materialization.

Learning Objective: Explain the mechanism and memory trade-offs of the im2col transformation for accelerating convolutions.
True or False: In modern AI accelerators, element-wise vector operations such as residual additions ($Y = X_1 + X_2$) achieve higher arithmetic intensity than large matrix-matrix multiplications ($C = A \cdot B$).

Answer: False. Element-wise vector operations perform only one arithmetic operation per two operand reads and one write (arithmetic intensity $\approx 1/12 \text{ FLOP/byte}$ in FP32), making them heavily memory bandwidth bound. In contrast, an $N \times N$ matrix-matrix multiplication performs $\mathcal{O}(N^3)$ operations on $\mathcal{O}(N^2)$ data elements, allowing extensive data reuse in on-chip registers and shared memory to achieve high arithmetic intensity ($\mathcal{O}(N) \text{ FLOP/byte}$).

Learning Objective: Compare the arithmetic intensity of element-wise vector operations with matrix multiplications.
The hardware transformation technique that enables convolutional layers to execute as matrix multiplications without physically duplicating overlapping patch data in memory is known as ____ GEMM.

Answer: implicit. Implicit GEMM computes input tensor memory addresses on-the-fly inside kernel index calculations during tile loading, avoiding the large $\mathcal{O}(K_h \cdot K_w)$ memory expansion of explicit im2col materialization while retaining matrix-unit execution efficiency.

Learning Objective: Explain how implicit GEMM eliminates memory duplication overhead in convolution acceleration.
Which of the following operations in a modern deep learning architecture exhibits the highest operational arithmetic reuse, making it most suitable for dense 2D systolic arrays and Tensor Cores?
1. Batched linear layer matrix multiplication ($Y = X W$)
2. Element-wise ReLU activation ($\max(0, x)$)
3. Channel-wise Batch Normalization mean computation
4. Token-wise embedding table lookup
Answer: The correct answer is A. Batched linear layers multiply an activation matrix by a weight matrix, enabling each weight element to be reused across all batch elements and each activation element to be reused across multiple output features, yielding high arithmetic intensity suitable for 2D matrix units. Element-wise ReLU performs only one comparison per element load. Batch Normalization mean computation performs reduction passes over data with minimal arithmetic reuse per byte transferred. Embedding table lookups are gather operations with zero arithmetic operations per byte fetched.

Learning Objective: Classify neural network layers based on operational reuse and hardware execution affinity.

← Back to Questions

Self-Check: Answer

In NVIDIA’s Ampere and Hopper architectures, how does 2:4 structured sparsity achieve an up to $2\times$ theoretical speedup in Tensor Core matrix multiplication?
1. Exactly two non-zero values are preserved in every contiguous four-element block, allowing weights to be stored in half the memory with 2-bit index metadata while sparse Tensor Cores perform math only on non-zeros
2. Every alternate row of the weight matrix is dropped completely, allowing the GPU to halve the grid launch dimensions
3. Four separate threads simultaneously execute one scalar multiply-accumulate instruction in a single clock cycle
4. Floating-point numbers are converted to 2-bit integers, quadrupling register file capacity
Answer: The correct answer is A. The 2:4 structured sparsity pattern mandates that exactly two out of every four contiguous values in a weight tensor are non-zero. The compressed weight matrix stores only the two non-zero values per 4-element block along with 2-bit index metadata (4 bits total per 4 values), halving memory storage and bandwidth requirements, while Sparse Tensor Cores use the metadata to select matching activations and compute matrix multiply-accumulate operations at twice the throughput of dense Tensor Cores. The row-dropping choice describes coarse block pruning rather than fine-grained structured sparsity. The four-thread SIMT scalar choice confuses thread scheduling with tensor hardware sparsity mechanics. The 2-bit integer conversion choice describes extreme quantization rather than sparsity.

Learning Objective: Explain the mechanics and hardware benefits of 2:4 structured sparsity in Tensor Cores.
Contrast the data movement mechanics and primary use cases of Weight-Stationary (WS) and Output-Stationary (OS) systolic array dataflows.

Answer: In a Weight-Stationary dataflow, model weights are loaded into local PE registers and held stationary while input activations and partial sums stream through the array, maximizing weight reuse and making it optimal for CNNs where small filter weights are reused extensively across spatial dimensions. In an Output-Stationary dataflow, partial sums remain stationary in PE accumulators while weights and inputs stream through, eliminating intermediate memory traffic for partial sum write-backs and making it optimal for large-batch matrix multiplications with high accumulation depth.

Learning Objective: Compare Weight-Stationary and Output-Stationary systolic array dataflow strategies.
**Place the following steps in the correct execution sequence for processing a matrix multiplication on an accelerator with Sparse Tensor Cores using 2:4 structured sparsity:

Fine-tune or prune the weight matrix to ensure exactly two non-zero values exist in every four-element contiguous group
Sparse Tensor Cores load compressed weights and decode metadata to gather matching input activation elements
Multiply non-zero weights by gathered activations and accumulate into output partial sums at $2\times$ dense throughput
Compress the sparse weight matrix by storing only the non-zero values alongside 2-bit per-value selection metadata**

Answer: The correct order is (1) Fine-tune or prune the weight matrix to ensure exactly two non-zero values exist in every four-element contiguous group -> (4) Compress the sparse weight matrix by storing only the non-zero values alongside 2-bit per-value selection metadata -> (2) Sparse Tensor Cores load compressed weights and decode metadata to gather matching input activation elements -> (3) Multiply non-zero weights by gathered activations and accumulate into output partial sums at $2\times$ dense throughput. Structured sparsity begins with pruning/fine-tuning to satisfy the 2:4 constraint, followed by offline compression and metadata creation. At runtime, hardware loads compressed data, uses metadata to multiplex input activations, and computes the sparse GEMM.

Learning Objective: Apply the end-to-end execution workflow of 2:4 structured sparse matrix multiplication.

What is the primary difference between the FP8 E4M3 and FP8 E5M2 numerical formats used in modern AI accelerators (such as NVIDIA Hopper and Ada Lovelace)?
1. E4M3 uses 4 sign bits and 3 exponent bits, whereas E5M2 uses 5 sign bits and 2 exponent bits
2. E4M3 has 4 exponent bits and 3 mantissa bits providing higher precision for forward-pass activations/weights, whereas E5M2 has 5 exponent bits and 2 mantissa bits providing wider dynamic range for gradients
3. E4M3 is exclusively an integer fixed-point format, whereas E5M2 is a standard IEEE floating-point format
4. E4M3 requires twice as many memory bytes per element as E5M2
Answer: The correct answer is B. Both formats occupy 8 bits (1 sign bit + exponent + mantissa). FP8 E4M3 allocates 4 exponent bits and 3 mantissa bits, providing higher precision (lower rounding error) suitable for forward-pass weights and activations where values are well-scaled. FP8 E5M2 allocates 5 exponent bits (matching FP16 dynamic range) and 2 mantissa bits, offering a wider dynamic range essential for preventing underflow in backward-pass gradients. The sign bit claim is incorrect as floating-point formats use a single sign bit. The integer format claim is incorrect as both are floating-point representations. The byte size claim is incorrect because both are 8-bit (1-byte) formats.

Learning Objective: Compare the numerical properties and intended use cases of FP8 E4M3 and E5M2 formats.
True or False: In NVIDIA’s SIMT (Single Instruction, Multiple Threads) execution model, when threads within the same 32-thread warp execute divergent branches of an if-else condition, both paths are executed concurrently in parallel at full hardware throughput.

Answer: False. When threads within a warp diverge on conditional branches, the warp executes each branch path serially while masking off (disabling) threads that do not take that path. The total execution time becomes the sum of the times of both paths, reducing hardware utilization and throughput.

Learning Objective: Analyze the performance penalty of warp divergence in SIMT architectures.
Describe the Tiling Principle in deep learning hardware mapping and explain why multi-level hierarchical tiling (from global memory down to registers) is necessary for high-throughput GEMM kernels.

Answer: The Tiling Principle partitions large matrix multiplication loops into smaller, block-sized submatrices that fit into each successive level of the hardware memory hierarchy (HBM $\to$ shared memory/SRAM $\to$ register files). Hierarchical tiling ensures that data loaded from high-latency, limited-bandwidth memory (such as HBM) is reused dozens of times in high-bandwidth, low-latency on-chip storage before being evicted, preventing arithmetic pipelines from stalling on memory bandwidth.

Learning Objective: Explain how hierarchical tiling maximizes data reuse and sustains peak compute utilization.

← Back to Questions

Self-Check: Answer

An accelerator delivers $R_{\text{peak}} = 1{,}000\text{ TFLOP/s}$ ($10^{15}\text{ FLOP/s}$) in FP16 and has a High-Bandwidth Memory (HBM) subsystem delivering $\text{BW} = 2.0\text{ TB/s}$ ($2\times 10^{12}\text{ bytes/s}$). What is the hardware balance point (ridge point $I_{\text{ridge}}$) of this system?
1. $50\text{ FLOP/byte}$
2. $200\text{ FLOP/byte}$
3. $500\text{ FLOP/byte}$
4. $2{,}000\text{ FLOP/byte}$
Answer: The correct answer is C. The hardware ridge point is defined as $I_{\text{ridge}} = \frac{R_{\text{peak}}}{\text{BW}} = \frac{10^{15}\text{ FLOP/s}}{2.0 \times 10^{12}\text{ bytes/s}} = 500\text{ FLOP/byte}$. Any kernel with an arithmetic intensity below $500\text{ FLOP/byte}$ will be memory-bandwidth bound on this hardware, whereas kernels with arithmetic intensity above $500\text{ FLOP/byte}$ can potentially achieve peak compute throughput. The other choices result from arithmetic miscalculations ($50$, $200$, or $2{,}000$).

Learning Objective: Calculate the hardware ridge point given peak compute throughput and memory bandwidth.
When training a 7-billion parameter model using standard mixed-precision (FP16/BF16) with the Adam optimizer, calculate the minimum memory required purely for model states (weights, gradients, and optimizer states) and explain why optimizer states dominate this footprint.

Answer: Model states require 16 bytes per parameter: 2 bytes for FP16 weights, 2 bytes for FP16 gradients, 4 bytes for FP32 master weights, 4 bytes for FP32 first momentum, and 4 bytes for FP32 second momentum ($2 + 2 + 4 + 4 + 4 = 16\text{ bytes/param}$). For a 7B model, this requires $7 \times 10^9 \times 16\text{ bytes} = 112\text{ GB}$. Optimizer states dominate ($12\text{ bytes/param}$, or $75\%$ of the total) because maintaining FP32 precision for master weights and running statistical moments is numerically necessary to prevent gradient underflow and truncation errors during updates.

Learning Objective: Calculate and justify the memory footprint of model weights, gradients, and optimizer states during mixed-precision training.
**Arrange the following levels of a modern GPU memory hierarchy in order of access latency, from lowest latency (fastest) to highest latency (slowest):

High-Bandwidth Memory (HBM3)
Register File
Pinned Host System Memory (DDR5 via PCIe)
Shared Memory / L1 Cache
On-Chip L2 Cache**

Answer: The correct order is (2) Register File -> (4) Shared Memory / L1 Cache -> (5) On-Chip L2 Cache -> (1) High-Bandwidth Memory (HBM3) -> (3) Pinned Host System Memory (DDR5 via PCIe). Registers are accessible in sub-nanosecond/single-cycle latency, followed by on-chip Shared Memory/L1 (~few cycles), L2 Cache (~tens of cycles), device HBM (~hundreds of cycles), and finally host DDR over the PCIe bus (~microseconds/thousands of cycles).

Learning Objective: Classify memory hierarchy levels by access latency and proximity to execution units.

Why does High-Bandwidth Memory (HBM) achieve significantly higher bandwidth (e.g., $>2\text{ TB/s}$) than traditional GDDR6X memory (e.g., $\approx 760\text{ GB/s}$) while maintaining comparable or lower power per bit?
1. HBM operates at a $10\times$ higher clock frequency than GDDR6X on standard PCB traces
2. HBM uses optical photonic signaling to transmit data across the motherboard
3. HBM eliminates all error correction codes and row buffer precharge cycles
4. HBM vertically stacks DRAM dies using Through-Silicon Vias (TSVs) and connects to the GPU via a wide 1024-bit per stack silicon interposer bus at lower clock speeds
Answer: The correct answer is D. HBM achieves ultra-high bandwidth by vertically stacking DRAM dies with Through-Silicon Vias (TSVs) and interfacing with the accelerator through a silicon interposer with extremely wide memory buses (1024 bits per stack vs. 32/64 bits for standard GDDR channels). Because physical wire distances over the silicon interposer are short and the bus is wide, HBM can run at lower pin clock frequencies, significantly reducing energy per bit ($pJ/\text{bit}$) compared to driving high-frequency GDDR signals over long PCB traces. The higher clock frequency choice is incorrect because GDDR6X actually runs at higher pin clock frequencies than HBM. The optical photonics and error-correction elimination choices are physically false.

Learning Objective: Explain the architectural design differences and physical advantages of HBM over GDDR.
Memory allocated on the host CPU that is locked into physical RAM and prevents operating system paging, enabling direct DMA transfers over PCIe to the GPU, is called ____ host memory.

Answer: pinned (or page-locked). Pinned host memory allows GPU DMA engines to transfer data asynchronously across the PCIe bus without CPU staging copies or page faults.

Learning Objective: Explain the role of pinned host memory in optimizing host-to-accelerator data transfers.
True or False: The AI Memory Wall refers solely to the limited physical capacity (GBs) of GPU DRAM, meaning that if an accelerator has sufficient gigabytes to store model weights, memory bandwidth will never bottleneck execution.

Answer: False. The AI Memory Wall encompasses both capacity and bandwidth disparities. While capacity determines whether a model fits on a chip, memory bandwidth ($\text{TB/s}$) dictates how fast data can be fed to compute units. Even if a model fits entirely in DRAM, low-arithmetic-intensity kernels (such as LayerNorm or token-by-token decoding) remain heavily bottlenecked by memory bandwidth, underutilizing peak compute throughput.

Learning Objective: Analyze the dual dimensions (capacity and bandwidth) of the AI memory wall.

← Back to Questions

Self-Check: Answer

A developer runs a LayerNorm kernel on an accelerator with $R_{\text{peak}} = 312\text{ TFLOP/s}$ and $\text{BW} = 1.5\text{ TB/s}$ ($I_{\text{ridge}} = 208\text{ FLOP/byte}$). The LayerNorm has an arithmetic intensity of $I = 4\text{ FLOP/byte}$. What is the maximum attainable performance of this kernel, and what is the binding bottleneck?
1. $6.0\text{ TFLOP/s}$, bound by memory bandwidth
2. $312\text{ TFLOP/s}$, bound by peak compute capacity
3. $78\text{ TFLOP/s}$, bound by warp scheduler instruction issue rate
4. $1.5\text{ TFLOP/s}$, bound by PCIe bus transfer limits
Answer: The correct answer is A. According to the Roofline model, $\text{Attainable Performance} = \min(R_{\text{peak}}, I \cdot \text{BW}) = \min(312\text{ TFLOP/s}, 4\text{ FLOP/byte} \times 1.5\text{ TB/s}) = \min(312, 6.0) = 6.0\text{ TFLOP/s}$. Because $I = 4 < I_{\text{ridge}} = 208\text{ FLOP/byte}$, the kernel operates deep within the memory-bound regime, achieving less than $2\%$ of the chip’s peak arithmetic throughput. The peak compute choice incorrectly assumes compute limits apply regardless of arithmetic intensity. The instruction issue and PCIe bus choices confuse the primary memory bandwidth limit with secondary bottlenecks.

Learning Objective: Calculate attainable performance using the Roofline model and identify the binding bottleneck.
Why has the hardware ridge point ($I_{\text{ridge}}$) increased dramatically across successive GPU generations (e.g., from Volta to Ampere to Hopper), and what pressure does this trend place on compiler and kernel developers?

Answer: The ridge point $I_{\text{ridge}} = R_{\text{peak}}/\text{BW}$ has increased because peak arithmetic throughput (driven by specialized Tensor Cores and lower-precision FP8/INT8 math) has grown much faster than physical HBM bandwidth. This shifts the knee of the roofline curve to the right, meaning kernels require significantly higher arithmetic intensity to achieve compute-bound peak efficiency. Developers and compilers are forced to implement aggressive kernel fusion, multi-level tiling, and activation caching to avoid being trapped in the memory-bound regime.

Learning Objective: Analyze the historical trend of increasing hardware ridge points and its architectural implications.
True or False: When a kernel operates in the memory-bound regime of the Roofline model ($I < I_{\text{ridge}}$), doubling the accelerator’s peak tensor compute capability ($R_{\text{peak}}$) without changing memory bandwidth will double the kernel’s execution speed.

Answer: False. In the memory-bound regime, performance is capped by $I \cdot \text{BW}$, which depends strictly on arithmetic intensity and memory bandwidth. Increasing $R_{\text{peak}}$ only raises the horizontal compute ceiling; the kernel’s execution time is dictated by the slanted bandwidth ceiling and will experience $0\%$ speedup unless memory bandwidth is increased or the kernel is restructured to improve data reuse.

Learning Objective: Evaluate the effect of hardware upgrades on memory-bound workloads using the Roofline model.
In the Roofline model, the transition point on the horizontal axis where the memory-bandwidth ceiling intersects the peak-compute ceiling is known as the hardware ____ point.

Answer: ridge. The ridge point ($I_{\text{ridge}} = R_{\text{peak}} / \text{BW}$) defines the minimum arithmetic intensity required for an algorithm to potentially reach peak hardware computational throughput.

Learning Objective: Explain the significance of the ridge point in roofline performance modeling.
An engineer profiles a transformer inference workload and discovers that the attention Softmax kernel is heavily memory bandwidth bound. Which of the following optimization techniques directly increases arithmetic intensity to move the kernel closer to the compute-bound regime?
1. Upgrading host CPU RAM to DDR5 to decrease kernel enqueue latency
2. Fusing the scale, mask, Softmax, and dropout operations into a single kernel to keep intermediate activations in registers/SRAM
3. Increasing the clock frequency of the GPU Tensor Cores by $15\%$
4. Disabling warp scheduler out-of-order instruction issue
Answer: The correct answer is B. Fusing scale, mask, Softmax, and dropout into a single kernel eliminates intermediate writes to and reads from DRAM (HBM), keeping intermediate values in fast on-chip registers and shared memory. This drastically reduces the total memory traffic $D_{\text{vol}}$, increasing the arithmetic intensity $I = \text{FLOP}/\text{Byte}$ and moving the kernel closer to or into the compute-bound regime. Upgrading host CPU RAM does not affect GPU HBM memory traffic. Increasing Tensor Core clock frequency only raises the peak compute ceiling without improving memory-bound throughput. Disabling warp scheduling hurts instruction issue efficiency.

Learning Objective: Design optimization strategies to shift memory-bound kernels toward the compute-bound regime.

← Back to Questions

Self-Check: Answer

In neural network hardware mapping, what distinguishes a spatial mapping decision from a temporal mapping decision?
1. Spatial mapping refers to compiling graph IR, whereas temporal mapping refers to runtime CUDA kernel launches
2. Spatial mapping determines precision formats (FP16 vs INT8), whereas temporal mapping determines memory allocation sizes
3. Spatial mapping assigns computational tasks to specific physical execution units (e.g., PEs or SMs) simultaneously in parallel, whereas temporal mapping determines the execution ordering and loop scheduling over time on those units
4. Spatial mapping operates only on convolutional layers, whereas temporal mapping operates only on transformer attention layers
Answer: The correct answer is C. Spatial mapping decides how tensor dimensions and parallel operations are partitioned and assigned across physical execution resources (such as array PEs, GPU SMs, or SIMD vector lanes) to execute concurrently in space. Temporal mapping decides the chronological schedule, loop ordering, and time steps in which operations and tile iterations execute on those assigned physical units over time. The compiler IR vs runtime launch distinction confuses compilation phases with mapping dimensions. The precision vs memory allocation choice describes quantization and memory planning. The layer-specific choice is incorrect as both mapping types apply to all neural network operations.

Learning Objective: Compare spatial mapping and temporal mapping dimensions in AI hardware acceleration.
Explain why finding the optimal hardware mapping (tiling sizes, loop orders, and spatial partitioning) for a deep neural network on a target accelerator is a combinatorially hard optimization problem.

Answer: Mapping involves searching a discrete combinatorial space of loop transformations across multi-dimensional tensor operations. For an $M$-deep nested loop across $K$ memory hierarchy levels, there are $M!$ possible loop orderings at each level, multiplied by all possible tile factor combinations that divide loop bounds, unrolling factors, and spatial allocation strategies across processing elements. Furthermore, memory capacity constraints at each tier create complex non-linear dependencies, making exhaustive evaluation computationally intractable and requiring heuristic or learning-based search spaces.

Learning Objective: Analyze the combinatorial complexity of hardware mapping for neural network compilation.
Explain why reordering loop nests in a tensor contraction (e.g., changing from $I \to J \to K$ to $K \to I \to J$) alters memory bandwidth demands and execution speed without changing the total mathematical operation count.

Answer: Loop reordering preserves the mathematical invariants and total scalar multiply-accumulate count ($\mathcal{O}(I \cdot J \cdot K)$ operations) but fundamentally changes the data access patterns and residency in cache and scratchpad memory tiers. The inner loop dictates which tensor operand ($A, B,$ or partial sum $C$) is held stationary in registers or fast SRAM while other operands stream through. Selecting a suboptimal loop ordering causes cache thrashing and repeated DRAM round-trips for the streamed operands, increasing memory bandwidth demand and drastically degrading execution performance.

Learning Objective: Explain why loop reordering impacts memory bandwidth demand without altering total arithmetic operations.
When mapping a tensor computation to a multi-level memory hierarchy, what is the primary objective function optimized by spatial and temporal tiling?
1. Maximizing the total number of intermediate tensors written to host DDR memory
2. Maximizing data reuse in the fastest, closest memory tiers (registers and SRAM) to minimize traffic to slower, energy-expensive DRAM
3. Ensuring every warp thread executes different instruction streams simultaneously
4. Converting all 2D matrix multiplications into 1D scalar operations
Answer: The correct answer is B. The primary goal of spatial and temporal mapping is to maximize data reuse in the highest levels of the memory hierarchy (registers and on-chip SRAM/shared memory), ensuring that each byte fetched from power-hungry, high-latency DRAM is reused as many times as possible before eviction, thereby minimizing DRAM bandwidth demand and maximizing compute throughput. Maximizing writes to host memory degrades performance. Executing different instructions causes severe warp divergence. Converting matrix operations to scalars eliminates vector and tensor unit acceleration.

Learning Objective: Justify the primary objective of tiling and memory hierarchy mapping.

← Back to Questions

Self-Check: Answer

Why do modern deep learning libraries (e.g., cuDNN, TensorRT) strongly prefer the Channels-Last (NHWC) tensor layout over Channels-First (NCHW) when executing convolutions on NVIDIA Tensor Cores?
1. NHWC eliminates the need for spatial convolutions by flattening images into 1D vectors
2. NCHW requires floating-point numbers to be stored in big-endian byte order
3. NHWC reduces model parameter count by sharing channel weights across batches
4. NHWC places channel values for a given spatial location in contiguous memory, aligning with the packed vector/matrix multiply requirements of Tensor Cores
Answer: The correct answer is D. In NHWC layout, the channel dimension $C$ is the fastest-varying (innermost) dimension, placing all channel features for a specific spatial coordinate $(n, h, w)$ contiguously in memory. Tensor Cores execute matrix operations on contiguous vectors of channels (e.g., 8 or 16 channels per memory vector load), enabling coalesced memory access and direct loading into matrix unit registers without expensive transpose or gather operations. The 1D flattening, big-endian, and parameter sharing choices are technically incorrect.

Learning Objective: Compare the memory layout efficiency of Channels-Last (NHWC) versus Channels-First (NCHW) for Tensor Cores.
Explain the mechanism of vertical kernel fusion (e.g., fusing Conv2D $\to$ BatchNorm $\to$ ReLU), and identify both its performance benefit and its primary architectural constraint.

Answer: Vertical kernel fusion combines a sequence of producer-consumer operations into a single GPU kernel, passing intermediate activation values directly through fast on-chip registers or shared memory instead of writing them out to and re-reading them from global DRAM (HBM). This dramatically reduces memory traffic and kernel launch overhead. Its primary architectural constraint is register pressure: storing intermediate variables and fusing complex pipelines increases register usage per thread, which can reduce warp occupancy and limit available parallelism on the SM.

Learning Objective: Explain the performance benefits and hardware constraints of vertical kernel fusion.
**Place the following loop transformation steps in the logical order applied by an optimizing compiler when targeting a matrix multiplication dataflow to an accelerator:

Loop Reordering: Permute loop indices to establish a specific stationary dataflow (e.g., Output-Stationary)
Loop Unrolling: Fully or partially unroll the innermost loop to expose instruction-level parallelism and map to hardware registers
Loop Tiling (Blocking): Partition global loop iterations into sub-tiles that fit into on-chip shared memory / SRAM
Spatial Partitioning: Assign outer tile loops to physical hardware compute clusters (e.g., GPU thread blocks / SMs)**

Answer: The correct order is (3) Loop Tiling (Blocking): Partition global loop iterations into sub-tiles that fit into on-chip shared memory / SRAM -> (4) Spatial Partitioning: Assign outer tile loops to physical hardware compute clusters (e.g., GPU thread blocks / SMs) -> (1) Loop Reordering: Permute loop indices to establish a specific stationary dataflow (e.g., Output-Stationary) -> (2) Loop Unrolling: Fully or partially unroll the innermost loop to expose instruction-level parallelism and map to hardware registers. Optimization starts with hierarchical tiling to match memory capacities, partitions tiles spatially across SMs/PEs, reorders the inner loops to minimize data movement (dataflow choice), and finally unrolls inner loops for instruction issue efficiency.

Learning Objective: Apply loop transformation pipelines for optimizing matrix dataflow on accelerators.

In a multi-head self-attention layer where a single input activation tensor is projected across multiple query, key, and value weight matrices ($W_q, W_k, W_v$), which stationary dataflow strategy provides the highest data reuse in local scratchpad memory?
1. Input-Stationary (Activation-Stationary)
2. Output-Stationary
3. No Local Reuse (NLR)
4. Weight-Stationary
Answer: The correct answer is A. In an Input-Stationary (Activation-Stationary) dataflow, an input activation tile is loaded into fast local memory once and kept stationary while multiple weight matrices ($W_q, W_k, W_v$) stream through. This maximizes the reuse of the common activation tensor across multiple matrix multiplications, minimizing activation re-reads. Weight-Stationary would require repeatedly reloading activations for each distinct weight matrix. Output-Stationary minimizes accumulation write-backs but does not exploit cross-projection activation sharing. No Local Reuse streams all operands from memory without local caching.

Learning Objective: Design dataflow strategies for multi-head attention workloads.
True or False: Row-Stationary (RS) dataflow, as implemented in architectures like Eyeriss, keeps only the final output activation stationary in registers while streaming 2D convolutional filter rows and input rows from DRAM on every clock cycle.

Answer: False. Row-Stationary dataflow keeps 1D rows of convolutional weights stationary in PE registers, slides 1D rows of input activations through the PEs, and accumulates 1D rows of partial sums locally across a 2D array of PEs, maximizing 2D convolution spatial reuse across all three tensor components simultaneously.

Learning Objective: Evaluate the operational mechanics of Row-Stationary dataflow in convolution accelerators.

← Back to Questions

Self-Check: Answer

What is the primary purpose of lifetime analysis in an ML compiler’s static memory planner?
1. To calculate the physical degradation and failure rate of HBM memory cells over time
2. To determine the precise intervals during which each intermediate activation tensor is needed, allowing disjoint tensors to share the same physical memory buffer
3. To predict the number of training epochs required for a neural network to converge
4. To prevent the compiler from generating out-of-order instruction streams
Answer: The correct answer is B. Lifetime analysis tracks when each intermediate tensor is created (produced) and last referenced (consumed) during graph execution. By identifying tensors whose lifetimes do not overlap, the compiler’s static memory planner can assign them to the same physical memory addresses (buffer reuse/aliasing), significantly reducing the model’s peak runtime memory footprint and avoiding dynamic memory allocation overhead. The physical degradation choice confuses tensor lifecycle with semiconductor physics. The training convergence choice describes algorithmic optimization rather than compiler memory management. The instruction stream choice describes scheduling.

Learning Objective: Explain the role of tensor lifetime analysis in static memory planning.
Explain how double buffering (software pipelining) implemented by an ML compiler hides memory access latency during loop execution on an accelerator.

Answer: Double buffering allocates two alternating memory buffers in on-chip SRAM/shared memory: while compute units execute arithmetic operations on data in the first buffer (tile $k$), asynchronous DMA or copy engines concurrently load subsequent data (tile $k+1$) into the second buffer from DRAM. In the next iteration, the roles swap. By overlapping data transfer with arithmetic computation, memory transfer latency is completely hidden as long as the compute time equals or exceeds the transfer time.

Learning Objective: Explain the mechanism of double buffering in hiding memory transfer latency.
**Place the following compilation stages in the correct order as an end-to-end ML compiler (such as TVM or XLA) transforms a high-level deep learning model into executable machine code:

Target Code Generation: Emit hardware-specific binary (e.g., PTX or machine instructions)
High-Level Graph Optimization: Perform operator fusion, constant folding, and dead code elimination on the computation graph
Front-End Ingestion: Parse framework model (e.g., PyTorch/ONNX) into High-Level Graph IR
Low-Level IR & Auto-Tuning: Lower fused operators to loop-level IR and optimize tile sizes, thread bindings, and unroll factors
Static Memory Planning: Analyze tensor lifetimes and allocate shared physical buffers**

Answer: The correct order is (3) Front-End Ingestion: Parse framework model (e.g., PyTorch/ONNX) into High-Level Graph IR -> (2) High-Level Graph Optimization: Perform operator fusion, constant folding, and dead code elimination on the computation graph -> (4) Low-Level IR & Auto-Tuning: Lower fused operators to loop-level IR and optimize tile sizes, thread bindings, and unroll factors -> (5) Static Memory Planning: Analyze tensor lifetimes and allocate shared physical buffers -> (1) Target Code Generation: Emit hardware-specific binary (e.g., PTX or machine instructions). Compilation begins with parsing graph IR, performs target-independent graph optimizations, lowers to loop IR for hardware auto-tuning and tiling, plans memory buffers, and generates native device code.

Learning Objective: Apply the multi-stage compilation pipeline of modern ML compilers.

How does an auto-tuning ML compiler (such as TVM/Ansor) differ from a traditional handwritten library approach (such as cuDNN) for kernel selection?
1. Auto-tuning compilers execute code only on the host CPU, whereas handwritten libraries run on GPUs
2. Handwritten libraries search an infinite combinatorial loop space at runtime, whereas auto-tuning compilers use static heuristics
3. Auto-tuning compilers explore large parameterized search spaces of loop transformations and tile sizes using cost models to generate custom kernels, whereas handwritten libraries rely on expert-tuned templates for specific fixed shapes
4. Auto-tuning compilers require all tensors to be quantized to 1-bit integers
Answer: The correct answer is C. Auto-tuning ML compilers define parameterized spaces of loop transformations (tiling, unrolling, vectorization, thread mapping) and evaluate configurations using statistical cost models or hardware measurements to generate custom kernels optimized for any arbitrary tensor shape and target architecture. In contrast, vendor libraries rely on human experts who write and optimize specific kernel templates for common fixed shapes and precisions. The host CPU only claim is false. The runtime infinite search claim is reversed. The 1-bit quantization claim is unrelated.

Learning Objective: Compare auto-tuning ML compilers with handwritten vendor libraries.
The optimization where an ML compiler merges two or more independent operators at the same graph depth into a single batched kernel to maximize GPU parallelism is known as ____ fusion.

Answer: horizontal. Horizontal fusion combines independent parallel operators (such as parallel projection layers or multi-head $Q, K, V$ linear transformations) into a single batched kernel, improving hardware occupancy and amortizing kernel launch overhead.

Learning Objective: Classify horizontal versus vertical kernel fusion techniques.

← Back to Questions

Self-Check: Answer

Why do deep learning frameworks implement custom caching memory allocators (such as PyTorch’s caching_allocator) rather than directly invoking cudaMalloc and cudaFree for every intermediate tensor?
1. cudaMalloc operates only in FP32 precision and cannot allocate FP16 memory buffers
2. Direct OS memory allocation encrypts all tensor data, introducing cryptographic decryption latency
3. GPU DRAM can only be allocated once during system boot time
4. cudaMalloc is a synchronous operation that stalls GPU execution and causes expensive driver and OS page-table synchronization overhead
Answer: The correct answer is D. cudaMalloc and cudaFree are synchronous system calls that require driver synchronization, virtual memory address mapping, and device-wide mutex locking, introducing significant latency ($\approx 10\text{--}100\,\mu\text{s}$) that stalls asynchronous execution streams. A caching memory allocator pre-allocates large memory blocks and manages a userspace pool of memory chunks on the device, servicing tensor allocations and deallocations in sub-microsecond time without GPU-host synchronization. Precision is independent of raw memory allocation. OS encryption is not part of standard cudaMalloc. GPU memory can be dynamically allocated at runtime.

Learning Objective: Analyze the architectural necessity and performance advantages of caching memory allocators in ML runtimes.
Explain how CUDA Graphs eliminate CPU kernel launch overhead during repeated training or inference iterations on small or low-latency models.

Answer: In standard execution, every GPU kernel requires the CPU to enqueue a launch request via the driver, incurring $\approx 5\text{--}10\,\mu\text{s}$ of CPU overhead per kernel, which severely bottlenecks short-duration kernels. CUDA Graphs capture the entire directed acyclic graph (DAG) of kernel launches, memory copies, and stream dependencies during an initial trace pass. In subsequent iterations, the entire graph is instantiated and replayed via a single driver call, offloading execution sequencing entirely to the GPU hardware scheduler and eliminating host-side dispatch overhead.

Learning Objective: Explain how CUDA Graphs capture and replay execution workflows to eliminate host launch latency.
True or False: In an asynchronous GPU runtime model, when a Python script executes y = torch.matmul(a, b), the CPU thread blocks and waits until the GPU hardware finishes computing the matrix multiplication before executing the next line of Python code.

Answer: False. GPU runtime calls are asynchronous: torch.matmul enqueues the kernel execution command onto a CUDA stream queue in the driver and immediately returns control to the CPU thread. The CPU continues executing subsequent Python instructions while the GPU executes the kernel in the background, blocking only when synchronization is explicitly requested (e.g., via torch.cuda.synchronize() or copying data back to CPU host memory).

Learning Objective: Analyze the non-blocking execution model of asynchronous GPU runtime streams.
A sequence of asynchronous GPU operations that execute strictly in FIFO order on an accelerator is called a CUDA ____.

Answer: stream. CUDA streams allow operations enqueued within the same stream to execute sequentially while operations in different streams can execute concurrently in parallel when hardware resources permit.

Learning Objective: Explain the role of CUDA streams in managing concurrent and asynchronous GPU execution.
An inference serving system processes dynamic batch sizes ranging from 1 to 32 tokens per request. Why does dynamic batching pose a significant challenge to runtime kernel selection and hardware efficiency?
1. Optimal tile sizes, thread block configurations, and memory bandwidth requirements change across batch sizes, making a single static kernel inefficient across all shapes
2. Dynamic batching forces the GPU to switch from FP16 to FP64 precision for even batch sizes
3. Tensor Cores cannot execute matrix multiplication when batch dimensions are not powers of two
4. The GPU memory controller must physically power down DRAM banks when batch size decreases
Answer: The correct answer is A. For small batch sizes (e.g., $B=1$), operations are memory-bandwidth bound and require kernels tuned for low latency and high memory throughput, whereas for large batch sizes (e.g., $B=32$), operations become compute bound and require larger tile sizes and high-occupancy thread block configurations to saturate Tensor Cores. A single static kernel configuration cannot achieve peak efficiency across disparate tensor shapes, requiring runtime kernel dispatch or multi-version code generation. Precision switching, power-of-two Tensor Core limitations, and DRAM bank power-downs are incorrect.

Learning Objective: Analyze the performance challenges of dynamic shapes on runtime kernel selection.

← Back to Questions

Self-Check: Answer

Why have accelerator architectures increasingly shifted from monolithic single-die designs toward Multi-Chip Module (MCM) and chiplet architectures?
1. Chiplets eliminate all need for semiconductor fabrication foundries
2. Monolithic dies are constrained by the physical lithography reticle limit ($\approx 858\text{ mm}^2$) and suffer exponential yield loss as die size increases
3. Monolithic dies cannot support high-bandwidth memory (HBM) interfaces
4. Chiplets allow electrical signals to travel faster than the speed of light
Answer: The correct answer is B. Monolithic silicon dies are physically bounded by optical lithography reticle limits (typically $\approx 858\text{ mm}^2$). Furthermore, manufacturing defect density causes wafer yield to drop exponentially as die area approaches the reticle limit, making large monolithic chips prohibitively expensive. Chiplet/MCM architectures break the system into smaller, high-yield dies interconnected via high-density silicon bridges or interposers, enabling much larger aggregate compute and memory capacity. The elimination of foundries is absurd; HBM is supported on monolithic dies (e.g., A100/H100); physical constants cannot be exceeded.

Learning Objective: Justify the transition from monolithic dies to chiplet-based accelerator architectures.
Explain how the non-uniform memory access (NUMA) effect and interconnect bandwidth degradation impact performance when scaling a neural network workload across multiple chiplets or accelerator chips.

Answer: While on-chip or intra-die memory access delivers high bandwidth ($>2\text{--}3\text{ TB/s}$) and low latency, crossing chiplet boundaries via inter-die bridges or board interconnects (e.g., NVLink or PCIe) suffers an order-of-magnitude reduction in bandwidth and increased latency. If a tensor computation frequently requires operands from remote chiplets without local caching, execution stalls on inter-chip communication, creating a NUMA bottleneck that degrades parallel scaling efficiency unless data is carefully partitioned.

Learning Objective: Analyze the NUMA and interconnect bandwidth constraints in multi-chip scaling.
Describe the architectural rationale behind Cerebras’s Wafer-Scale Engine and explain how fabricating an accelerator across an entire uncut silicon wafer overcomes traditional multi-chip scaling bottlenecks.

Answer: Standard multi-chip scaling suffers significant latency and bandwidth penalties when signals cross chip package boundaries and PCB traces. Cerebras fabricates hundreds of thousands of cores across an entire uncut $300\text{ mm}$ silicon wafer, utilizing on-wafer routing to connect adjacent reticle fields. This provides uniform, ultra-high-bandwidth, single-cycle communication and massive distributed on-chip SRAM across the entire wafer, eliminating discrete chip packaging, external SerDes transceivers, and NUMA interconnect bottlenecks.

Learning Objective: Explain the architectural rationale and communication advantages of wafer-scale computing systems.
What is the primary bottleneck introduced by inter-node scaling (scaling out across separate servers over Ethernet or InfiniBand) compared to intra-node NVLink scaling?
1. Inter-node network adapters cannot transmit FP16 or BF16 floating-point values
2. Inter-node communication requires GPUs to switch to single-threaded CPU emulation mode
3. Inter-node network bandwidth (e.g., $400\text{ Gbps} \approx 50\text{ GB/s}$) is roughly an order of magnitude lower than intra-node NVLink bandwidth ($\approx 900\text{ GB/s}$), increasing collective communication latency
4. Inter-node scaling eliminates the need for gradient synchronization in distributed training
Answer: The correct answer is C. Intra-node GPU interconnects like NVLink 4 deliver up to $900\text{ GB/s}$ of bidirectional bandwidth per GPU, whereas inter-node network interfaces (e.g., $400\text{ Gbps}$ InfiniBand or RoCE NICs) deliver $\approx 50\text{ GB/s}$ per port. This sharp bandwidth drop across the node boundary makes inter-node collective communication (such as All-Reduce or All-to-All) a dominant bottleneck in distributed scaling, requiring hierarchical communication algorithms. The network adapter precision and CPU emulation choices are false. The gradient synchronization elimination choice is incorrect because distributed data parallelism requires gradient synchronization.

Learning Objective: Compare the bandwidth constraints of intra-node interconnects with inter-node scale-out networks.

← Back to Questions

Self-Check: Answer

On a modern mobile heterogeneous System-on-Chip (SoC) featuring big.LITTLE CPUs, a mobile GPU, and a dedicated NPU, which compute engine is most energy-efficient for running continuous, low-latency 8-bit quantized convolutional inference?
1. The high-performance ‘big’ CPU core running single-threaded FP32 instructions
2. The out-of-order system memory controller
3. The dedicated Neural Processing Unit (NPU) optimized for quantized INT8 matrix operations
4. The host operating system virtualization hypervisor
Answer: The correct answer is C. The Neural Processing Unit (NPU) is a domain-specific accelerator with fixed-point MAC arrays, specialized activation units, and local scratchpad SRAM designed specifically for INT8/INT4 neural network operations. It delivers an order-of-magnitude higher energy efficiency (TOPS/Watt) than general-purpose CPU cores or high-power GPUs by avoiding instruction fetch/decode overhead and general register file transfers. The big CPU core consumes substantially more power per operation. Memory controllers and hypervisors do not execute tensor math.

Learning Objective: Classify mobile SoC compute engines based on workload efficiency and architectural specialization.
In automotive autonomous driving SoCs, explain why deterministic worst-case execution time and lockstep redundancy are required, even if they reduce peak average-case throughput.

Answer: Automotive systems are subject to strict functional safety standards (such as ISO 26262 ASIL-D), where missing a perception or control deadline in real-time sensor processing can lead to catastrophic physical collisions. Dual-core lockstep execution runs identical computations redundantly across replicated hardware cores to detect transient hardware faults and bit flips immediately. While these safety mechanisms and strict scheduling guarantees introduce hardware overhead and reduce peak average throughput, they guarantee deterministic worst-case latency and fault tolerance necessary for life-critical autonomy.

Learning Objective: Explain the architectural trade-offs between safety-critical determinism and peak throughput in automotive AI SoCs.
True or False: In mobile SoCs with unified system memory (LPDDR), sharing physical RAM between the CPU, GPU, and NPU eliminates all data movement overhead between heterogeneous processors.

Answer: False. While unified memory eliminates physical PCIe bus transfers and duplicate memory copies across discrete devices, data must still be moved across the shared on-chip interconnect and through different cache hierarchy domains. Furthermore, cache coherency protocols, differing tensor layout requirements (e.g., NCHW vs NHWC), and memory bus contention between competing SoC engines still incur significant latency and energy overhead.

Learning Objective: Evaluate the memory movement and cache coherency trade-offs of unified SoC memory.
Describe how Dynamic Voltage and Frequency Scaling (DVFS) and thermal throttling constrain sustained AI inference performance on edge and mobile devices.

Answer: Mobile SoCs operate within a tight thermal design power (TDP) envelope (typically $3\text{--}5\text{ W}$) with passive cooling. When continuous AI workloads generate sustained heat, the thermal management system triggers DVFS to lower operating voltage and clock frequencies, preventing device overheating. This thermal throttling causes execution time to degrade over time, meaning peak burst performance cannot be sustained for continuous video or audio streaming workloads without proactive power-budget scheduling.

Learning Objective: Analyze the impact of thermal throttling and DVFS on sustained edge AI inference.

← Back to Questions

Self-Check: Answer

In the lifecycle carbon assessment of advanced deep learning accelerators, what constitutes ‘embodied carbon’?
1. The electrical energy consumed by the GPU during model forward and backward passes
2. The carbon emitted by datacenter air conditioning units during peak summer load
3. The carbon credits purchased by cloud providers to offset datacenter energy usage
4. The greenhouse gas emissions generated during raw material extraction, semiconductor silicon manufacturing, packaging, and hardware transportation
Answer: The correct answer is D. Embodied carbon refers to the total greenhouse gas emissions generated throughout the supply chain and manufacturing lifecycle of the hardware before it ever runs a workload, including silicon ingot purification, advanced extreme ultraviolet (EUV) lithography, cleanroom fabrication, multi-chip packaging, assembly, and transportation. Operational carbon refers to emissions resulting from electricity consumed during active operation and cooling of the hardware. Carbon credits are financial offsets, not physical emissions.

Learning Objective: Compare embodied carbon and operational carbon in AI hardware lifecycle analysis.
Explain why maximizing accelerator utilization (e.g., via multi-tenant sharing or continuous pipeline saturation) reduces the amortized carbon footprint per trained model.

Answer: The total carbon footprint of a machine learning model includes both operational carbon (energy consumed per training step) and a fractional share of the hardware’s embodied carbon amortized over its operational lifespan. When accelerator utilization is low (e.g., GPUs idling on data stalls), the embodied carbon is wasted on idle time. Maximizing utilization ensures that more useful compute operations are extracted over the device’s operational lifetime, minimizing the embodied carbon cost allocated to each trained model.

Learning Objective: Explain how hardware utilization impacts the amortized carbon footprint of AI workloads.
True or False: In a datacenter with a Power Usage Effectiveness (PUE) of 1.1, the cooling and electrical distribution infrastructure consumes more power than the actual computing IT equipment (servers and accelerators).

Answer: False. $\text{PUE} = \frac{\text{Total Facility Power}}{\text{IT Equipment Power}}$. A PUE of 1.1 means that for every $1.0\text{ Watt}$ consumed by the computing IT equipment, only $0.1\text{ Watts}$ (roughly $9\%$ of total power) is consumed by cooling, lighting, and power distribution overhead, indicating a highly efficient facility where computing equipment dominates power consumption.

Learning Objective: Calculate and interpret Power Usage Effectiveness (PUE) for AI datacenters.

← Back to Questions

Self-Check: Answer

A team prunes $70\%$ of the weights in a large language model using unstructured magnitude pruning, setting those weights to zero. However, when executing the pruned model on standard GPU dense Tensor Cores, inference latency is identical to or slower than the unpruned baseline. What is the primary cause of this pitfall?
1. Standard dense hardware cannot skip individual zero elements without structured patterns (e.g., 2:4) or specialized sparse matrix indexing, so dense matrix units still execute all multiplications while sparse formats add indexing overhead
2. Floating-point units automatically convert zero values into infinite loops
3. Unstructured pruning forces the GPU driver to downclock memory bandwidth to prevent overheating
4. The operating system kernel intercepts every zero multiplication and raises a hardware page fault
Answer: The correct answer is A. Dense Tensor Cores and SIMT pipelines execute instructions in lockstep on dense contiguous matrices. Unstructured zero values do not change the dense matrix dimensions, so standard hardware still loads and computes all values. If converted to general sparse formats (e.g., CSR), irregular non-coalesced memory access and index metadata decoding overhead negate any FLOP reduction on GPUs unless sparsity is extremely high ($>90\text{--}95\%$) or structured (such as 2:4). The infinite loop, driver downclocking, and OS page fault choices are physically incorrect.

Learning Objective: Explain why unstructured sparsity fails to accelerate execution on dense hardware architectures.
Describe the pitfall of ‘micro-offloading’ small tensor operations from CPU to GPU, and explain why a sequence of scalar operations can run slower on an accelerator than on the host CPU.

Answer: Offloading small tensor operations incurs PCIe host-to-device data transfer latency, host driver scheduling overhead, and kernel launch latency ($\approx 5\text{--}10\,\mu\text{s}$ per kernel). When a tensor contains only a few dozen or hundred elements, computation time is sub-microsecond, meaning fixed transfer and launch overhead completely dominates execution time. Furthermore, small tensors cannot provide enough parallel work to saturate thousands of GPU cores, making scalar or vector execution on low-latency CPU caches much faster.

Learning Objective: Analyze the performance pitfalls of kernel launch and data transfer overhead in micro-offloading.
**Place the following diagnostic steps in the recommended sequence when troubleshooting an underperforming neural network training workload on an accelerator cluster:

Roofline & Hardware Counter Analysis: Determine whether individual kernels are compute-bound, memory-bandwidth-bound, or latency-bound
Amdahl & Host-Side Profiling: Identify serial bottlenecks, data loading stalls, and CPU-GPU synchronization delays
Kernel-Level Optimization: Apply operator fusion, hierarchical tiling, or precision reduction targeted to the identified bottleneck
Multi-Device Communication Profiling: Check for gradient all-reduce synchronization delays and interconnect saturation**

Answer: The correct order is (2) Amdahl & Host-Side Profiling: Identify serial bottlenecks, data loading stalls, and CPU-GPU synchronization delays -> (1) Roofline & Hardware Counter Analysis: Determine whether individual kernels are compute-bound, memory-bandwidth-bound, or latency-bound -> (3) Kernel-Level Optimization: Apply operator fusion, hierarchical tiling, or precision reduction targeted to the identified bottleneck -> (4) Multi-Device Communication Profiling: Check for gradient all-reduce synchronization delays and interconnect saturation. Diagnostics begin at the macro level (Amdahl unaccelerated fractions and host bottlenecks), zoom in to single-device kernel roofline profiling, optimize the binding kernel bottlenecks, and finally scale up to inter-device communication analysis.

Learning Objective: Apply a systematic diagnostic sequence to identify and resolve accelerator performance bottlenecks.

Which of the following statements represents a classic fallacy regarding peak accelerator specifications?
1. High-bandwidth memory reduces the latency of memory-bound operations compared to standard DDR5
2. An accelerator with $2\times$ higher peak theoretical TFLOP/s will automatically deliver a $2\times$ speedup on any neural network workload
3. Kernel fusion can improve arithmetic intensity by reducing global memory traffic
4. Warp divergence reduces the execution efficiency of SIMT processor lanes
Answer: The correct answer is B. Assuming that doubling peak theoretical TFLOP/s automatically doubles real-world workload performance is a classic fallacy. If the workload is memory-bandwidth bound (e.g., LayerNorm, embedding lookup, or autoregressive decoding), bound by host communication (Amdahl’s law), or limited by kernel launch overhead, increasing peak compute capacity produces negligible or zero speedup. The other choices represent well-established architectural facts.

Learning Objective: Evaluate common fallacies regarding peak accelerator hardware specifications.

← Back to Questions

Self-Check: Answer

What is the central architectural insight of the hardware acceleration chapter regarding the interaction between model architecture and accelerator efficiency?
1. Accelerators will soon eliminate all memory hierarchies in favor of infinite register files
2. Hardware efficiency depends entirely on maximizing clock frequency regardless of memory bandwidth
3. High delivered hardware efficiency requires co-design across computational primitives, dataflow reuse strategies, memory hierarchies, and compiler-runtime systems
4. General-purpose out-of-order CPUs remain superior to specialized TPUs for all deep learning workloads
Answer: The correct answer is C. The core thesis of AI hardware acceleration is that raw compute scaling alone cannot sustain performance gains. Achieving high delivered efficiency ($\eta_{\text{hw}}$) requires comprehensive co-design: aligning neural network mathematical primitives (matrix, vector, transcendental) with specialized execution units (Tensor Cores, systolic arrays, SFUs), selecting dataflow strategies that maximize on-chip data reuse, and leveraging compilers and runtimes to optimize memory allocation, kernel fusion, and asynchronous execution. The other choices contradict the chapter’s fundamental principles.

Learning Objective: Evaluate the principles of hardware-software co-design across the AI acceleration stack.
Summarize how the Roofline model serves as a unified diagnostic bridge connecting high-level neural network operations to low-level hardware architecture choices.

Answer: The Roofline model characterizes any neural network operation by its arithmetic intensity ($I = \text{FLOP}/\text{byte}$) and maps it against the hardware’s peak compute capacity ($R_{\text{peak}}$) and memory bandwidth ($\text{BW}$). By comparing $I$ to the hardware ridge point ($I_{\text{ridge}} = R_{\text{peak}}/\text{BW}$), the model immediately diagnoses whether an operator is memory-bound or compute-bound, directing engineers to the appropriate optimization: kernel fusion and tiling to increase data reuse for memory-bound operators, or Tensor Core utilization and parallelism tuning for compute-bound operators.

Learning Objective: Explain how the Roofline model bridges neural network algorithms and accelerator hardware design.
Which combination correctly summarizes the primary function of each layer in the modern AI hardware acceleration software stack?
1. Framework: Silicon manufacturing; Compiler: Host PCIe routing; Runtime: Floating-point unit logic
2. Framework: Direct transistor clocking; Compiler: Operating system page fault handling; Runtime: Mathematical differentiation
3. Compiler: Real-time sensor power regulation; Runtime: Neural network gradient backpropagation; Hardware: Python interpreter dispatch
4. Framework: Graph definition and automatic differentiation; Compiler: Graph optimization, operator fusion, and tiling; Runtime: Memory pooling, stream scheduling, and kernel dispatch; Hardware: Parallel matrix/vector execution
Answer: The correct answer is D. In the acceleration stack, the framework (e.g., PyTorch) defines the neural network computation graph and manages autograd; the compiler (e.g., TVM, XLA, TensorRT) performs graph-level optimizations, operator fusion, memory planning, and loop tiling; the runtime (e.g., CUDA runtime) manages device memory pools, asynchronous stream queues, and kernel launches; and the hardware executes instructions across specialized matrix units, vector lanes, and memory hierarchies. The other options misattribute responsibilities across the stack.

Learning Objective: Classify the functional responsibilities of frameworks, compilers, runtimes, and hardware accelerators.

← Back to Questions

Hardware Acceleration

Purpose

Hardware Specialization

Specialized computing

Parallel computing and graphics processing

Emergence of domain-specific architectures

The technology S-curve: Why architectures shift

Machine learning hardware specialization

The integration bottleneck

AI Compute Primitives

Vector operations

Matrix operations

Matrix operations in neural networks

Matrix operations hardware acceleration

Special function units

Nonlinear functions

Hardware implementation of nonlinear functions

SFU hardware implementation

Compute Units and Execution Models

Mapping primitives to execution units

Evolution from SIMD to SIMT architectures

Tensor Cores

Processing elements

N:M structured sparsity mechanics

Systolic arrays

The tiling principle: Bridging graph and silicon

Numerics in AI acceleration

Precision trade-offs

Mixed-precision computing

Intra-node interconnects: Scaling the stack

Cost-performance analysis

AI Memory Systems

Understanding the AI memory wall

Quantifying the compute-memory performance gap

Hardware balance (\(I_{\text{ridge}}\)): The paradigm partition

Memory access patterns in ML workloads

Irregular memory access

Memory hierarchy

On-chip memory

Off-chip memory

Memory bandwidth and architectural trade-offs

Host-accelerator communication

Node-level interconnect topology

Transfer optimization

Model memory pressure

Multilayer perceptrons

Convolutional neural networks

Transformer networks

Accelerator design implications

Roofline Model

Hardware ridge points

Calculating memory bandwidth bounds

Optimization by intensity regime

Hardware Mapping

Placement and allocation

Computation placement

Memory allocation

Combinatorial complexity

Ordering computation and execution

Parallelization across processing elements

Memory placement and data movement

Mapping search space

Dataflow Optimization

Building blocks of mapping strategies

Data movement patterns

Weight stationary

Output stationary

Input stationary

Memory-efficient tensor layouts

Channels-last layout

Channels-first layout

Comparing channels-last and channels-first layouts

Kernel fusion

Intermediate memory write

Kernel fusion for memory efficiency

Performance benefits and constraints

Memory-efficient tiling strategies

Tiling fundamentals

Performance benefits of tiling

Tiling methods