Machine Foundations

A datasheet reports peak throughput, and a workload almost never reaches it. Closing that gap requires knowing which physical limit binds execution first: memory bandwidth, arithmetic throughput, fixed overhead, or energy. This appendix collects the hardware models, ratios, and order-of-magnitude values used for single-node estimates in the book, so that chapter estimates can be checked against them. Architecture essentials support readers encountering memory hierarchies and hardware cost models for the first time. Familiarity with undergraduate computer architecture is helpful but not required.

How to Use This Appendix

This appendix is designed as a reference. When diagnosing performance issues, use it to translate a vague symptom (“it is slow”) into a specific constraint (“bandwidth bound at batch size one”) and then choose the lever that addresses it. Each need in table 1 leads to the section that carries the model or worked template and to the chapter that applies it to the book’s workloads.

Table 1: Where to Look: Each need maps to the section of this appendix that carries the model or worked template and to the chapter that applies it to the book’s workloads.

Need	Reference Section	Applied In
Sanity-check feasibility with orders of magnitude	section 1.1	System Balance and Hardware
Compare access times across memory, links, and storage	section 1.1.1	Memory hierarchy
Decide whether a kernel is compute or bandwidth bound	section 1.2.1	Performance Bounds
Check that a performance equation is consistent	section 1.2.2	Iron Law of ML Systems
Predict whether more accelerators shorten a run	section 1.2.3	Single-node multi-GPU training
Estimate the wall-clock time of a training run	section 1.2.4	Training-Time Law
Size request concurrency for a serving target	section 1.2.5	Little's Law
Explain why data movement dominates energy	section 1.3.1	Memory hierarchy
Separate latency-dominated from bandwidth-dominated transfers	section 1.3.2	Host-accelerator communication
Place a device class by compute, memory, and power	section 1.3.3	Choosing an Accelerator
Choose a numerical format	section 1.4.1	Numerical formats
Map floating-point values to integers	section 1.4.2	Numerical Precision: Changing How Values Are Stored

Conventions used here follow the book-wide notation (for example, \(\BatchSize\) is reserved for batch size and \(\Bandwidth\) for bandwidth). The values in this appendix are ratios and orders of magnitude, which change slowly because they follow from physics and circuit design. The dated specifications of particular accelerators, links, and storage devices change with every product generation. System Assumptions tabulates them so a reader can substitute current values without disturbing the relationships here.

Numbers to Know

In general systems engineering, Jeff Dean’s “Latency Numbers Every Programmer Should Know”¹ gave developers a shared set of order-of-magnitude anchors for reasoning about cache misses, disk seeks, and network round trips before writing code. Machine learning systems rest on a parallel hierarchy of physical ratios: fetching a 32-bit weight from off-chip DRAM costs hundreds of times more energy and roughly a thousand times more latency than reading it from a register, a gap that no compiler flag can erase. Although absolute timings and wattages shift with every process node and workload, the relative gaps across memory tiers, precision widths, and fiber links remain stable across hardware generations. Memorizing those relationships turns back-of-envelope sizing into a fast sanity check on any design.

¹ Jeff Dean: A Google Senior Fellow and one of the architects of MapReduce, BigTable, and TensorFlow. His latency numbers, originally presented with Peter Norvig around 2010, became a canonical reference for systems engineers; Colin Scott’s interactive visualization shows that the hierarchy of latencies has stayed stable across hardware generations (Scott 2012).

Scott, Colin. 2012. “Numbers Every Programmer Should Know by Year.”

Systems Perspective 1.1: Three numbers that matter most

• Energy ratio: In the cited 45 nm reference, a 32-bit DRAM read uses ~581× the energy of one FP16 multiply.
• Training-state footprint: Model weights (2 bytes FP16) + gradients (2 bytes FP16) + master weights (4 bytes FP32) + optimizer states for Adaptive Moment Estimation (Adam) at 8 bytes. That totals 16 bytes per parameter, so Llama 3 8B (8.03B parameters) needs 128.5 GB to start full training, more than one accelerator’s memory.
• Fiber propagation limit: Light travels about 200 km/ms in fiber, so propagation sets a floor on every round trip.

Reference relationships

The numbers above mix two kinds of quantities: invariant physical or arithmetic bounds, such as propagation delay in fiber and bytes per parameter, and technology-specific energy ratios that shift with each process node while preserving their ordering. The three tables below separate those reference anchors into propagation latency, memory-to-compute energy cost, and model-sizing arithmetic.

Speed of light

No network upgrade or routing optimization can deliver a cross-country round trip in 5 ms when light in fiber takes roughly 40 ms to make the journey. Each approximate round trip in table 2 carries that irreducible propagation component.

Table 2: Speed of Light Reference: Light in fiber travels ~200 km/ms. These approximate round trips include reducible overhead; only propagation is irreducible.

Distance	Round-Trip Latency	Implication
Same data center	~1 ms	Distributed training feasible
Cross-country (US)	~40 ms	Reduces the remaining request budget
Cross-Atlantic	~60 ms	Further reduces the request budget
Cross-Pacific	~100 ms	Data locality is critical

Energy hierarchy

Moving data across wires costs far more energy than computing on it once it arrives in an arithmetic unit. Raising reuse per byte therefore pays off across every hardware generation (table 3).²

² Energy hierarchy sources: The DRAM and arithmetic values come from Horowitz (2014) (45 nm). The L1 SRAM/register ratio is an illustrative hierarchy anchor, not a Horowitz measurement. The headline ratio compares a 32-bit DRAM access with a 16-bit floating-point multiply. Both the absolute values and the ratio vary with process, circuit, memory technology, and operand width; wire distance helps explain why movement remains expensive.

Horowitz, Mark. 2014. “1.1 Computing’s Energy Problem (and What We Can Do about It).” 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 10–14. https://doi.org/10.1109/isscc.2014.6757323.

Table 3: Energy Reference Ratios: The first three ratios use Horowitz’s 45 nm measurements; the L1/register ratio is an illustrative on-chip hierarchy anchor.

Relationship	Reference Ratio	Source and Interpretation
DRAM access vs. FP16 multiply	~581×	Horowitz 45 nm; wire capacitance scales with distance
FP32 vs. INT8 energy	~18×	Horowitz 45 nm; bit width determines switching energy
FP32 vs. FP16 energy	~3.4×	Horowitz 45 nm; narrower arithmetic reduces switching and datapath energy
L1 SRAM vs. register	~5×	Illustrative on-chip hierarchy anchor; distance to the arithmetic unit

Scaling rules

Memory and compute requirements for training and inference follow from a few arithmetic relationships (table 4).³ The examples use Llama 3 8B. System cost of optimizer state derives the 16 bytes per parameter of training state and why switching to 16-bit arithmetic does not shrink it.

³ Training memory (Adam): The 16 bytes/parameter rule assumes mixed-precision training with Adam. ZeRO optimization can reduce per-accelerator memory by sharding optimizer states across accelerators, but the total memory across all accelerators remains ~16\(\times\) parameters.

Table 4: Scaling Rules: These are arithmetic, not hardware-specific. Training memory includes FP16 weights (2 bytes), FP16 gradients (2 bytes), FP32 master weights (4 bytes), and Adam optimizer states (8 bytes for momentum + variance).

Rule	Formula	Example (Llama 3 8B)
Inference memory (FP16)	2 bytes\(\times\) parameters	16.1 GB
Inference memory (INT8)	1 byte\(\times\) parameters	8.0 GB
Training memory (Adam)	16 bytes\(\times\) parameters	128.5 GB
Inference FLOPs (transformer)	~2\(\times\) parameters per token	16.1 GFLOP per token
Training FLOPs	~6\(\times\) parameters\(\times\) tokens	\(4.8 \times 10^{22}\) FLOPs on 1T tokens

The latency hierarchy

Distance costs time. Each step away from the arithmetic units, from registers through on-chip cache, off-chip memory, device and host links, the network, and storage, multiplies the wait for the first byte. The largest jumps come from crossing a physical boundary (table 5). If a register access took ten seconds, a read from accelerator memory would take hours and a read from local flash storage would take weeks.

Table 5: The Latency Hierarchy: Access latency by tier, as the decade that contains it, as a multiple of a register access (rounded to one significant figure), and on a human time scale where one register access takes ten seconds. The largest gaps come from crossing physical boundaries: off-chip memory, device and host links, network fabric, and storage.

Tier	Latency (Decade)	Relative to a Register	If a Register Access Took 10 s
Register	0.1–1 ns	1\(\times\)	10 seconds
L1 cache	1–10 ns	3×	33.3 seconds
L2 cache	1–10 ns	10×	2.2 minutes
Accelerator memory (HBM)	100–1,000 ns	900×	2.5 hours
Accelerator-to-accelerator link	100–1,000 ns	2,000×	4.6 hours
Host link (PCIe)	1–10 μs	3,000×	9.3 hours
Network fabric	1–10 μs	20,000×	1.9 days
Local flash storage (NVMe)	100–1,000 μs	300,000×	38.6 days

Bandwidth falls along the same path. On the reference accelerator, high-bandwidth memory delivers about 52× the bandwidth of the host link, and a network access costs about 18× the latency of an accelerator-memory access. Data that must cross the host link or the network should cross it once and be reused many times on the device.

Illustrative latency budgets

Hardware access latencies span nanoseconds to milliseconds, but the application decides how much total delay a response can tolerate: an autonomous braking controller must act within milliseconds while a batch training job can run for days (table 6). These targets reflect safety envelopes, human perception thresholds, and service-level objectives rather than universal hardware limits. The latency budget shows how a serving system divides such an end-to-end budget among its stages.

Table 6: Illustrative Latency Targets: Production requirements depend on the application and its service-level objectives.

Application	Budget	Constraint
Autonomous braking	<10 ms	At 100 km/h, 10 ms = 28 cm of travel
Voice assistant	<100 ms	Human perception of “instant”
Web search	<200 ms	User patience threshold
Video streaming	<1 s	Buffer tolerance
Batch training	hours–days	Throughput dominates latency

Systems Perspective 1.2: A note on terminology: GPUs and accelerators

Throughout this book, “accelerator” is the general term for hardware acceleration. The principles of roofline analysis, memory hierarchies, numerical precision, and performance modeling apply to GPUs, Tensor Processing Units (TPUs), NPUs, application-specific integrated circuits, and other specialized AI accelerators. Vendor-specific features such as CUDA and NVLink are identified explicitly.

Accelerator Specifications tabulates the dated specifications for these accelerators, including the NVIDIA H100 that serves as the book’s reference machine. Those device peaks, together with the reference latencies, energy ratios, and application budgets in this section, supply the raw inputs for back-of-envelope checks, but raw specifications alone do not reveal which resource binds a workload. Turning hardware numbers into a bottleneck diagnosis requires compact analytical models that relate workload demand to machine ceilings.

Physics of Computing

An accelerator rated at hundreds of teraflops can still spend most of a training step waiting on memory reads, serial host dispatch, or in-flight request state. Raw hardware specifications (FLOP/s, bytes per second, watts) bound each resource in isolation, yet without analytical models an engineer cannot tell whether a slow kernel is compute-bound or bandwidth-bound, verify that a time equation is dimensionally consistent, predict whether doubling the accelerator count will halve training time, or size serving concurrency from arrival rate and latency. Four foundational models turn those hardware limits into quantitative checks.

The Roofline model

A kernel that achieves 1 percent of an accelerator’s peak FLOP/s may be poorly written, or it may already be saturating the memory bus at the physical limit of the chip. The Roofline Model (Williams et al. 2009) separates those two cases by bounding how fast a workload can run on a given hardware target according to whether it runs out of compute or memory bandwidth first.

Williams, Samuel, Andrew Waterman, and David Patterson. 2009. “Roofline: An Insightful Visual Performance Model for Multicore Architectures.” Communications of the ACM 52 (4): 65–76. https://doi.org/10.1145/1498765.1498785.

Every operation has an arithmetic intensity: the ratio of computations performed to bytes moved from memory. Matrix multiplication achieves high arithmetic intensity because each loaded element is reused across multiple multiply-accumulate steps. In contrast, element-wise operations like rectified linear unit (ReLU) have low intensity because each element is fetched from memory, evaluated once, and written back. Plotting achievable performance against arithmetic intensity yields the roofline (figure 1): a diagonal memory-bandwidth ceiling rises linearly from the origin until it meets the horizontal peak-compute ceiling at the ridge point. A workload’s arithmetic intensity determines which ceiling governs execution, and therefore whether optimizations must target memory traffic or arithmetic throughput.

Figure 1: The Roofline Model: The sloped memory-bandwidth ceiling and horizontal peak-compute ceiling meet at the ridge point. Plotting a workload by arithmetic intensity identifies which roofline ceiling applies.

The ridge point determines the hardware’s balance. Formally, when computation and data movement overlap completely (the overlapped form of the iron law), idealized wall-clock time is bounded below by the critical path: \[ \Time\geq \max\left( \frac{\Operations}{\PeakRate}, \frac{\DataVolume}{\Bandwidth} \right) \] Attainable throughput \(\AttainableComputeRate = \Operations/ \Time\) is therefore bounded by: \[ \AttainableComputeRate \leq \frac{\Operations}{\max\left( \frac{\Operations}{\PeakRate}, \frac{\DataVolume}{\Bandwidth} \right)} = \min\left( \PeakRate, \frac{\Operations}{\DataVolume} \times \Bandwidth \right) = \min(\PeakRate, \ArithmeticIntensity\times \Bandwidth) \] where \(\ArithmeticIntensity= \Operations/ \DataVolume\) is the arithmetic intensity measured at the DRAM interface. The ridge point \(\RooflineRidgePoint\) is the transition intensity where the compute and bandwidth ceilings meet: \[ \PeakRate = \RooflineRidgePoint \times \Bandwidth \implies \RooflineRidgePoint = \frac{\PeakRate}{\Bandwidth} \] If a workload’s arithmetic intensity falls below \(\RooflineRidgePoint\), it is bandwidth-bound (the roofline literature says memory-bound). Above that value, it is compute bound. The memory-bound regime is the bandwidth-bound archetype of Workload archetypes, whose lighthouse is GPT-2 decode. The compute-bound regime is the compute-bound archetype, whose lighthouse is ResNet-50. Performance Bounds develops both regimes and the levers that move a workload between them.

Systems Perspective 1.3: Batch size controls arithmetic intensity

For matrix multiplications, arithmetic intensity scales with the batch dimension. For \(\LayerOutputMatrix= \LayerInputMatrix\WeightMatrix\), where \(\LayerInputMatrix\) is \((\BatchSize\times \LayerInputWidth)\) and \(\WeightMatrix\) is \((\LayerInputWidth\times \LayerOutputWidth)\):

• FLOPs: \(2 \times \BatchSize\times \LayerInputWidth \times \LayerOutputWidth\) (multiply-adds).
• Bytes: Moving \(\LayerInputMatrix\), \(\WeightMatrix\), and \(\LayerOutputMatrix\) costs approximately \(\ElementStorageSize(\BatchSize\LayerInputWidth + \LayerInputWidth\LayerOutputWidth + \BatchSize\LayerOutputWidth)\) bytes.

Doubling \(\BatchSize\) doubles FLOPs while weight traffic stays nearly constant when weights dominate, increasing arithmetic intensity. This is why batching often helps inference serving (Why batching helps develops the serving side, including the latency it adds). Activation traffic, cache behavior, shape, precision, and the ridge point still determine whether it reaches the compute ceiling.

A concrete example: The H100 analysis

Consider an NVIDIA H100, the book’s reference machine, with dense FP16 Tensor Core performance of 989 TFLOP/s and HBM3 bandwidth of 3.35 TB/s. The ridge point is 989 TFLOP/s/3.35 TB/s = 295.2 FLOP/byte (the Tera prefixes cancel, yielding FLOP/byte).

Two common operations fall on opposite sides of that ridge. General matrix multiply (GEMM) on two square matrices of size 4096 by 4096 has arithmetic intensity of approximately 1,365 FLOP/byte, so 1,365 FLOP/byte > 295.2 FLOP/byte and the operation is compute bound. An element-wise ReLU on an FP16 tensor performs 1 comparison while reading 2 bytes and writing 2 bytes (4 bytes total), yielding an arithmetic intensity of only \(1/4 =\) 0.25 FLOP/byte. Because 0.25 FLOP/byte \(\ll\) 295.2 FLOP/byte, the operation is severely bandwidth-bound, achieving only about 0.08 percent of peak FLOP/s. This contrast is why frameworks fuse an element-wise operation into the preceding matrix multiply, so the intermediate result never makes a round trip through memory. The memory wall and kernel fusion develops fusion and the launch overhead it removes.

Precision moves the ridge. At INT8 the H100’s peak rate doubles to 1,979 TOPS while its memory bandwidth stays fixed, so the ridge rises to about 591 operations per byte. A convolution layer that sits above the FP16 ridge can fall below the INT8 ridge, which is why quantized ResNet-50 inference can turn bandwidth bound on the same chip (Hardware ridge points).

Dimensional analysis

In the roofline calculation, dividing peak throughput (\(\text{FLOP/s}\)) by memory bandwidth (\(\text{bytes/s}\)) cancels seconds and leaves \(\text{FLOP/byte}\). Dividing byte volume directly by \(\text{FLOP/s}\) or adding milliseconds of network delay to seconds of compute, by contrast, produces a quantity whose units do not reduce to time, turning a back-of-envelope estimate into a silent category error. Before plugging numbers into any performance equation, dimensional analysis verifies that the equation is dimensionally homogeneous, meaning every additive or equated term resolves to the same units.

Consider both levels of the iron law of ML systems (principle 3) introduced in Iron Law of ML Systems. First, in the multiplicative D·A·M form (equation 1.2), intermediate units cancel across the three factors to leave seconds per task: \[ \Time[\text{s}] = \frac{\text{Samples}}{\text{Task}} \times \frac{\text{FLOPs}}{\text{Sample}} \times \frac{\text{s}}{\text{FLOP}} = \frac{\text{s}}{\text{Task}} \]

Second, for serialized execution phases (equation 1.3), every additive term must resolve to time (seconds): \[ \Time[\text{s}] = \underbrace{ \frac{\DataVolume [\text{bytes}]}{\Bandwidth [\text{bytes/s}]} }_{\text{seconds}} + \underbrace{ \frac{\Operations[\text{FLOPs}]}{\PeakRate [\text{FLOP/s}] \cdot \EfficiencyHw [1]} }_{\text{seconds}} + \underbrace{ \Latency [\text{s}] }_{\text{seconds}} \]

• Data term: \(\frac{\text{bytes}}{\text{bytes/s}} = \text{bytes} \times \frac{\text{s}}{\text{bytes}} = \text{s}\)
• Compute term: \(\frac{\text{FLOPs}}{\text{FLOP/s}} = \text{FLOPs} \times \frac{\text{s}}{\text{FLOP}} = \text{s}\)
• Latency term: Already in seconds.

The equation is physically consistent. Apply this technique to any systems equation: if the dimensions do not match, the formula is wrong. “FLOPs” and “Bandwidth” cannot be traded directly because they have different units. Any such trade-off must convert through time, as the iron law does.

Once a single-device time equation is dimensionally sound, the next question is how its terms behave when work is partitioned across multiple processors.

Amdahl’s law and Gustafson’s law

Doubling the number of accelerators on a training job can either shorten the wall-clock time of a fixed batch or process twice as many tokens in the same time window, and serial overhead limits those two choices very differently. Amdahl’s law and Gustafson’s law formalize the two regimes according to what they hold fixed: Amdahl’s law bounds how much faster a fixed task can run (optimizing latency), whereas Gustafson’s law bounds how much more work can be completed in a fixed time budget (optimizing throughput).

Strong scaling (Amdahl’s Law)

When an engineer splits a fixed training batch or a single inference query across more accelerators to reduce step latency, only the parallelizable matrix math shrinks; host dispatch, barrier synchronization, and sequential aggregation remain fixed. Strong scaling measures how much faster such a fixed-size problem runs as processors are added, and Amdahl’s law (Amdahl 1967) states that the serial fraction of the task sets a hard ceiling on the resulting speedup.⁴ Let \(\Time(1) = 1\) be normalized single-processor execution time, composed of serial fraction \(1-\ParallelFraction\) and parallelizable fraction \(\ParallelFraction\). If \(\WorkerCount\) processors execute the parallel fraction with linear scaling and zero communication overhead, execution time on \(\WorkerCount\) processors is: \[ \Time(\WorkerCount) = (1-\ParallelFraction) + \frac{\ParallelFraction}{\WorkerCount} \] The resulting theoretical speedup is: \[ \text{Speedup}(\WorkerCount) = \frac{\Time(1)}{\Time(\WorkerCount)} = \frac{1}{(1-\ParallelFraction) + \frac{\ParallelFraction}{\WorkerCount}} \]

Amdahl, Gene M. 1967. “Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities.” Proceedings of the April 18-20, 1967, Spring Joint Computer Conference on - AFIPS ’67 (Spring), AFIPS ’67 (spring), 483–85. https://doi.org/10.1145/1465482.1465560.

⁴ Gene Amdahl (1922–2015): A legendary computer architect at IBM, where he was the chief architect of the System/360. He later founded Amdahl Corporation to compete with IBM in the mainframe market.

As \(\WorkerCount \to \infty\), the term \(\frac{\ParallelFraction}{\WorkerCount} \to 0\), and the speedup converges to \(1/(1-\ParallelFraction)\). To see Amdahl’s Law in action, suppose 5 percent of a training step is serial overhead (for example, Python global interpreter lock (GIL), kernel launch latency) and 95 percent is parallelizable matrix math:

• With \(\WorkerCount=1\), speedup is 1.
• With \(\WorkerCount=\) 8, speedup is 1/(0.05 + 0.95/8) ≈ 5.9×.
• With \(\WorkerCount \to \infty\), speedup is capped at 1/0.05 = 20×.

No matter how many accelerators are added, this fixed workload cannot run faster than 20×. The same ceiling limits a single-node optimization. End-to-end vs. component latency applies it to a measured pipeline, where an unchanged preprocessing stage caps the gain from a faster model.

Weak scaling (Gustafson’s Law)

Rather than adding accelerators to finish a small, fixed dataset in milliseconds, large-model training grows the dataset and global batch size alongside the cluster so that more work completes in the same wall-clock window. Weak scaling measures how much larger a problem can become while holding runtime fixed as processors are added.⁵ For \(\WorkerCount\) processors, let \(1-\ParallelFraction\) be the serial fraction of execution time measured on those processors for the scaled workload, and \(\ParallelFraction\) the parallel fraction. Gustafson’s law (Gustafson 1988) derives “scaled speedup” by asking how long that scaled problem would take on a single processor. On \(\WorkerCount\) processors, normalized runtime is \(\Time(\WorkerCount) = (1-\ParallelFraction) + \ParallelFraction = 1\). Executing the same total work on a single processor requires running the parallel fraction sequentially, taking \(\Time(1) = (1-\ParallelFraction) + \WorkerCount \cdot \ParallelFraction = \WorkerCount - (1-\ParallelFraction)(\WorkerCount-1)\). The resulting scaled speedup is: \[ \text{Scaled Speedup}(\WorkerCount) = \frac{\Time(1)}{\Time(\WorkerCount)} = \WorkerCount - (1-\ParallelFraction)(\WorkerCount - 1) \]

⁵ John Gustafson: A computer scientist known for his work in parallel computing and for introducing the Unum (universal number) format. His law was a direct response to the perceived “limits” of Amdahl’s Law when applied to massive scale.

Gustafson, John L. 1988. “Reevaluating Amdahl’s Law.” Communications of the ACM 31 (5): 532–33. https://doi.org/10.1145/42411.42415.

Because the parallel part grows linearly with \(\WorkerCount\) while the serial fraction \(1-\ParallelFraction\) remains fixed, weak scaling can keep efficiency high as the problem grows. Using the same 5 percent serial overhead (\(1-\ParallelFraction\) = 0.05), Gustafson’s Law tells a different story:

• With \(\WorkerCount=1\), speedup is 1.
• With \(\WorkerCount=\) 8, Scaled Speedup is 8 − 0.05 \(\times\) (7) = 8 − 0.35 = 7.65×.
• With \(\WorkerCount=\) 1000, Scaled Speedup is 1000 − 0.05 \(\times\) (999) ≈ 950×.

In weak scaling, efficiency remains high because the useful work (training the model) scales up to dwarf the fixed overheads. Single-node multi-GPU training shows how data-parallel training uses weak scaling to keep accelerators busy as the cluster grows.

The training-time equation

Weak scaling explains why large training runs grow the dataset rather than shrink the wall-clock time, but budgeting a run still requires translating billions of parameters and tokens into days on a specific cluster. The training-time equation combines model size, token count, accelerator count, peak rate, and sustained hardware utilization into that wall-clock estimate, which Training-Time Law applies to the book’s training workloads.

Napkin Math 1.1: The training time equation

Problem: How long does dense-transformer training take for a stated model size, token count, accelerator count, peak rate, and utilization?

Math: \[ \Time\approx \frac{6 \cdot \Parameters\cdot \DatasetSize}{\AcceleratorCount \cdot \PeakRate \cdot \EfficiencyHw} \]

Variables:

• Training FLOP factor: The factor 6 derives from \(2\Parameters\text{ FLOP/token}\) for the forward pass and \(4\Parameters\text{ FLOP/token}\) for the backward pass (\(2\Parameters\) for activation gradients and \(2\Parameters\) for weight gradients), yielding \(6\Parameters \DatasetSize\text{ FLOPs}\) total across \(\DatasetSize\) training tokens.
• \(\Parameters\): Number of model parameters.
• \(\DatasetSize\): Number of training tokens.
• \(\AcceleratorCount\): Number of accelerators.
• \(\PeakRate\): Peak FLOP/s of one accelerator.
• \(\EfficiencyHw\): Hardware utilization, typically 30 percent–50 percent in this training estimate.

Example: Training a 1B parameter model on 20B tokens using 1 H100 (989 TFLOP/s) at 40 percent utilization. \(\text{Total FLOPs} = 6 \times 1 \times 10^{9} \times 2 \times 10^{10} = 1.2 \times 10^{20} \text{ FLOPs}\). \(\text{Throughput} = 1 \times (9.89 \times 10^{14}) \times 0.40 \approx 3.956 \times 10^{14} \text{ FLOP/s}\). \(\Time = \frac{1.2 \times 10^{20}}{3.956 \times 10^{14}} \approx 303,336.7 \text{ seconds} \approx 5,055.6 \text{ minutes}\).

Systems insight: Under these assumptions, training takes about 3.5 days. In D·A·M terms, this equation is the macroscopic product of Data (\(\DatasetSize\) tokens), Algorithm (\(6\Parameters\text{ FLOPs/token}\)), and Machine (\((\AcceleratorCount \cdot \PeakRate \cdot \EfficiencyHw)^{-1}\text{ s/FLOP}\)). Doubling the token count doubles the time, while doubling the accelerator count or the utilization halves it.

Little’s Law

Whereas the training-time equation sizes a batch job whose total token volume is known up front, an online inference server receives a continuous stream of queries, and every in-flight request holds working state in memory until its response completes. An accelerator can therefore exhaust its memory capacity from concurrent request state before its arithmetic units saturate. For capacity planning in stable inference systems, Little’s law (Little 1961) relates long-run mean concurrency (\(\RequestConcurrency\)), mean arrival rate (\(\ArrivalRate\)), and mean time in system (\(\RequestTimeInSystem\)):⁶ \[ \RequestConcurrency = \ArrivalRate \times \RequestTimeInSystem \]

Little, John D. C. 1961. “A Proof for the Queuing Formula: \(L = \lambda W\).” Operations Research 9 (3): 383–87. https://doi.org/10.1287/opre.9.3.383.

⁶ John Little: An Institute Professor at MIT and a pioneer in the field of operations research. His law, proved in 1961, is fundamental to queuing theory and is used across fields from manufacturing to computer network analysis.

The law follows from operational conservation of requests over any observation interval \([0, \ObservationIntervalLength]\). Let \(\CumulativeArrivals(\TimeInstant)\) denote cumulative arrivals and \(\CumulativeCompletions(\TimeInstant)\) cumulative completions, so the instantaneous in-flight queue occupancy is \(\InstantaneousRequestOccupancy(\TimeInstant) = \CumulativeArrivals(\TimeInstant) - \CumulativeCompletions(\TimeInstant)\). The average number of requests in the system over \([0, \ObservationIntervalLength]\) is \(\IntervalMeanOccupancy= \frac{1}{\ObservationIntervalLength}\int_0^{\ObservationIntervalLength} \InstantaneousRequestOccupancy(\TimeInstant)\,d\TimeInstant\). The total accumulated request-time \(\int_0^{\ObservationIntervalLength} \InstantaneousRequestOccupancy(\TimeInstant)\,d\TimeInstant\) equals the sum of residence times \(\sum_{i=1}^{\CompletedRequestCount(\ObservationIntervalLength)} \RequestResidenceTime_i\) of all \(\CompletedRequestCount(\ObservationIntervalLength)\) requests completed by time \(\ObservationIntervalLength\) (plus boundary terms that vanish as \(\ObservationIntervalLength\to \infty\)). Dividing both sides by \(\ObservationIntervalLength\): \[ \IntervalMeanOccupancy= \frac{1}{\ObservationIntervalLength}\int_0^{\ObservationIntervalLength} \InstantaneousRequestOccupancy(\TimeInstant)\,d\TimeInstant= \frac{\CompletedRequestCount(\ObservationIntervalLength)}{\ObservationIntervalLength} \cdot \frac{1}{\CompletedRequestCount(\ObservationIntervalLength)}\sum_{i=1}^{\CompletedRequestCount(\ObservationIntervalLength)} \RequestResidenceTime_i \] In the limit as \(\ObservationIntervalLength\to \infty\) for a stable queue, \(\lim_{\ObservationIntervalLength\to\infty} \frac{\CompletedRequestCount(\ObservationIntervalLength)}{\ObservationIntervalLength} = \ArrivalRate\) and \(\lim_{\CompletedRequestCount\to\infty} \frac{1}{\CompletedRequestCount}\sum_{i=1}^\CompletedRequestCount \RequestResidenceTime_i = \RequestTimeInSystem\). These limits yield \(\RequestConcurrency = \ArrivalRate \times \RequestTimeInSystem\) regardless of arrival distribution or service-time distribution. Because the identity requires no Poisson-arrival or exponential-service assumption, it applies directly to heavy-tailed large language model (LLM) generation workloads so long as the queue is stationary.

To see this in practice, consider sustaining 1,000 QPS with 50 ms average latency. The law dictates that the system must support 1000 \(\times\) 0.05 s = 50 concurrent requests.

This sizes worker pools, but memory needs another assumption. If every in-system request simultaneously holds 1 GB of device state, the average population implies 50 GB; queued requests may instead stay in host memory or share batched state. If 24 device slots (\(\DeviceRequestCapacity\), the most requests the device can hold at once) each remain occupied for 50 ms, their throughput proxy is \(\DeviceRequestCapacity/\RequestTimeInSystem = 24/0.05 = 480 \text{ QPS}\). Little’s law alone does not fix a maximum. Device capacity does.

Serving systems use this relation to size request concurrency, and Little's Law pairs it with a queueing model to show how latency degrades as utilization nears saturation. Together, these physics-based models (roofline, Amdahl, Gustafson, and Little) locate where a bottleneck lies. Acting on that diagnosis requires the hardware structures that impose the bounds: the memory hierarchy, the links that move data between tiers, and the classes of hardware that package them.

Computer Architecture Essentials

A kernel can run at a small fraction of an accelerator’s advertised peak when data movement, insufficient parallelism, synchronization, launch overhead, or software inefficiency stands in the way. Physics sets the theoretical ceilings. Computer architecture defines the machinery that determines how close a real workload can get to them, across three structures: the memory hierarchy and its energy cost, the paths that move data between tiers, and the class of hardware that packages both.

The memory hierarchy

Computer systems rely on tiered storage architectures because no single physical medium simultaneously provides large storage capacity and the lowest access latency. Faster, smaller, and costlier tiers sit close to the arithmetic units; larger and cheaper tiers sit farther away (figure 2). When data movement sits on the critical execution path, staging tensors into the upper tiers avoids the orders-of-magnitude latency penalties of off-chip DRAM and persistent storage that table 5 quantifies.

Figure 2: The Memory Hierarchy: Performance depends on data proximity. Accessing HBM is roughly 900× slower than registers; accessing SSD is roughly 300,000× slower.

The memory hierarchy limits ML systems in both time and energy. Each access farther from the arithmetic units costs more energy, because the signal must drive longer, higher-capacitance wires (table 7). Memory hierarchy shows how accelerator design stages data in the upper tiers to avoid the off-chip cost.

Table 7: Energy Across the Memory Hierarchy: Energy per 32-bit access in the cited 45 nm reference technology. The DRAM value is the Horowitz measurement; the register and SRAM values are illustrative hierarchy anchors. Absolute values fall with each process node, but the ordering and the gap between on-chip and off-chip access persist.

Tier	Technology	Energy per 32-bit Access	Relative to a Register
Register	Flip-flops	0.1 pJ	1\(\times\)
L1 cache	SRAM	0.5 pJ	5×
L2 cache	SRAM	2 pJ	20×
Off-chip memory	DRAM/HBM	640 pJ	6,400×

Energy explains why data should stay high in the hierarchy. The table gives per-access costs, so the energy of a whole workload also depends on how many accesses reach each tier and how often data is reused. How long a transfer between tiers takes depends on two distinct properties of the path it crosses.

Bandwidth versus latency

Widening a data link tenfold barely changes the delivery time of a small inference request or control message, yet it cuts the transfer time of a multi-gigabyte model checkpoint nearly tenfold. Bandwidth (throughput once bytes are flowing) and latency (the fixed delay before the first byte arrives) govern those two regimes as distinct constraints. For data volume \(\DataVolume\) sent over effective bandwidth \(\Bandwidth\) with fixed path latency \(\Latency\), and assuming the latency and serialization costs do not overlap, total transfer time is: \[ \Time= \Latency + \frac{\DataVolume}{\Bandwidth} \]

The crossover point is the transfer size where the two terms are equal: \[ \LatencyBandwidthCrossoverSize = \Latency \times \Bandwidth \]

For transfers below \(\LatencyBandwidthCrossoverSize\) (for example, small request or control messages), latency dominates. For transfers above it (for example, loading weights), bandwidth dominates.

Consider sending data over a 10 Gb/s link with 10 ms base request/response latency. The crossover size is 12.5 MB, so the dominant term depends on which side of that threshold the transfer falls.

• Latency-dominated packet (1 KB): Transmission time is 1 KB \(\times\) 8/10 Gb/s ≈ 0.8 μs. Total time is approximately 10 ms + 0.8 μs ≈ 10 ms. Result: Fixed path and protocol latency dominates; propagation is one component.
• Bandwidth-dominated checkpoint (1 GB): Transmission time is \(1\text{ GB} \times 8/10\text{ Gbps} \approx 800\text{ ms}\). Total time is \(\approx 10\text{ ms} + 800\text{ ms} = 810\text{ ms}\). Result: Base latency is negligible; bandwidth is the bottleneck.

How much bandwidth, compute, and memory capacity sit on either side of those links depends on the target form factor, which ranges from liquid-cooled data center racks to milliwatt sensor boards.

Accelerator classes

A model trained on a 700 W data center accelerator cannot be dropped onto a battery-powered phone or a microcontroller with a few hundred kilobytes of SRAM without redesigning the workload around the target’s physical envelope. The deployment paradigms of Four Deployment Regimes Across Nine Orders of Magnitude run on four hardware classes whose peak throughput, memory bandwidth, memory capacity, and power budgets span orders of magnitude (table 8). Each range covers representative devices in its class rather than any single product; Master accelerator matrix tabulates the individual chips.

Table 8: Accelerator Classes: Order-of-magnitude ranges across representative devices in each class, rounded to one significant figure. Peak rate is the dense rate at each device’s native ML precision (FP16 or BF16 for data center accelerators, mostly INT8 for edge, mobile, and microcontroller parts), so compare rates within a class rather than across classes. The ridge point is peak rate divided by memory bandwidth.

Class	Peak Rate	Memory Bandwidth	Memory Capacity	Power	Ridge Point
Datacenter accelerator	100–2,000 TFLOP/s	0.9–8 TB/s	30–200 GB	300–1,000 W	100–300 FLOP/byte
Edge module	4–300 TFLOP/s	8–200 GB/s	1–70 GB	2–60 W	100–1,000 FLOP/byte
Mobile SoC	20–40 TFLOP/s	50–80 GB/s	9–10 GB	5 W	300–700 FLOP/byte
Microcontroller	60–500 MFLOP/s	60–700 MB/s	0.2–4 MB	5–400 mW	0.2–6 FLOP/byte

Two patterns stand out across these classes. First, they span roughly 7 orders of magnitude in peak rate and 5 in power, so a model sized for one class rarely runs unchanged on another. Second, ridge points sit in the hundreds of FLOP per byte from data center to mobile, which leaves the same low-intensity operations bandwidth bound on all three. A microcontroller’s ridge, by contrast, sits within a few FLOP per byte, which makes nearly every layer of a keyword-spotting model compute bound and shifts the binding constraint to the power budget. Choosing an Accelerator applies these classes to the choice of accelerator. Moving between classes also changes which numerical formats the hardware supports natively, and numerical precision directly controls how much data must move.

Numerical Representations

Halving a tensor’s representation from 32 bits to 16 bits cuts its memory footprint and bandwidth traffic in half and doubles peak Tensor Core throughput on the same silicon, yet choosing the wrong 16-bit layout can cause small gradients to underflow to zero during training. Because the numerical format fixes both how many bytes each value occupies and how much energy each arithmetic operation costs, choosing among FP32, BF16, FP16, FP8, and INT8 is a direct trade-off between statistical fidelity and hardware throughput.

Floating-point formats

A gradient of magnitude \(10^{-6}\) survives in 16-bit brain floating point (BF16) but flushes to zero in standard 16-bit IEEE floating point (FP16), even though both formats occupy two bytes. IEEE 754 standards and AI-specific floating-point representations divide a fixed bit budget between a sign bit, an exponent field that sets dynamic range (the span of representable magnitudes), and a mantissa field that sets precision (the granularity of values within that range). Operational ranges and typical uses differ by format (table 9), and the bit partitions of FP32, BF16, FP16, and INT8 show where each format spends its bits (figure 3). BF16 retains FP32’s full 8-bit exponent field, preserving the same dynamic range while truncating the mantissa to 7 bits, whereas FP16 widens the mantissa to 10 bits at the cost of a narrower 5-bit exponent.

Table 9: Numerical Format Comparison: Each format trades off precision, dynamic range, memory footprint, and compute throughput.

Format	Bits	Exponent	Mantissa	Dynamic Range	Typical Use Case
FP32	32	8	23	\(\sim 10^{-38}\) to \(10^{38}\)	Training (full precision), reference inference
FP16	16	5	10	Normal: \(6.10 \times 10^{-5}\)–\(6.55 \times 10^{4}\)	Training often with loss scaling, inference
BF16	16	8	7	FP32 normal exponent range	Training, usually without loss scaling
FP8	8	4 or 5	3 or 2	Varies	Training/inference on supported hardware
INT8	8	N/A	N/A	-128 to 127	Inference after quantization

Figure 3: Numerical Format Bit Layouts: A visual comparison of how FP32, BF16, FP16, and INT8 allocate their bits.

BF16⁷ (Wang and Kanwar 2019; Kalamkar et al. 2019) occupies the same 16-bit storage class as FP16, yet the two formats behave very differently during training. Because BF16 allocates three more bits to the exponent than FP16, nominal bit width alone does not predict whether a training run will need gradient scaling.

⁷ BF16 (brain floating point 16): Originally introduced with Google TPU v2 and later adopted by Intel, Arm, and NVIDIA, BF16 became a common training format across accelerator ecosystems. Its adoption made format support in hardware, kernels, and distributed-training libraries a portability concern rather than a vendor-specific feature.

Wang, Shibo, and Pankaj Kanwar. 2019. BFloat16: The Secret to High Performance on Cloud TPUs.

Systems Perspective 1.4: The dynamic range wall

The choice of numerical format is a direct application of the iron law of ML systems (principle 3). Reducing precision from FP32 to BF16 or FP16 halves the data volume term, potentially doubling throughput on memory-bound workloads. However, the type of 16-bit format determines the engineering complexity:

• Dynamic range (the exponent): BF16 preserves the eight-bit exponent of FP32 and therefore the same normal exponent range (Kalamkar et al. 2019).
• Precision (the mantissa): FP16 has a larger 10-bit mantissa than BF16 (7 bits), offering higher precision for values within its range. Its five-bit exponent, however, raises underflow risk. FP16 training therefore often uses loss scaling, an operational step that multiplies gradients by a large constant to keep more values representable (Micikevicius et al. 2018). Mixed-precision training develops the full mixed-precision recipe.
• Energy efficiency: INT8 operations can be substantially more efficient than floating-point equivalents because they move less data and use simpler integer arithmetic paths. Moving to INT8 for inference is a primary lever for deploying neural networks under tight memory, latency, or power budgets (Jacob et al. 2018; Krishnamoorthi 2018). Numerical formats compares the formats on accuracy cost and hardware support.

Krishnamoorthi, Raghuraman. 2018. “Quantizing Deep Convolutional Networks for Efficient Inference: A Whitepaper.” arXiv Preprint arXiv:1806.08342 abs/1806.08342.

Jacob, Benoit, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2704–13. https://doi.org/10.1109/cvpr.2018.00286.

Micikevicius, Paulius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, et al. 2018. “Mixed Precision Training.” International Conference on Learning Representations (ICLR).

Kalamkar, Dhiraj, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, et al. 2019. A Study of BFLOAT16 for Deep Learning Training.

Integer quantization

Storing weights and activations in 32-bit floating point spends four bytes per value and drives wide floating-point multipliers when 256 discrete levels often suffice for inference accuracy. Quantization cuts that cost by mapping continuous floating-point values to discrete integers, typically INT8, using either a symmetric or an asymmetric mapping to assign the floating-point range to 256 integer codes.

For symmetric quantization, let \(\SymmetricClippingThreshold= \max |\QuantizerInput|\) over the calibrated range, assume \(\QuantizerInput\) is clipped to \([-\SymmetricClippingThreshold, \SymmetricClippingThreshold]\), and set the scale \(\QuantizationScale= \SymmetricClippingThreshold/127\). The signed INT8 code is: \[ \QuantizedIntegerCode = \text{round}\left(\frac{\QuantizerInput}{\QuantizationScale}\right) \]

This symmetric mapping works well for weight distributions centered around zero, whereas asymmetric quantization handles distributions that are not centered (common after ReLU, which produces only nonnegative values) by shifting the range before scaling. Let \(\ClippingRangeLowerBound\) and \(\ClippingRangeUpperBound\) bound the calibrated range, clip \(\QuantizerInput\) to \([\ClippingRangeLowerBound, \ClippingRangeUpperBound]\), and set the scale \(\QuantizationScale= (\ClippingRangeUpperBound- \ClippingRangeLowerBound)/255\). A common unsigned 8-bit mapping is: \[ \QuantizedIntegerCode = \text{round}\left(\frac{\QuantizerInput- \ClippingRangeLowerBound}{\QuantizationScale}\right) \]

The choice between symmetric and asymmetric quantization depends on the tensor’s distribution and has measurable accuracy implications. Numerical Precision: Changing How Values Are Stored develops calibration, granularity, and the accuracy cost of each mapping on the book’s workloads. Together with the floating-point formats, integer quantization completes the machine-level levers that determine how many bytes each operation moves and how much energy each arithmetic step consumes.

Summary

Machine-side diagnosis begins with units, ceilings, and working sets. The roofline model and the iron law identify which ceiling binds, and Amdahl’s and Gustafson’s laws, the memory hierarchy, the accelerator classes, and the numerical formats explain the gap between a measurement and that ceiling.

Treat peak specifications as ceilings, not predictions. Compare a measurement with the relevant ceiling, then trace the remaining gap to utilization, data movement, runtime overhead, architecture, or software. A back-of-envelope estimate is useful when it states its assumptions and units and makes the next measurement obvious. Chip specifications age with every product generation; the relationships here outlast them.