Notation

To a machine learning researcher, $\Glyph{B}$ denotes batch size, while to a network engineer, $\Glyph{B}$ denotes bandwidth. In distributed systems, $\Glyph{N}$ represents the count of worker accelerators, whereas in statistics, $\Glyph{N}$ denotes sample count. In high-performance computing, $\Glyph{P}$ is commonly used for processor count, but in deep learning, $\Glyph{P}$ is reserved for model parameter count. Similarly, in this book’s communication models, $\Glyph{\alpha}$ denotes per-message network latency and $\Glyph{\beta}$ denotes link bandwidth, whereas in statistical learning, $\Glyph{\alpha}$ represents a power-law scaling exponent and $\Glyph{\beta}$ represents optimizer momentum.

These collisions arise because machine learning at scale combines three fields that developed their mathematical conventions independently: machine learning, distributed systems, and high-performance data center networking. Training and serving modern foundation models requires orchestrating workloads across thousands of accelerators connected by high-speed networks, which places symbols from all three fields into the same equation.

This book establishes a unified mathematical notation for distributed machine learning systems. The conventions defined below eliminate ambiguity across scaling laws, collective communication algorithms, parallel execution schedules, and cluster reliability models.

The Governing Equations of Distributed Performance

A distributed training step or serving request spends time both inside each accelerator and across the network that connects them, so performance modeling follows a two-tier hierarchy: the single-device execution foundation and its fleet-wide composition. The subsections that follow state each governing equation and define the symbols that appear in its terms.

The single-device foundation (the iron law of ML systems)

Every distributed training step or inference pass builds on local execution on individual accelerators. For serialized execution phases on a single accelerator, the iron law of ML systems decomposes execution time into memory access, arithmetic computation, and latency overhead:

\[\Time = \underbrace{\frac{\DataVolume}{\Bandwidth}}_{\text{Memory Time}} + \underbrace{\frac{\Operations}{\PeakRate \cdot \EfficiencyHw}}_{\text{Compute Time}} + \underbrace{\Latency}_{\text{Latency Overhead}}\]

Each symbol was chosen deliberately to preserve machine learning conventions while maintaining architectural precision. The table below records the definition, physical unit, and collision-avoidance rationale for each term:

Symbol	Definition	Unit	Notes
$\Time$	Execution Time	seconds	Total serialized execution time in the iron-law performance decomposition.
$\Bandwidth$	Bandwidth	bytes/s	Avoids collision with $\BatchSize$ (Batch Size). Systems literature often uses $\Glyph{B}$ for bandwidth, while ML literature often uses $\Glyph{B}$ for batch size. The notation preserves the ML convention.
$\DataVolume$	Data Volume	bytes	Avoids collision with $\DatasetSize$ (Dataset Size). In scaling laws, $\Glyph{D}$ means training tokens. Here we need bytes moved through memory. The subscript disambiguates.
$\EfficiencyHw$	Efficiency	—	Hardware utilization $(0 \le \EfficiencyHw \le 1)$. Avoids collision with learning rate $(\LearningRate)$.
$\Latency$	Latency	seconds	Avoids collision with $\Glyph{L}$ (Loss). Fixed overhead time (kernel launch, network RTT). The subscript distinguishes from the loss function.
$\Operations$	Operations	FLOPs	Total floating-point operations. Clean in equations (vs. “$\text{Ops}$”).
$\PeakRate$	Peak Rate	FLOP/s	Avoids collision with $\Parameters$ (Parameters). Roofline presentations often use $\Glyph{P}$ for performance, while ML literature commonly uses $\Glyph{P}$ for parameter count. The notation preserves the ML convention.
$\AsymptoticTightBound$	Asymptotic Tight Bound	Operator	Asymptotic tight bound: grows at the same rate as its argument, up to constant factors.
$\BatchDataVolume$	Batch Data Volume	bytes	Bytes of input data one batch carries to the accelerator.
$\ClockFrequency$	Clock Frequency	hertz	Rate at which a processor’s clock ticks, setting how many switching cycles occur per second.
$\FiniteChange$	Finite Change	Operator	Finite change in the quantity it prefixes, between two states, configurations or points in time.
$\MicroUnitPrefix$	Micro Unit Prefix	Operator	SI unit prefix micro, one millionth, written before a unit such as seconds.
$\NumericBitWidth$	Numeric Bit Width	Integer	Number of bits used to represent one numeric value, such as an arithmetic operand or a quantized weight or activation.
$\PipelinedExecutionTime$	Pipelined Execution Time	seconds	Execution time of a workload whose data movement and computation overlap, set by the slower of the two plus fixed latency.
$\SequentialExecutionTime$	Sequential Execution Time	seconds	Execution time when data movement, computation and fixed overhead run one after another with no overlap, the sum of the iron-law terms.
$\StepIoTime$	Step I/O Time	seconds	Time the input pipeline needs to read, decompress, decode and stage the data for one training step.
$\StorageBandwidth$	Storage Bandwidth	bytes/s	Sustained rate at which the storage system delivers bytes to the input pipeline.
$\TrainingStepTime$	Training Step Time	seconds	Wall-clock time of one training step, bounded below by the slower of compute and input I/O.
$\UnitAliasMultiplier$	Unit Alias Multiplier	Dimensionless	Number of base units in one unit alias, the conversion multiplier of a unit-conventions table row.

Why these choices matter

Without careful notation, statements about distributed systems become ambiguous. Consider a typical unsubscripted claim:

“Reducing $\Glyph{D}$ improves performance.”

In distributed machine learning, this could mean reducing the dataset size (fewer training samples), shrinking the data-parallel degree $\DataParallelDegree$, or reducing the data volume $\DataVolume$ moved across the memory bus through quantization. With our notation, we distinguish them precisely:

“Reducing $\DataVolume$ through FP8 quantization cuts memory traffic across the interconnect while $\DatasetSize$ (training tokens) and $\DataParallelDegree$ remain unchanged.”

Subscripted variants

The root symbol names the fundamental physical quantity, while an upright subscript specifies the precise domain or measurement point. Three recurring families illustrate this convention across the volume:

$\Glyph{\text{BW}_{\text{memory}}}$, $\NetworkBandwidth$, $\Glyph{\text{BW}_{\text{interconnect}}}$: bandwidth across distinct tiers of the distributed memory and fabric hierarchy.
$\DataVolume$, $\PeakRate$, $\Latency$, $\EfficiencyHw$: single-node iron-law terms, held clear of shared ML symbols.
$\EnergyPerByteMoved$, $\EnergyPerOperation$, $\TotalEnergy$: energy budget components across distributed compute nodes.

The subscript therefore carries part of the claim. Memory and network bandwidth differ by orders of magnitude, so a figure is reproducible only when the subscript says where it was measured.

The Fleet Law (distributed step time)

When scaling a model across a fleet of $\DeviceCount= \AcceleratorCount$ accelerators, single-device compute time is distributed across workers, while communication and synchronization overheads are introduced. Under partitioned execution of fixed work, the governing equation is the Fleet Law (the distributed step-time model, equation 1.3):

\[\DistributedStepTime(\DeviceCount) = \frac{\IronLawComputeTime}{\DeviceCount} + \CommunicationTime(\DeviceCount) + \SynchronizationTime(\DeviceCount) - \OverlappedTime\]

Symbol	Definition	Unit	Notes
$\DeviceCount, \AcceleratorCount$	Number of Devices	Integer	Accelerator count in a distributed job, the core’s $\AcceleratorCount$ ($\DeviceCount= \AcceleratorCount$); also the rank count of a collective. (Not parameters, use $\Parameters$.)
$\AcceleratorsPerNode$, $\ClusterNodeCount$	Accelerators per Node, Node Count	Integer	Hierarchical collectives split the fleet as $\DeviceCount= \AcceleratorsPerNode \ClusterNodeCount$. (Bare $\Glyph{G}$ is the LogGP gap per byte.)
$\DistributedStepTime(\DeviceCount)$	Distributed Step Time	seconds	Wall-clock time for one training step at scale $\DeviceCount$.
$\IronLawComputeTime$	Single-Device Compute Time	seconds	The core’s compute term for one training step (forward + backward pass) on one device.
$\CommunicationTime(\DeviceCount)$	Communication Time	seconds	Time for collective operations (AllReduce, AllGather); depends on payload, topology, and $\DeviceCount$.
$\OverlappedTime$	Overlapped Time	seconds	Communication hidden behind computation. Reduces effective overhead.
$\SynchronizationTime(\DeviceCount)$	Synchronization Time	seconds	Total nonoverlapped synchronization or coordination cost per step at scale $\DeviceCount$.
$\CommunicationComputeRatio$	Communication-Computation Ratio	Dimensionless	$\CommunicationComputeRatio= \CommunicationTime(\DeviceCount)/(\IronLawComputeTime/\DeviceCount)$. Values near or above 1 indicate communication-bound scaling.
$\CommunicationIntensity$	Communication Intensity	bytes/FLOP	Network bytes moved per local FLOP, $\CommunicationIntensity = \CollectivePayloadSize/\Operations$; the fleet analogue of arithmetic intensity $\ArithmeticIntensity$.
$\FleetCommunicationTaxonomyAxis$	Communication Axis	Set	Communication axis of the C$^3$ taxonomy, data movement across the network fabric, viewed as the set of techniques it covers so that axis intersections name zones of the taxonomy.
$\FleetComputeTaxonomyAxis$	Compute Axis	Set	Compute axis of the C$^3$ taxonomy, local arithmetic and memory within a node, viewed as the set of techniques it covers so that axis intersections name zones of the taxonomy.
$\FleetCoordinationTaxonomyAxis$	Coordination Axis	Set	Coordination axis of the C$^3$ taxonomy, synchronization, failure recovery and scheduling across the fleet, viewed as the set of techniques it covers so that axis intersections name zones of the taxonomy.
$\OperationalTaxonomyDimension$	Operational Dimension	Record	One of the three operational dimensions of the C$^3$ taxonomy, indexed 1 to 3 as compute, communication and coordination.

Scaling laws of foundation models

Scaling laws anchor the book’s account of why machine learning moved onto fleets. The single-resource power law and the dense-transformer compute rule are:

\[\LossFunction(\ScaledResource) = \ScaleCoefficient \ScaledResource^{-\ScalingExponent} + \IrreducibleLoss, \qquad \TrainingCompute \approx 6\Parameters \DatasetSize\]

Symbol	Definition	Unit	Notes
$\ScaledResource$	Scaled Resource	varies	Training compute, parameter count $\Parameters$, or training tokens $\DatasetSize$, depending on the fitted law.
$\ScalingExponent$	Scaling Exponent	Dimensionless	Fitted power-law exponent; valid only within the regime in which it was measured. (Not network latency $\MessageStartupLatency$.)
$\ScaleCoefficient$	Scale Coefficient	loss units	Task-dependent fitted constant.
$\IrreducibleLoss$	Irreducible Loss	loss units	Loss floor that added resources do not remove within the fitted regime.
$\TrainingCompute$	Training Compute	FLOPs	About $6\Parameters \DatasetSize$ for a dense transformer: $2\Parameters$ per token forward and $4\Parameters$ backward.
$\BaseBatchSize$	Base Batch Size	Integer	Reference batch size at which the base learning rate was tuned.
$\BaseLearningRate$	Base Learning Rate	Dimensionless	Learning rate tuned at the reference batch size before linear scaling.
$\EpochCount$	Epoch Count	Integer	Number of full passes a training run makes over its dataset.
$\ScaledBatchSize$	Scaled Batch Size	Integer	Batch size a training recipe is scaled to under the linear scaling rule.
$\ScaledLearningRate$	Scaled Learning Rate	Dimensionless	Learning rate used at a new batch size under the linear scaling rule.
$\ScalingLawLoss$	Scaling Law Loss	Function	Loss a trained model reaches as a function of the resource scaled, such as parameters, training tokens or compute, in an empirical power-law scaling law.

The distributed energy corollary

At data center scale, energy and power delivery are primary system constraints. For $\DeviceCount$ devices that each move the same $\DataVolume$ bytes and execute the same $\Operations$ FLOPs over a workload interval, effective per-byte data-movement costs and per-operation compute costs give the following fleet energy model. Here the work and traffic terms are per-device quantities, so multiplying by $\DeviceCount$ sums the device contributions:

\[\TotalEnergy \approx \DeviceCount\times \left( \DataVolume \times \EnergyPerByteMoved + \Operations \times \EnergyPerOperation \right)\]

Symbol	Definition	Unit	Notes
$\EnergyPerByteMoved$	Energy per Byte Moved	joules/byte	Effective movement cost; total energy also depends on bytes moved and operations executed.
$\EnergyPerOperation$	Energy per Operation	joules/FLOP	Energy cost of a single arithmetic operation.
$\TotalEnergy$	Total Energy	joules	Total energy consumed by an ML workload, decomposed into data-movement and compute terms.
$\AcceleratorSupplyCurrent$	Accelerator Supply Current	amperes	Electrical current an accelerator draws from its power supply.
$\ActiveInferencePower$	Active Inference Power	watts	Average power an accelerator draws while actively running inference.
$\AveragePower$	Average Power	watts	Power drawn averaged over a stated period.
$\CloudOffloadEnergy$	Cloud Offload Energy	joules	Energy a device spends transmitting an input to the cloud for remote inference.
$\CoolantHeatLoad$	Coolant Heat Load	watts	Heat per unit time that a liquid cooling loop must carry away from a rack, the load that sets its coolant mass flow.
$\CoolantMassFlowRate$	Coolant Mass Flow Rate	kg/s	Mass of coolant, liquid or air, passing through a cooling loop per second.
$\CoolantTemperature$	Coolant Temperature	kelvin	Temperature of the liquid or air coolant in a cooling loop, whose finite change is its rise from inlet to outlet.
$\DevicePower$	Device Power	watts	Electrical power one device draws.
$\DynamicPower$	Dynamic Power	watts	Power a chip draws switching its transistors, proportional to switched capacitance, supply voltage squared and clock frequency.
$\ElectricalResistance$	Electrical Resistance	ohms	Electrical resistance of a circuit or winding.
$\FacilityHeatRemovalCapacity$	Facility Heat Removal Capacity	watts	Largest rate at which an installed facility and cooling configuration can remove heat.
$\FacilityPower$	Facility Power	watts	Total electrical power a data center facility draws from the grid, its IT load plus cooling, power conversion, lighting and other overhead.
$\GeneralPurposeEnergyProductivity$	General-Purpose Energy Productivity	FLOP/J	Useful floating-point operations general-purpose silicon delivers per joule on a given workload.
$\InferenceDutyCycle$	Inference Duty Cycle	Dimensionless	Fraction of time an accelerator actively runs inference rather than waiting in standby.
$\InformationTechnologyHeatLoad$	IT Heat Load	watts	Rate at which a facility’s information technology equipment (servers, accelerators, network gear) produces heat.
$\InformationTechnologyPower$	IT Power	watts	Electrical power delivered to a data center’s IT equipment: servers, accelerators, storage and networking.
$\LeakageCurrent$	Leakage Current	amperes	Current that leaks through a chip’s transistors when they are not switching.
$\LeakagePower$	Leakage Power	watts	Static power a chip dissipates through leakage current regardless of switching.
$\LocalInferenceEnergy$	Local Inference Energy	joules	Energy a device spends running one inference on its own hardware.
$\NetworkEnergy$	Network Energy	joules	Energy the network fabric, its switches, links and adapters, consumes over a stated interval.
$\ParasiticLoopInductance$	Parasitic Loop Inductance	henries	Unintended inductance of the current loop that feeds an accelerator’s core rail, which turns current slew into voltage droop.
$\SpecializedEnergyProductivity$	Specialized Energy Productivity	FLOP/J	Useful floating-point operations specialized, domain-specific silicon delivers per joule on a workload suited to its design.
$\SpecificHeatCapacity$	Specific Heat Capacity	J/(kg K)	Heat needed to raise one kilogram of a coolant by one kelvin at constant pressure.
$\StandbyPower$	Standby Power	watts	Power a device draws in standby, when it is not running inference.
$\SupplyVoltage$	Supply Voltage	volts	Supply voltage that powers a digital circuit, the quantity dynamic voltage and frequency scaling lowers.
$\SwitchedCapacitance$	Switched Capacitance	farads	Capacitance of the wires and gates a circuit charges when it switches.
$\SwitchingActivityFactor$	Switching Activity Factor	Dimensionless	Average fraction of a chip’s gate capacitance that switches each clock cycle.
$\TotalChipPower$	Total Chip Power	watts	Total power a chip dissipates, the sum of dynamic switching power and static leakage power.

Network Communication Models and Collectives

Collective communications (AllReduce, AllGather, ReduceScatter, AllToAll) move payloads across the network fabric, following ring or tree schedules over topologies such as the fat tree. The models below price those transfers by separating fixed per-message latency from bandwidth-proportional transmission time.

The communication model

In the standard first-order model, network communication time decomposes into fixed message startup latency and bandwidth-dependent transfer. For a single point-to-point message, transfer time is the sum of those two terms:

\[\PointToPointTransferTime(\PointToPointMessageSize) = \MessageStartupLatency+ \frac{\PointToPointMessageSize}{\LinkBandwidth}\]

The model keeps its literature’s letters as aliases of core concepts: $\MessageStartupLatency$ is the per-message $\Latency$ and $\LinkBandwidth$ is a link’s $\Bandwidth$, written in $\MessageStartupLatency$-$\LinkBandwidth$ form. Inside $\MessageStartupLatency$-$\LinkBandwidth$ formulas, write $\Glyph{\alpha}$ and $\Glyph{\beta}$; elsewhere, write $\Latency$ and a subscripted $\Bandwidth$.

Symbol	Definition	Unit	Notes
$\PointToPointTransferTime(\PointToPointMessageSize)$	Point-to-Point Transfer Time	seconds	Wall-clock time to transmit one message of size $\PointToPointMessageSize$ across one link.
$\PointToPointMessageSize$	Point-to-Point Message Size	bytes	Size of one message in $\PointToPointTransferTime(\PointToPointMessageSize)$.
$\MessageStartupLatency$	Per-Message Latency	seconds	The core’s $\Latency$ for one message, in $\MessageStartupLatency$-$\LinkBandwidth$ form: the fixed startup cost. (Not significance level or Rényi order, which carry subscripts.)
$\LinkBandwidth$	Link Bandwidth	bytes/s	The core’s $\Bandwidth$ for one link ($\SingleLinkBandwidth$), in $\MessageStartupLatency$-$\LinkBandwidth$ form: effective throughput per link.
$\LatencyBandwidthCrossoverSize$	Crossover Point	bytes	$\LatencyBandwidthCrossoverSize= \MessageStartupLatency\cdot \LinkBandwidth$. Below $\LatencyBandwidthCrossoverSize$: latency-bound. Above: bandwidth-bound.
$\CollectivePayloadSize$	Collective Payload Size	bytes	Per-rank gradient, model-state, activation, or compressed payload that a collective moves. Not a memory footprint (those are $\TotalMemoryFootprint$, $\WeightMemory$, and so on) and not the core’s GEMM dimension $\GemmRowDimension$, which appears only in GEMM shape statements.
$\CollectiveTensorSize$	Collective Tensor Size	bytes	Full tensor acted on by Broadcast, Reduce, or AllReduce, reconstructed by AllGather, or partitioned by ReduceScatter. An equally sharded AllGather starts with $\CollectiveTensorSize/\DeviceCount$ bytes per rank; AllToAll uses the per-rank payload $\CollectivePayloadSize$.
$\RingAllReduceCompletionTime$, $\TreeAllReduceCompletionTime$	Collective Completion Time	seconds	AllReduce time under the ring or tree algorithm in $\MessageStartupLatency$-$\LinkBandwidth$ form, for example $\RingAllReduceCompletionTime = 2(\DeviceCount-1)\MessageStartupLatency+ 2\frac{\DeviceCount-1}{\DeviceCount}\frac{\CollectivePayloadSize}{\LinkBandwidth}$. $\AllReduceCompletionTime$ names an AllReduce whose algorithm the text leaves open.
$\RingTreeCrossoverPayload$	Ring-Tree Crossover Payload	bytes	Payload at which $\RingAllReduceCompletionTime = \TreeAllReduceCompletionTime$; approximately $\DeviceCount\MessageStartupLatency\LinkBandwidth/\log_2 \DeviceCount$ for large $\DeviceCount$.
$\AllReduceCompletionTime$	AllReduce Completion Time	seconds	Time to complete one AllReduce collective over a group of devices, whatever algorithm implements it.
$\AllToAllCompletionTime$	AllToAll Completion Time	seconds	Time to complete one AllToAll collective, in which every device exchanges a distinct share of its payload with every other device.
$\BisectionBandwidth$	Bisection Bandwidth	bytes/s	Total bandwidth across the narrowest cut that divides a network’s nodes into two equal halves.
$\LocalRankIndex$	Local Rank Index	Integer	Position of an accelerator within its node, its local rank, which in a rail-optimized fabric also names the rail it attaches to.
$\NetworkBandwidth$	Network Bandwidth	bytes/s	Sustained rate at which a network path delivers bytes between machines.
$\NetworkTrafficVolume$	Network Traffic Volume	bytes	Bytes one worker moves across the network over a stated interval, such as one training step.
$\RankCountBinaryExponent$	Rank Count Binary Exponent	Dimensionless	Integer exponent for which the rank count equals two raised to it, the condition algorithms such as butterfly AllReduce require.
$\RankIndex$	Rank Index	Integer	Position of one rank, a device or worker, within a collective communication group, counted from zero.
$\SingleLinkBandwidth$	Single Link Bandwidth	bytes/s	Bandwidth of one network link in one direction, written in the core bandwidth notation outside alpha-beta formulas.
$\SwitchRadix$	Switch Radix	Integer	Number of ports on one network switch, its radix.
$\TopologyEndpointCount$	Topology Endpoint Count	Integer	Number of endpoints, accelerators or hosts, that one network topology connects, the size variable of its scaling relations.

The LogGP extension

LogGP refines the $\MessageStartupLatency$-$\LinkBandwidth$ model by separating network transit latency, processor initiation overhead, and long-message injection gaps. The book uses descriptive subscripts for the highest-collision terms; the bare letters $\Glyph{g}$ and $\Glyph{G}$ keep their LogGP meanings inside this model only.

Symbol	Definition	Unit	Notes
$\LoggpLongMessageTime(\PointToPointMessageSize)$	LogGP Long-Message Time	seconds	Transfer time for an $\PointToPointMessageSize$-byte long message. Reuses $\PointToPointMessageSize$ from the $\MessageStartupLatency$-$\LinkBandwidth$ model.
$\LogpNetworkLatency$	LogP/LogGP Network Latency	seconds	Network transit time of one message, the part of the core’s $\Latency$ that excludes processor overhead, so $\MessageStartupLatency\approx \SenderOverhead + \LogpNetworkLatency + \ReceiverOverhead$. The subscript avoids bare $\Glyph{L}$ collisions.
$\SenderOverhead$	Sender Overhead	seconds	Processor/NIC overhead to initiate a message.
$\ReceiverOverhead$	Receiver Overhead	seconds	Processor/NIC overhead to complete a message. Symmetric LogP writes $\SymmetricMessageOverhead= \SenderOverhead = \ReceiverOverhead$, so one message costs $2\SymmetricMessageOverhead$.
$\MessageInjectionGap$	Message Injection Gap	seconds	Minimum interval between consecutive message injections. (Gradients are bold, $\Glyph{\mathbf{g}}$.)
$\LongMessageGapPerByte$	Long-Message Gap per Byte	seconds/byte	Per-byte long-message gap; distinct from $\LinkBandwidth$ link bandwidth. (Accelerators per node are $\AcceleratorsPerNode$.)
$\SymmetricMessageOverhead$	Symmetric Message Overhead	seconds	Processor time spent initiating or completing one message in the symmetric LogP model, where sender and receiver overheads are equal, during which the processor cannot compute.

Compression cost and savings

Gradient and activation compression trade local computation for reduced network traffic. The resulting models compare codec overhead against the communication time saved by shrinking the payload:

Symbol	Definition	Unit	Notes
$\TransferTimeForPayload$	Transfer Time for Payload	seconds	Communication time for an $\Glyph{x}$-byte payload: $\PointToPointTransferTime(\Glyph{x})$ for one link, or the relevant collective time.
$\CompressedCommunicationTime$	Compressed Communication Time	seconds	Total time after encode, transfer of $\CollectivePayloadSize/\CompressionRatio$, and decode.
$\CodecEncodeTime$	Codec Encode Time	seconds	Local compression overhead before transfer.
$\CodecDecodeTime$	Codec Decode Time	seconds	Local decompression overhead after transfer.
$\QuantizationBlockSize$	Quantization Block Size	Integer	Number of tensor elements in one block of block-wise quantization, all sharing one scale.
$\QuantizationScale$	Quantization Scale	Varies	Real-valued step between adjacent integer levels of a uniform quantizer.
$\RetainedGradientElementCount$	Retained Gradient Element Count	Integer	Number of largest-magnitude gradient elements top-k sparsification keeps and transmits.

Parallelism Dimensions and Scaling Efficiency

Scaling large foundation models requires combining multiple parallelism strategies across device dimensions. The notation below tracks how devices are partitioned across those dimensions and how efficiently the combined schedule converts added hardware into speedup.

Parallelism dimensions

A single parallelism strategy rarely scales across an entire cluster without hitting memory or bandwidth limits. To stay within both bounds, 3D parallelism partitions the training workload across three orthogonal dimensions:

\[\DeviceCount= \DataParallelDegree \times \PipelineParallelDegree \times \TensorParallelDegree\]

Symbol	Definition	Unit	Notes
$\DataParallelDegree$	Data-Parallel Degree	Integer	Number of model replicas. (Bare $\Glyph{d}$ stays the core’s hidden dimension.)
$\PipelineParallelDegree$	Pipeline-Parallel Degree	Integer	Number of physical pipeline stages in model-depth partitioning.
$\TensorParallelDegree$	Tensor-Parallel Degree	Integer	Degree of intra-layer partitioning (model width). (Bare $\Glyph{t}$ stays time or a step index.)
$\MicrobatchCount$	Microbatch Count	Integer	Number of microbatches used to fill and drain the pipeline.
$\PipelineBubbleFraction$	Pipeline Bubble Fraction	Dimensionless	Fraction of pipeline GPU-time spent idle, $\PipelineBubbleFraction = (\PipelineParallelDegree-1)/(\PipelineParallelDegree-1+\MicrobatchCount)$; avoids bare $\Glyph{b}$.
$\VirtualPipelineStagesPerDevice$	Virtual Pipeline Stages per Device	Integer	Number of interleaved virtual chunks assigned to each physical pipeline stage. (The attention value matrix is bold, $\AttentionValuesMatrix$.)
$\GradientStaleness$	Gradient Staleness	steps	Number of parameter updates between the parameters a gradient was computed on and the parameters it updates; $\GradientStaleness = 0$ is synchronous training.
$\AccumulationStepCount$	Accumulation Step Count	Integer	Number of micro-batches whose gradients are accumulated before one optimizer update.
$\ClientIndex$	Client Index	Integer	Position of one client among the clients of a federated round or secure aggregation.
$\ExpertCapacityFactor$	Expert Capacity Factor	Dimensionless	Multiplier on an expert’s uniform share of token assignments that sets how many assignments it may accept per step.
$\ExpertCount$	Expert Count	Integer	Number of expert subnetworks in one mixture-of-experts layer.
$\FederatedRoundCount$	Federated Round Count	Integer	Number of communication rounds a federated training run performs, or needs to reach a quality target.
$\GlobalBatchSize$	Global Batch Size	Integer	Number of examples one synchronous optimizer update averages over across all data-parallel workers.
$\LocalGradient$	Local Gradient	Vector	Gradient vector one data-parallel worker computes from its local batch in one step, its contribution to the gradient collective.
$\MicrobatchSize$	Microbatch Size	Integer	Number of examples in one micro-batch, the unit run through a single forward and backward pass.
$\ParticipatingClientCount$	Participating Client Count	Integer	Number of clients whose updates one federated round aggregates.
$\RoutedExpertCount$	Routed Expert Count	Integer	Number of experts a mixture-of-experts router sends each token to.
$\RouterGateProbability$	Router Gate Probability	Dimensionless	Probability a mixture-of-experts router assigns to the expert it selects for a token.

Scaling efficiency

Adding accelerators to a job increases communication and synchronization overhead alongside raw compute capacity. Scaling efficiency measures the resulting realized speedup relative to ideal linear scaling:

\[\ScalingEfficiency = \frac{\SingleDeviceRunTime}{\DeviceCount\times \MultiDeviceRunTime}\]

Symbol	Definition	Unit	Notes
$\ScalingEfficiency$	Scaling Efficiency	Dimensionless	Ratio of measured speedup to ideal linear speedup. $1.0$ is the ideal linear baseline, not a hard upper bound.
$\SingleDeviceRunTime$	Single-Device Time	seconds	Baseline wall-clock time for the same productive work on one device.
$\MultiDeviceRunTime$	N-Device Time	seconds	Wall-clock time for the same productive work on $\DeviceCount$ devices; excludes operational downtime counted by $\RuntimeGoodput$.
$\SerialFraction$	Serial Fraction	Dimensionless	Share of a fixed workload’s single-device execution time that does not shrink when the work is spread across devices, the serial term of Amdahl’s law.

Cluster Reliability, Fault Tolerance, and Checkpointing

At cluster scale, hardware component failures are standard operating conditions rather than rare anomalies. The symbols in this section quantify failure rates, checkpoint intervals, recovery overheads, and their net effect on fleet goodput.

Fault tolerance and reliability

When a job requires many components to remain healthy simultaneously, individual failure hazards compound across the fleet. Assuming independent components with exponential failure distributions, system reliability across $\RequiredComponentCount$ components is:

\[\SystemReliability(\TimeInstant) = e^{-\RequiredComponentCount\ComponentFailureRate \TimeInstant}\]

The aggregate exponent uses the system-level rate $\RequiredComponentCount\ComponentFailureRate$, where $\ComponentFailureRate$ is the per-component failure rate under the independent, identical-component assumption. The exponent $\RequiredComponentCount\ComponentFailureRate \TimeInstant$ is dimensionless, so the elapsed time $\TimeInstant$ must use the same time unit as $\ComponentFailureRate$ (hours, when $\ComponentFailureRate$ comes from the FIT-derived rates below).

Checkpointing too frequently wastes wall time writing state to storage, whereas checkpointing too rarely loses long stretches of training work when a node fails. The optimal checkpoint interval balances that I/O overhead against recomputation rework:

\[\OptimalCheckpointInterval = \sqrt{2 \cdot \CheckpointWriteTime \cdot \SystemMtbf}\]

The formula uses the system-wide mean time between failures (the failure interval for the whole job), not a component lifetime; substituting the per-component value would overestimate $\OptimalCheckpointInterval$ by $\sqrt{\RequiredComponentCount}$. For dimensional consistency, $\CheckpointWriteTime$ and $\SystemMtbf$ must be in the same time unit. Industry-standard $\MeanTimeBetweenFailures$ is reported in hours; to combine it with seconds-scale write times, multiply the hour value by 3600. The result $\OptimalCheckpointInterval$ then carries the time unit chosen for the inputs.

Symbol	Definition	Unit	Notes
$\SystemReliability(\TimeInstant)$	Reliability Function	Probability	Probability of no failure across the whole system before time $\TimeInstant$. One component has $\ComponentReliability(\TimeInstant) = e^{-\ComponentFailureRate \TimeInstant}$.
$\ComponentFailureRate$	Per-Component Failure Rate	failures/hour	Per-component failure rate, written $\Glyph{\lambda_{\text{fail}}}$ wherever another rate shares the passage; the system-wide rate is $\RequiredComponentCount\ComponentFailureRate$ under independent, identical components. The industry-standard reporting unit is FIT (failures per $10^9$ device-hours): $\Glyph{\lambda_{\text{hour}}}= \text{FIT} \times 10^{-9}$. (Other rates carry subscripts, such as the core’s arrival rate $\ArrivalRate$.)
$\ComponentAvailability$	Component Availability	Probability	Probability that one component is up at a given instant. A job that needs all $\RequiredComponentCount$ components at once has availability $(\ComponentAvailability)^{\RequiredComponentCount}$ under independent, identical components.
$\RequiredComponentCount$	Required Component Count	Integer	Components (accelerators, hosts, NICs, switches) the job needs up at the same time. Not $\DeviceCount$, which counts accelerators only.
$\MeanTimeBetweenFailures$	Mean Time Between Failures	hours	Average time between consecutive failures of a repairable, composed system. $\SystemMtbf = 1/(\RequiredComponentCount\ComponentFailureRate) = \ComponentMttf / \RequiredComponentCount$.
$\SystemMtbf$	System MTBF	hours	Mean time between failures that interrupt the whole job, equal to the per-component $\MeanTimeToFailure$ divided by $\RequiredComponentCount$ under independent, identical components. It is the input to the Young-Daly interval. Same time unit as $\CheckpointWriteTime$ when combined with it.
$\MeanTimeToFailure$	Mean Time To Failure	hours	Per-component lifetime constant derived from FIT rates, $\ComponentMttf = 1/\ComponentFailureRate$. Distinct from system-level $\MeanTimeBetweenFailures$ when repair/replacement cycles are modeled.
$\MeanTimeToRepair$	Mean Time To Repair	hours	Average recovery time after a failure.
$\CheckpointInterval$	Checkpoint Interval	seconds	Useful compute time between consecutive checkpoints.
$\OptimalCheckpointInterval$	Optimal Checkpoint Interval	seconds	The $\CheckpointInterval$ that minimizes total wasted time; Young-Daly formula.
$\CheckpointWriteTime$	Checkpoint Write Time	seconds	Time to persist model state to storage.
$\CheckpointLoadTime$	Checkpoint Load Time	seconds	Time to read the latest checkpoint state from storage back into accelerator memory during recovery.
$\ComponentMttf$	Component Mean Time to Failure	hours	Mean time to failure of one component, the reciprocal of its constant failure rate.
$\ComponentReliability$	Component Reliability	Dimensionless	Probability that one component of a system runs without failure up to a given time, its survival function.
$\ErasureDataFragmentCount$	Erasure Data Fragment Count	Integer	Number of data fragments an erasure code splits an object into, any that many of the stored fragments recovering the original.
$\ErasureParityFragmentCount$	Erasure Parity Fragment Count	Integer	Number of parity fragments an erasure code adds to the data fragments of an object.
$\FailureDetectionTime$	Failure Detection Time	seconds	Time from a hardware or process fault until the system classifies it as a failure.
$\FailureRecoveryTime$	Failure Recovery Time	seconds	Time from a failure until useful training work resumes: detection, restart, checkpoint load and warmup.
$\GpuMttf$	GPU Mean Time to Failure	hours	Mean time to failure of one GPU, its average operating time before it fails.
$\JobAvailability$	Job Availability	Dimensionless	Probability that every component a job needs is up at the same instant.
$\JobRestartTime$	Job Restart Time	seconds	Time for the scheduler to allocate resources and launch replacement processes after a failure.
$\LostWorkReplayTime$	Lost Work Replay Time	seconds	Time to recompute the training steps lost since the last checkpoint after a restart.
$\RedundantReplicaCount$	Redundant Replica Count	Integer	Number of independent replicas of a service in a redundancy availability calculation, any one of which can serve the load.
$\RedundantServiceAvailability$	Redundant Service Availability	Dimensionless	Probability that at least one of a service’s independent replicas is up at a randomly chosen instant.
$\ReplicaAvailability$	Replica Availability	Dimensionless	Probability that one serving replica is up at a randomly chosen instant.
$\SeriesPsuCount$	Series Power Supply Count	Integer	Number of power supply units in one node whose failure alone stops the node.
$\SteadyStateAvailability$	Steady-State Availability	Dimensionless	Probability that a component, replica or repairable system is up at a randomly chosen instant, its long-run fraction of up time.

Fleet capacity factors and goodput

Fleet capacity metrics compound single-device arithmetic efficiency, scaling efficiency, and operational losses. ML productivity goodput multiplies the scheduling, runtime, and program factors that Fleet Efficiency defines.

Symbol	Definition	Unit	Notes
$\UsefulComputeFraction$	Useful Compute-Time Fraction	Dimensionless	$(\IronLawComputeTime/\DeviceCount)/\DistributedStepTime(\DeviceCount)$. Measures how much step time is useful local arithmetic after distribution.
$\OperationalOverheadFraction$	Operational Overhead Fraction	Dimensionless	Fraction of allocated wall time lost outside the productive steps measured by $\MultiDeviceRunTime$, such as checkpoint pauses, failure recovery, scheduler preemption, or maintenance.
$\EffectiveFleetThroughput$	Effective Fleet Throughput	FLOP/s	$\EffectiveFleetThroughput = (\DeviceCount\PeakRate) \times \ModelFlopsUtilization \times \ScalingEfficiency \times \RuntimeGoodput$, where $\ModelFlopsUtilization$ is the single-device baseline, $\ScalingEfficiency$ captures all in-step scaling loss, and $\RuntimeGoodput$ captures only out-of-step operational loss excluded from $\MultiDeviceRunTime$.
$\UsefulModelFlops$	Useful Model FLOPs	FLOPs	Model FLOPs that advance training, excluding recomputation and padding; $\ModelFlopsUtilization = \UsefulModelFlops/(\DeviceCount\PeakRate \DistributedStepTime)$ measures fleet MFU and already includes in-step scaling loss. The throughput factorization uses single-device baseline MFU instead.
$\MlProductivityGoodput$, $\ProgramGoodput$, $\RuntimeGoodput$, $\SchedulingGoodput$	ML Productivity Goodput and Its Factors	Dimensionless	Program, runtime, and scheduling goodput, with $\MlProductivityGoodput = \SchedulingGoodput \times \RuntimeGoodput \times \ProgramGoodput$. For a job that holds its devices, $\EffectiveFleetThroughput$ equals peak capacity times $\RuntimeGoodput \times \ProgramGoodput$, so $\RuntimeGoodput$ is also the operational factor of that product; $\MlProductivityGoodput$ adds the scheduling factor.
$\FleetEnergyProductivity$	Fleet Energy Productivity	FLOP/J	Useful computation per unit energy, $\UsefulModelFlops$ divided by the energy drawn inside the stated boundary.
$\TrainingCost$, $\InferenceCost$	Dollar Cost	dollars	The subscript names the cost component. Carbon quantities use $\text{CO}_{2,\dots}$, never $\Glyph{C}$.
$\AllocatedWallTime$	Allocated Wall Time	seconds	Total wall-clock time for which a training job holds its allocated resources, productive or not.
$\JobDuration$	Job Duration	seconds	Wall-clock time one job or run spends executing, from start to finish.
$\UsefulTrainingTime$	Useful Training Time	seconds	Wall-clock time a training job spends on committed, productive steps that contribute to the final model.

Distributed Memory Hierarchy and Serving at Scale

Managing distributed memory requires partitioning weights, activations, optimizer states, and KV-cache tensors across nodes and tiers. At the same time, serving fleets must bound queueing delay and batching overhead under variable request arrivals.

Distributed memory and serving metrics

Reusable performance and serving quantities follow the same collision-avoidance rule as the iron law. Rates use $\Glyph{R}$ or descriptive Greek symbols; request counts use $\RequestConcurrency$ rather than overloading $\Glyph{N}$; queue utilization uses a subscripted $\Glyph{\rho}$ so bare $\Glyph{\rho}$ remains available for other ratio models.

Symbol	Definition	Unit	Notes
$\ActivationMemory$	Activation Memory	bytes	Retained activations for the backward pass or serving intermediates.
$\ArrivalRate$	Arrival Rate	requests/s	Request arrival rate. The subscript keeps bare $\Glyph{\lambda}$ free for fitted sensitivities.
$\AttainableComputeRate$	Attainable Compute Rate	FLOP/s	Roofline bound: $\AttainableComputeRate \leq \min(\PeakRate, \ArithmeticIntensity\times \Bandwidth)$. Uses $\Glyph{R}$, not $\Glyph{T}$, because this quantity is a rate.
$\BatchServiceTime(\BatchSize)$	Batch Service Time	seconds	Time to serve a batch of size $\BatchSize$.
$\CompressionRatio$	Compression Ratio	Dimensionless	Uncompressed size divided by compressed size. A compressed payload has $1/\CompressionRatio$ of the uncompressed size; the subscript avoids bare $\Glyph{C}$ collisions.
$\EffectiveServiceRate(\BatchSize)$	Effective Service Rate	requests/s	Batched service rate, typically $\EffectiveServiceRate(\BatchSize) = \BatchSize/\BatchServiceTime(\BatchSize)$. The unbatched service rate is $\ServiceRate= \EffectiveServiceRate(1)$.
$\ElementStorageSize$	Element Storage Size	bytes/element	Bytes per stored tensor element.
$\GradientMemory$	Gradient Memory	bytes	Memory occupied by stored gradients.
$\ModelFlopsUtilization$	Model FLOPs Utilization	Dimensionless	Useful model FLOP/s divided by available peak FLOP/s. Text acronym avoids overloading $\Glyph{\eta}$.
$\OptimizerStateMemory$	Optimizer-State Memory	bytes	Momentum, variance, master weights, and related optimizer buffers.
$\ReplicaCount$	Replica Count	Integer	Serving replicas among which a router chooses (for example, probing two of $\ReplicaCount$ and sending to the less loaded). Distinct from $\DeviceCount$, the devices in one job.
$\RequestConcurrency$	Request Concurrency	requests	Average in-flight requests in a stable serving system ($\RequestConcurrency = \ArrivalRate \cdot \RequestTimeInSystem$ via Little’s Law). Avoids overloading $\Glyph{N}$, which counts devices, elements, or samples elsewhere.
$\RequestTimeInSystem$	Request Time in System	seconds	End-to-end queueing/serving latency for Little’s Law. Distinct from $\Latency$, the fixed-latency term in the iron law.
$\ArithmeticIntensity$	Arithmetic Intensity	FLOP/byte	Workload FLOPs per byte moved. The Roofline Model uses $\ArithmeticIntensity$ as the independent variable.
$\RooflineRidgePoint$	Roofline Ridge Point	FLOP/byte	$\RooflineRidgePoint = \PeakRate/\Bandwidth$. Prefer this explicit form over starred shorthand so the meaning remains clear in prose.
$\ServingUtilization$	Serving Utilization	Dimensionless	Queue/server utilization. The subscript keeps bare $\Glyph{\rho}$ free for other ratio models.
$\TotalMemoryFootprint$	Total Memory Footprint	bytes	Sum of explicit memory components; avoids bare $\Glyph{M}$ ambiguity; the core’s quantity.
$\WeightMemory$	Weight Memory	bytes	Memory occupied by model weights ($\Parameters\times \text{bytes-per-parameter}$); the core’s quantity.
$\SustainedMemoryBandwidth$	Sustained Memory Bandwidth	bytes/s	Empirically achievable memory bandwidth under target access patterns.
$\AcceleratorDataConsumptionRate$	Accelerator Data Consumption Rate	bytes/s	Rate at which one accelerator consumes training input data while computing at full speed.
$\BatchArrivalRate$	Batch Arrival Rate	batches/s	Rate at which batches arrive at a server when each batch is treated as one super-request, the request arrival rate divided by the batch size.
$\BatchFillArrivalCount$	Batch Fill Arrival Count	Integer	Number of arrivals after a batch’s triggering request needed to fill the batch, one less than the maximum batch size.
$\BatchFixedOverheadTime$	Batch Fixed Overhead Time	seconds	Part of a batch’s service time paid once per batch whatever its size, such as weight loading and kernel launch.
$\BatchFormationDelay$	Batch Formation Delay	seconds	Time a request waits for its batch to fill before the batch is dispatched.
$\BatchIndex$	Batch Index	Integer	Position of one batch in the sequence of batches a data loader or runtime delivers.
$\BatchingWindow$	Batching Window	seconds	Length of the batching window, the longest a dynamic batcher holds requests before dispatching a batch.
$\DailyQueryVolume$	Daily Query Volume	queries/day	Number of queries, requests or transactions a deployed model serves per day.
$\DecodeArithmeticIntensity$	Decode Arithmetic Intensity	FLOP/byte	Arithmetic intensity of the weight-dominated part of an LLM decode step, operations per byte of weights and activations moved, as a function of batch size.
$\DeviceMemoryCapacity$	Device Memory Capacity	bytes	Total memory a device provides for a model’s weights and activations.
$\DraftTokenAcceptanceProbability$	Draft Token Acceptance Probability	Dimensionless	Probability that the target model accepts one drafted token in speculative decoding, taken as independent across draft positions.
$\EndToEndRequestLatency$	End-to-End Request Latency	seconds	Time from a request leaving the client to its response returning, summed over every stage on the request path.
$\ErrorBudgetConsumption$	Error Budget Consumption	Dimensionless	Share of one reliability target’s error budget a service has used: the fraction of eligible observations that violate the target divided by the violating fraction the target allows.
$\InferenceLatency$	Inference Latency	seconds	Time the model takes to compute a request’s output.
$\LatencySlo$	Latency Service-Level Objective	seconds	Latency target a serving system’s service-level objective sets for a request.
$\MarginalRequestServiceTime$	Marginal Request Service Time	seconds	Additional batch service time that one more request in the batch adds.
$\MaxBatchSize$	Max Batch Size	Integer	Largest batch a dynamic batcher forms, the size at which it dispatches without waiting for the window to close.
$\MinimumStableBatchSize$	Minimum Stable Batch Size	Integer	Batch size at which a batched server’s effective service rate equals the arrival rate, the boundary that stable batch sizes must exceed.
$\NetworkLatency$	Network Latency	seconds	Time a request and its response spend crossing the network, including propagation, routing and protocol handling.
$\PrefetchBufferCount$	Prefetch Buffer Count	Integer	Number of buffers holding batches loaded ahead of use.
$\QueueServerCount$	Queue Server Count	Integer	Number of identical servers in a multi-server queue with random arrivals and service.
$\QueueSystemCapacity$	Queue System Capacity	Integer	Largest number of requests a queueing system can hold at once, waiting plus in service, beyond which arrivals are rejected.
$\QueueWaitTime$	Queue Wait Time	seconds	Mean time a request waits in a queue before its service begins, excluding service time.
$\QueueingLatency$	Queueing Latency	seconds	Time a request waits in queues before it is served.
$\RealizedArrivalCount$	Realized Arrival Count	Integer	One possible value of the number of requests that arrive after the trigger in a batching window, used as a summation variable.
$\ReliabilityTargetIndex$	Reliability Target Index	Integer	Position of one reliability target, such as availability, latency, freshness, or model quality, among the targets an evaluation contract budgets.
$\RequestResponseTime$	Request Response Time	seconds	Time one request spends in a serving system, the random variable whose distribution gives tail latency.
$\ResponseTimeStandardDeviation$	Response Time Standard Deviation	seconds	Standard deviation of the time a request spends in a serving system.
$\ServiceRate$	Service Rate	items/s	Rate at which one server, such as a worker, consumer, or model endpoint, completes items or calls.
$\SpeculativeDraftLength$	Speculative Draft Length	Integer	Number of tokens a draft model proposes for the target model to verify in one speculative-decoding round.
$\SubsequentArrivalTime$	Subsequent Arrival Time	seconds	Time from the request that opened a batching window until a later arrival, indexed by arrival order.
$\TailRequestLatency$	Tail Request Latency (99th Percentile)	seconds	99th-percentile time a request spends in a serving system.
$\TargetServingUtilization$	Target Serving Utilization	Dimensionless	Serving utilization a system is sized not to exceed, chosen below one to keep latency headroom.
$\TargetUtilizationBatchSize$	Target Utilization Batch Size	Integer	Smallest batch size whose serving utilization stays at or below the target utilization, under the linear batch service model.
$\TriggeredWindowArrivalCount$	Triggered Window Arrival Count	Integer	Number of requests that arrive after the request that opened a batching window and before the window closes.

Serving memory and data storage

At scale, serving is bounded by the memory its key-value cache occupies, and the data behind training is bounded by what it costs to keep. These symbols size both.

Symbol	Definition	Unit	Notes
$\KvCacheMemory$	KV-Cache Memory	bytes	Total key-value cache footprint for autoregressive serving, $\KvCacheMemory = \BatchSize\cdot \SequenceLength\cdot \KvCacheBytesPerToken$.
$\KvCacheBytesPerToken$	KV-Cache Bytes per Token	bytes/token	$\KvCacheBytesPerToken = 2 \LayerCount\KvHeadCount \AttentionHeadDimension \ElementStorageSize$ (keys and values across all layers).
$\DataStorageCost$	Data Storage Cost	dollars	Cost of keeping a training dataset in storage over its retention period.
$\EntityCount$	Entity Count	Integer	Number of entities, such as users or items, a recommendation table or feature store holds rows for.
$\StorageRetentionTime$	Storage Retention Time	months	Length of time a dataset is kept in storage.
$\StoredDataVolume$	Stored Data Volume	bytes	Bytes a retained dataset occupies in storage.
$\TotalDataCost$	Total Data Cost	dollars	Total cost of a training dataset over its lifecycle: acquisition, labeling, storage and processing.

Deep Learning Foundations in Distributed Contexts

Distributed algorithms partition neural architectures defined by standard deep learning conventions (Goodfellow et al. 2016):

Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press.

Symbol	Definition	Unit	Notes
$\HiddenDimension$	Hidden Dimension	Integer	Size of the hidden state vector. (Parallelism degrees carry subscripts, such as $\DataParallelDegree$.)
$\AttentionHeadDimension$	Attention Head Dimension	Integer	Per-head hidden dimension in attention layers.
$\LayerInputWidth$	Layer Input Width	Integer	Number of input features in a layer.
$\LayerOutputWidth$	Layer Output Width	Integer	Number of output features in a layer.
$\ModelParameters$	Model Parameters	Vector	Trainable model weight and bias parameter vector $\ModelParameters$.
$\Policy$	Policy	Function	Target neural network policy $\pi_\ModelParameters(\Glyph{a}\mid \Glyph{s})$.
$\SequenceLength$	Sequence Length	Integer	Input context or sequence length $\SequenceLength$ in tokens.
$\ell$	Layer Index	Integer	Index for a layer. Use instead of bare $\Glyph{L}$ when indexing layers, since $\Glyph{L}$ collides with loss and latency. A numeric subscript marks a norm instead ($\ell_1$, $\ell_2$).
$\LossFunction$	Loss Function	Scalar	The objective function minimized during training.
$\AttentionHeadCount, \KvHeadCount$	Number of Attention Heads	Integer	Number of attention heads in a multi-head attention layer, and key-value heads $\KvHeadCount$ (in GQA/MQA).
$\LayerCount$	Number of Layers	Integer	Total number of layers in a network.
$\ProbabilityFunction(\ModelInput)$	Distribution	Probability mass/density	Use lowercase $\Glyph{p}(\cdot)$ for generic distributions to avoid collision with $\Parameters$ (parameter count).
$\ProbabilityFunction(\ExampleLabel\mid \ModelInput)$	Conditional Distribution	Conditional probability/density	Use this form for generic label relationships.
$\Pr(\cdot)$	Event Probability	Probability	Probability of an event $\Glyph{E}$, as in $\Pr(\text{batch}=0)$; use $\ProbabilityFunction(\ModelInput)$ and $\ProbabilityFunction(\ExampleLabel\mid \ModelInput)$ for distributions.
$\AdamFirstMoment$	Adam First Moment	Varies	Per-parameter exponential moving average of gradients that Adam keeps, its first moment.
$\AdamSecondMoment$	Adam Second Moment	Varies	Per-parameter exponential moving average of squared gradients that Adam keeps, its second moment.
$\AdapterDownProjectionMatrix$	Adapter Down Projection Matrix	Matrix	First low-rank adapter matrix, which projects a layer’s input down to the adapter rank, the LoRA down-projection.
$\AdapterRank$	Adapter Rank	Integer	Rank of a low-rank adapter update, equivalently the bottleneck width of a residual adapter, which sets its trainable parameter count.
$\AdapterUpProjectionMatrix$	Adapter Up Projection Matrix	Matrix	Second low-rank adapter matrix, which projects the rank-sized intermediate up to the layer’s output width, the LoRA up-projection.
$\AttentionKeyMatrix$	Attention Key Matrix	Matrix	Matrix of key vectors, one row per token, in an attention layer.
$\AttentionQueryMatrix$	Attention Query Matrix	Matrix	Matrix of query vectors, one row per token, in an attention layer.
$\AttentionScoreEntry$	Attention Score Entry	Dimensionless	Attention score of query i against key j, one entry of the score matrix.
$\AttentionScoreMatrix$	Attention Score Matrix	Matrix	Matrix of attention scores, the scaled dot products of every query with every key before the softmax.
$\AttentionValuesMatrix$	Attention Values Matrix	Matrix	Matrix of value vectors, one row per token, that attention weights combine into the output.
$\BiasVector$	Bias Vector	Vector	Vector of learned biases one layer adds to its weighted sums, one per neuron or output channel.
$\ClassCount$	Class Count	Integer	Number of classes a classifier distinguishes, or vocabulary entries it scores, the length of its softmax output.
$\ClassIndex$	Class Index	Integer	Index of an output class.
$\EncoderLatentRepresentation$	Encoder Latent Representation	Vector	Intermediate representation a shared encoder computes from an input, consumed by the heads that sit on top of it.
$\GaussianDistribution$	Gaussian Distribution	Distribution	Gaussian (normal) distribution with the mean and covariance given as its arguments.
$\GemmLeftOperand$	GEMM Left Operand	Matrix	Left input matrix of a general matrix multiply, whose rows match the output and whose columns match the right operand’s rows.
$\GemmOutputMatrix$	GEMM Output Matrix	Matrix	Output matrix of a general matrix multiply, with the left operand’s row count and the right operand’s column count, which the operation updates in place.
$\GemmRightOperand$	GEMM Right Operand	Matrix	Right input matrix of a general matrix multiply, whose rows match the left operand’s columns.
$\GemmRowDimension$	GEMM Row Dimension	Integer	Number of rows of the left operand and of the output in a matrix multiply.
$\HiddenActivation$	Hidden Activation	Vector	Activation vector a hidden layer outputs and passes to the next layer.
$\HiddenUnitOutput$	Hidden Unit Output	Dimensionless	Output of one hidden unit of a small network.
$\IdentityMatrix$	Identity Matrix	Matrix	Square matrix with ones on the diagonal and zeros elsewhere.
$\KeyVector$	Key Vector	Vector	Key vector of one token, one row of the key matrix.
$\KvTileIndex$	Key-Value Tile Index	Integer	Index of one key-value tile in tiled attention.
$\LabelCategoryIndex$	Label Category Index	Integer	Position of one label category among a classifier’s output classes.
$\LayerIndex$	Layer Index	Integer	Position of one layer within a network, counted from the input.
$\LayerInputMatrix$	Layer Input Matrix	Matrix	Input to a layer in matrix form, one row per example or token, or a single-channel image.
$\LayerInputVector$	Layer Input Vector	Vector	Vector of activations one layer receives as input.
$\LayerOutputMatrix$	Layer Output Matrix	Matrix	Output matrix of a layer’s matrix multiplication of the input matrix by the weight matrix, one row per example.
$\LayerOutputVector$	Layer Output Vector	Vector	Vector of activations one layer produces.
$\LossGradient$	Gradient	Varies	Gradient of the training loss with respect to one model parameter, as an optimizer update uses it.
$\LossHessian$	Loss Hessian	Matrix	Matrix of second derivatives of the loss with respect to every pair of model parameters.
$\MatrixMultiplyDestination$	Multiply-Accumulate Destination	Matrix	Destination matrix of a matrix multiply-accumulate instruction, which receives the product plus the accumulator input.
$\MatrixRowIndex$	Matrix Row Index	Integer	Position of a row within a matrix.
$\ModelPredictor$	Model Predictor	Function	Trained model viewed as a function that maps an input to its prediction.
$\ModelScore$	Model Score	Dimensionless	Score a classifier assigns to one individual, read as the predicted probability of the positive outcome.
$\ModelWeight$	Model Weight	Dimensionless	Value of one learnable weight of a model, such as the weight on one input of a neuron.
$\NetworkOutput$	Network Output	Dimensionless	Scalar output of a neuron or of a small network.
$\OptimalModelParameters$	Optimal Model Parameters	Vector	Model parameters that minimize the training loss for a given architecture.
$\OptimizerStepIndex$	Optimizer Step Index	Integer	Position of one optimizer update in the sequence of training steps.
$\PredictedClassProbability$	Predicted Class Probability	Dimensionless	Probability a classifier assigns to one label category for a given input.
$\PredictedOutput$	Predicted Output	Vector	Model output for one example, such as its vector of class probabilities.
$\QueryComponent$	Query Component	Dimensionless	One component of one token’s query vector.
$\QueryVector$	Query Vector	Vector	Query vector of one token, one row of the query matrix.
$\RawPipelineInput$	Raw Pipeline Input	Record	Raw input record before feature transformation, as received by the training or serving pipeline.
$\ServingTransformFunction$	Serving Transform	Function	Transformation the serving pipeline applies to a raw input to produce model features.
$\SquareGemmDimension$	Square GEMM Dimension	Integer	Side length of the square matrices in a matrix multiply, the shared row, column and inner dimension.
$\StandardNormalCumulativeDistribution$	Standard Normal Cumulative Distribution	Function	Cumulative distribution function of the standard normal distribution.
$\TopClassCount$	Top Class Count	Integer	Number of highest-scoring classes a top-k prediction returns or checks for the correct label.
$\TrainingTransformFunction$	Training Transform	Function	Transformation the training pipeline applies to a raw input to produce model features.
$\WeightMatrix$	Weight Matrix	Matrix	Matrix of learned weights one layer multiplies its input by, or the kernel tensor of a convolution layer.
$\WeightMatrixColumnCount$	Weight Matrix Column Count	Integer	Number of columns of a layer’s weight matrix.
$\WeightMatrixRowCount$	Weight Matrix Row Count	Integer	Number of rows of a layer’s weight matrix.
$\WeightPenaltyStrength$	Weight Penalty Strength	Dimensionless	Coefficient that scales an absolute-value or squared penalty on weight magnitude added to the training loss, equivalent to weight decay under plain gradient descent.
$\ZeroVector$	Zero Vector	Vector	Vector whose entries are all zero, such as the mean of zero-centered Gaussian noise.

Latin matrices and vectors are set in bold ($\Glyph{\mathbf{W}}$, $\Glyph{\mathbf{x}}$); scalars, dimensions, indices, and individual matrix entries stay italic ($\Glyph{N}$, $\Glyph{d}$, $\Glyph{i}$, $\Glyph{W}_{ij}$). The generic parameter symbol $\Glyph{\theta}$ is the conventional exception and stays italic. Matrix operands in general matrix multiply (GEMM) are bold ($\GemmOutputMatrix= \Glyph{\alpha}\GemmLeftOperand\GemmRightOperand+ \Glyph{\beta}\GemmOutputMatrix$), distinct from the scalar batch size $\BatchSize$. GEMM dimensions are $\GemmRowDimension$, $\Glyph{N}$, and $\Glyph{K}$. Asymptotic complexity is written $\mathcal{O}(\cdot)$, distinct from total operations $\Operations$.

Privacy, Robustness, Governance, and Distributed Constants

Federated systems, distributed coordination, and cluster operations add the statistical, privacy, and governance quantities collected below. These symbols standardize how the later chapters express differential privacy budgets, adversarial perturbation radii, drift statistics, and operational deadlines.

Privacy, robustness, and statistical testing

Part IV uses the standard symbols of differential privacy and adversarial robustness, and the operations and robustness chapters use those of hypothesis testing. The same Greek letters appear in several of these fields, so every use other than the one named below carries a subscript.

Symbol	Definition	Unit	Notes
$(\PrivacyBudget, \PrivacyAdditiveSlack)$	Differential Privacy Budget	Dimensionless	$\PrivacyBudget$ bounds the privacy loss from adding or removing one protected record; $\PrivacyAdditiveSlack$ is the allowed failure probability.
$\AdversarialPerturbationBudget$	Adversarial Perturbation Budget	input units	Radius of the allowed input perturbation under a stated norm. (Not the privacy budget $\PrivacyBudget$.)
$\AttackStepsPgd$	Attack Steps	Integer	Inner iterations of a projected gradient descent (PGD) attack; adversarial training costs about $\AttackStepsPgd+1$ passes per batch.
$\HypothesisTestSignificanceLevel$, $\HypothesisTestFalseNegativeRate$	Type I and Type II Error Rates	Probability	Significance level and miss rate of a hypothesis test; statistical power is $1-\HypothesisTestFalseNegativeRate$.
$\mathbb{E}[\cdot]$	Expectation	—	Expected value. Blackboard bold keeps it distinct from an event $\Glyph{E}$ and the energy terms such as $\TotalEnergy$.
$\AdversarialExample$	Adversarial Example	Record	Adversarial example, an input an attacker forms by adding a bounded perturbation to a clean input so the model mispredicts.
$\AdversarialPerturbation$	Adversarial Perturbation	Vector	Perturbation an adversary adds to an input to change a model’s prediction.
$\AnonymitySetSize$	Anonymity Set Size	Integer	Smallest number of records that share every quasi-identifier value in an anonymized release.
$\AttributionFeatureCount$	Attribution Feature Count	Integer	Number of input features a feature-attribution method assigns contributions to.
$\ClassificationThreshold$	Classification Threshold	Dimensionless	Score threshold above which a classifier outputs the positive class.
$\ConfidenceCriticalQuantile$	Confidence Critical Quantile	Dimensionless	Positive standard normal critical value for a two-sided test at the chosen confidence level.
$\ConformalMiscoverageRate$	Conformal Miscoverage Rate	Dimensionless	Allowed probability that the conformal prediction set misses the true referent.
$\ConformalPredictionSet$	Conformal Prediction Set	Set	Calibrated set of referent candidates that cannot be excluded at the miscoverage rate.
$\CurrentBinProportion$	Current Bin Proportion	Dimensionless	Proportion of current production observations falling in one bin of the population stability index.
$\CurrentDistribution$	Current Distribution	Distribution	Data distribution a deployed model sees in service at time t.
$\DriftTestSampleSize$	Drift Test Sample Size	Integer	Number of observations in one of the two samples a two-sample drift test compares.
$\EnvironmentResponseFunction$	Environment Response Function	Function	Response of the environment to a deployed model, mapping the current data distribution and the model’s decisions to the next data distribution.
$\FairnessGapTolerance$	Fairness Gap Tolerance	Dimensionless	Largest allowed gap between protected groups in the fairness metric a release gate or monitor enforces.
$\FeatureRecordTime$	Feature Record Time	seconds	Time at which a feature record became available, which a point-in-time join requires to be no later than the observation timestamp.
$\FirstComparedGroupLabel$	First Compared Group Label	Record	First of two group labels compared in a group-fairness condition, one value of the protected group attribute.
$\JensenShannonDivergence$	Jensen-Shannon Divergence	Function	Jensen-Shannon divergence between two probability distributions, a symmetric bounded measure of how far they differ.
$\JensenShannonMixtureDistribution$	Jensen-Shannon Mixture Distribution	Distribution	Equal-weight average of the current and training distributions, the reference against which Jensen-Shannon divergence measures each.
$\KlDivergence$	KL Divergence	Function	Kullback-Leibler divergence, the expected extra code length from encoding one distribution with a code built for another.
$\KolmogorovSmirnovStatistic$	Kolmogorov Smirnov Statistic	Dimensionless	Two-sample Kolmogorov-Smirnov statistic, the largest absolute difference between two empirical cumulative distribution functions.
$\McnemarTestStatistic$	McNemar Test Statistic	Dimensionless	McNemar test statistic, the squared difference of the two discordant counts divided by their sum.
$\MechanismOutputSet$	Mechanism Output Set	Set	Set of possible outputs of a randomized mechanism over which a differential privacy bound must hold.
$\MinimumDetectableEffect$	Minimum Detectable Effect	Dimensionless	Smallest absolute difference in the outcome metric that an A/B test is designed to detect.
$\MonitoredFeatureLevel$	Monitored Feature Level	Varies	One value of a monitored feature, the argument at which its distribution or empirical cumulative distribution function is evaluated.
$\MutualInformation$	Mutual Information	Function	Mutual information between two random variables, how much observing one reduces uncertainty about the other.
$\NeighboringDataset$	Neighboring Dataset	Record	Dataset input to a randomized mechanism in the differential privacy definition, compared against a copy that differs in one record.
$\ObservableModelOutput$	Observable Model Output	Record	Model output an adversary can observe, such as a prediction, score or generated text.
$\ObservedSignificanceLevel$	Observed Significance Level	Dimensionless	P-value of a hypothesis test, the probability under the null hypothesis of a result at least as extreme as the one observed.
$\PairwiseMaskSecret$	Pairwise Mask Secret	Vector	Shared random mask vector two clients derive from a common secret, which one adds and the other subtracts so that it cancels in the secure-aggregation sum.
$\PopulationStabilityBinCount$	Population Stability Bin Count	Integer	Number of fixed bins over which the population stability index sums.
$\PowerCriticalQuantile$	Power Critical Quantile	Dimensionless	Positive standard normal critical value corresponding to the chosen statistical power of a test.
$\PredictedDecision$	Predicted Decision	Dimensionless	Binary decision a classifier outputs for a randomly drawn individual, one for a positive prediction.
$\PrivacyNoiseMultiplier$	Privacy Noise Multiplier	Dimensionless	Ratio of the Gaussian noise standard deviation to the gradient clipping norm in differentially private training.
$\ProportionTestStatistic$	Proportion Test Statistic	Dimensionless	Test statistic of a two-proportion z-test, the difference of two sample proportions divided by its pooled standard error.
$\ProtectedGroupMembership$	Protected Group Membership	Record	Protected group a randomly drawn individual belongs to, as a random variable over group labels.
$\ProtectedTrainingRecord$	Protected Training Record	Record	Training record of one protected individual, the unit whose leakage a privacy claim bounds.
$\RandomizedMechanism$	Randomized Mechanism	Function	Randomized algorithm that maps a dataset to a released output, the object a differential privacy guarantee constrains.
$\ReferenceBinIndex$	Reference Bin Index	Integer	Position of one reference bin among the bins a continuous feature is discretized into for the population stability index.
$\ReferenceBinProportion$	Reference Bin Proportion	Dimensionless	Proportion of reference (training-time) observations falling in one bin of the population stability index.
$\RenyiOrder$	Rényi Order	Dimensionless	Order of the Rényi divergence at which a Rényi differential privacy guarantee is stated, a number greater than one.
$\RequiredUsersPerVariant$	Required Users Per Variant	Integer	Number of users each variant of an A/B test needs to detect the minimum detectable effect at the chosen confidence and power.
$\SecondComparedGroupLabel$	Second Compared Group Label	Record	Second of two group labels compared in a group-fairness condition, one value of the protected group attribute.
$\StatisticalHypothesis$	Statistical Hypothesis	Record	A statistical hypothesis about a population, such as the null hypothesis or the alternative hypothesis.
$\SubgroupIndex$	Subgroup Index	Integer	Label of one population subgroup whose outcomes are compared against a reference group.
$\TrainingDistribution$	Training Distribution	Distribution	Data distribution the model was trained on, the baseline that drift is measured against.
$\TrueOutcome$	True Outcome	Dimensionless	True binary outcome of a randomly drawn individual, such as whether an applicant repays, one for the positive class.
$\TrueOutcomeLevel$	True Outcome Level	Integer	One value of a binary true outcome, zero or one, on which a fairness condition is evaluated.

Distributed governance, reliability, and safety constants

A few thresholds and limits recur across the governance and reliability chapters, from classifier decision thresholds to failure-detection timeouts and sequential test statistics. The table below collects these shared constants so their symbols stay distinct from model parameters and network latencies:

Symbol	Definition	Unit	Notes
$\BinaryClassifier(\ModelInput)$	Classifier	Function	Binary classifier $\BinaryClassifier(\ModelInput)$ whose positive-prediction rates the fairness criteria constrain.
$\GradientClippingNorm$	Gradient Clipping Norm	Scalar	Per-example gradient clipping bound $\GradientClippingNorm$ in differentially private training; not a C$^3$ axis.
$\CompletionDeadline$	Deadline	seconds	Target latency deadline for batch or phase completion.
$\HandoverTimeout$	Failure-Detection Timeout	seconds	Interval without a heartbeat after which the coordinator declares a rank failed.
$\ResponseThreshold$	Response Threshold	Scalar	Decision or classification threshold $\ResponseThreshold$.

Units, Precision, and Disambiguation

A link rate quoted in gigabits per second looks eight times larger than the same rate in gigabytes per second, and an 80 GB memory label can mean either $80 \times 10^9$ or $80 \times 2^{30}$ bytes depending on whether decimal or binary prefixes apply. The unit, precision, and disambiguation conventions below keep numerical comparisons consistent across every chapter.

Units and precision

Physical units: This book uses SI (metric) units throughout, including meters, kilograms, seconds, watts, and °C, consistent with standard engineering and scientific practice. Where source data was originally reported in imperial units, the book converts to SI and notes the original values parenthetically. A space always separates the number from the unit (for example, 100 ms, 2 TB/s).
Data and memory: This book uses decimal SI prefixes only: KB = $10^3$ bytes, MB = $10^6$, GB = $10^9$, TB = $10^{12}$. Binary-prefixed units do not appear in prose; model sizes, activation footprints, throughputs, and transfer volumes are reported in decimal units (for example, 2 TB/s, 102 MB). Hardware memory capacities retain the vendor’s nameplate figure and label. A modeled capacity of $80 \times 2^{30}$ bytes follows the registered device assumption, not a guarantee implied by an 80 GB label. Where a worked example subtracts a computed footprint from a device capacity, the registered capacity in bytes and the footprint are both converted to decimal; System Assumptions states this exception in full.
Compute: The notation distinguishes operation counts from rates. Total work uses FLOPs (for example, GFLOPs, TFLOPs), while throughput uses FLOP/s with decimal prefixes (for example, GFLOP/s, TFLOP/s).
- 1 TFLOP = $10^{12}$ FLOPs
- 1 TFLOP/s = $10^{12}$ FLOPs per second
- Arithmetic intensity conventionally uses FLOP/byte as a unit ratio (floating-point operations per byte moved).
Network throughput: Reported in bytes/s (GB/s, TB/s) or bits/s (Gbps) depending on convention. InfiniBand and Ethernet specifications use Gbps; application-level throughput uses GB/s. The text states the convention on first use.
Power: Power carries a subscript ($\AveragePower$, $\DevicePower$, $\Glyph{P_{\text{cluster}}}$, $\InformationTechnologyPower$); bare $\Glyph{P}$ remains parameter count.
Latency percentiles: Written p50, p95, and p99 in upright type, inside math as well as in prose.
Currency: Dollar amounts use the dollar sign ($); unless otherwise noted, dollar-denominated costs are U.S. dollars (USD).
Precision:
- FP64: Double precision (8 bytes)
- FP32: Single precision (4 bytes)
- TF32: TensorFloat-32 (19-bit Tensor Core compute mode; inputs and storage remain FP32)
- FP16: Half precision (2 bytes, standard range)
- BF16: Brain float (2 bytes, wide dynamic range)
- FP8: Quarter precision (1 byte, E4M3 or E5M2 format)
- FP4: 4-bit floating-point format
- INT8: 8-bit integer (1 byte)
- INT4: 4-bit integer; lower integer precisions follow the same uppercase INTn pattern (for example, INT3, INT2)

Quick reference: Resolving collisions

Several single-letter symbols carry conflicting default meanings in machine learning and systems papers. Common collision points in distributed ML systems literature include:

Symbol	ML meaning	Systems meaning	Book convention
$\BatchSize$	Batch Size	Bandwidth	Batch Size. Use $\Bandwidth$ for bandwidth.
$\DatasetSize$	Dataset Size	Data Volume	Dataset Size. Use $\DataVolume$ for bytes moved.
$\LearningRate$	Learning Rate	Efficiency	Learning Rate. Use $\EfficiencyHw$ for efficiency.
$\Parameters$	Parameters	Peak FLOP/s	Parameters. Use $\PeakRate$ for peak rate.

As a general principle, ML conventions take precedence for single letters, while systems concepts get subscripts or multi-letter symbols. This reflects the primary audience (ML practitioners learning systems) and preserves compatibility with the ML literature.

Additional collision points in distributed execution

Moving from a single accelerator to a multi-node cluster brings additional naming conflicts across networking, queuing theory, and fault tolerance. The distributed context introduces further symbol collisions beyond single-device notation:

Symbol	ML meaning	Distributed meaning	Book convention
$\DeviceCount$	(none)	Number of Devices	Device Count in distributed context (accelerators, not host nodes); $\DeviceCount= \AcceleratorCount$.
$\Glyph{M}$	GEMM dimension; memory root ($\TotalMemoryFootprint$)	Collective Payload	Payload when bare, outside GEMM shape statements. Memory footprints always carry a subscript.
$\Glyph{\lambda}$	Regularization weight	Failure Rate	Subscript every other use. Bare $\Glyph{\lambda}$ is the per-component failure rate; arrival rate is $\ArrivalRate$.
$\Glyph{d}$, $\Glyph{t}$, $\Glyph{p}$	Hidden dimension; time; distribution $\ProbabilityFunction(\cdot)$	Data-, tensor-, and pipeline-parallel degree	Core meanings. Parallelism degrees are $\DataParallelDegree$, $\TensorParallelDegree$, and $\PipelineParallelDegree$.
$\Glyph{\tau}$	Temperature or threshold	Checkpoint Interval or Delay	Subscript every distributed use. Checkpoint intervals are $\CheckpointInterval$ and $\OptimalCheckpointInterval$; gradient staleness is $\GradientStaleness$.
$\Glyph{\alpha}$	Significance Level	Network Latency	Subscript every other use. Bare $\Glyph{\alpha}$ is $\MessageStartupLatency$-$\LinkBandwidth$ per-message latency; $\ScalingExponent$ is the scaling exponent, $\HypothesisTestSignificanceLevel$ the significance level, and $\RenyiOrder$ the Rényi order.
$\Glyph{\rho}$	Correlation	Utilization or Ratio	Subscript every other use. Bare $\Glyph{\rho}$ is the communication-computation ratio; $\ServingUtilization$ is serving or queue utilization.
$\Glyph{\epsilon}$	Small Constant	Privacy Budget	Subscript every other use. Bare $\Glyph{\epsilon}$ is the privacy budget; $\AdversarialPerturbationBudget$ is the adversarial radius, and stabilizing constants carry their own subscripts.
$\Glyph{g}$, $\Glyph{G}$	Gradient $\Glyph{\mathbf{g}}$	LogGP gaps; GPUs per node	LogGP only when bare. Gradients are bold $\Glyph{\mathbf{g}}$; accelerators per node are $\AcceleratorsPerNode$.
$\Glyph{K}$	Top-$\Glyph{K}$, key matrix $\AttentionKeyMatrix$	Attack Steps	Local when bare. PGD attack steps are $\AttackStepsPgd$; other counts are defined at use or subscripted ($\RetainedGradientElementCount$).

Symbol	ML meaning	Distributed meaning	Book convention
\(\DeviceCount\)	(none)	Number of Devices	Device Count in distributed context (accelerators, not host nodes); \(\DeviceCount= \AcceleratorCount\).
\(\Glyph{M}\)	GEMM dimension; memory root (\(\TotalMemoryFootprint\))	Collective Payload	Payload when bare, outside GEMM shape statements. Memory footprints always carry a subscript.
\(\Glyph{\lambda}\)	Regularization weight	Failure Rate	Subscript every other use. Bare \(\Glyph{\lambda}\) is the per-component failure rate; arrival rate is \(\ArrivalRate\).
\(\Glyph{d}\), \(\Glyph{t}\), \(\Glyph{p}\)	Hidden dimension; time; distribution \(\ProbabilityFunction(\cdot)\)	Data-, tensor-, and pipeline-parallel degree	Core meanings. Parallelism degrees are \(\DataParallelDegree\), \(\TensorParallelDegree\), and \(\PipelineParallelDegree\).
\(\Glyph{\tau}\)	Temperature or threshold	Checkpoint Interval or Delay	Subscript every distributed use. Checkpoint intervals are \(\CheckpointInterval\) and \(\OptimalCheckpointInterval\); gradient staleness is \(\GradientStaleness\).
\(\Glyph{\alpha}\)	Significance Level	Network Latency	Subscript every other use. Bare \(\Glyph{\alpha}\) is \(\MessageStartupLatency\)-\(\LinkBandwidth\) per-message latency; \(\ScalingExponent\) is the scaling exponent, \(\HypothesisTestSignificanceLevel\) the significance level, and \(\RenyiOrder\) the Rényi order.
\(\Glyph{\rho}\)	Correlation	Utilization or Ratio	Subscript every other use. Bare \(\Glyph{\rho}\) is the communication-computation ratio; \(\ServingUtilization\) is serving or queue utilization.
\(\Glyph{\epsilon}\)	Small Constant	Privacy Budget	Subscript every other use. Bare \(\Glyph{\epsilon}\) is the privacy budget; \(\AdversarialPerturbationBudget\) is the adversarial radius, and stabilizing constants carry their own subscripts.
\(\Glyph{g}\), \(\Glyph{G}\)	Gradient \(\Glyph{\mathbf{g}}\)	LogGP gaps; GPUs per node	LogGP only when bare. Gradients are bold \(\Glyph{\mathbf{g}}\); accelerators per node are \(\AcceleratorsPerNode\).
\(\Glyph{K}\)	Top-\(\Glyph{K}\), key matrix \(\AttentionKeyMatrix\)	Attack Steps	Local when bare. PGD attack steps are \(\AttackStepsPgd\); other counts are defined at use or subscripted (\(\RetainedGradientElementCount\)).