Data Foundations

Purpose

What makes “data” a first-class systems constraint, and how do we measure when it is silently breaking our models?

In ML systems, data is not an abstract dataset but physical volume that must move through disks, networks, CPUs, and accelerator memory. Many expensive training runs are limited not by FLOP/s, but by I/O bandwidth, serialization overhead, and avoidable scans of irrelevant bytes. In production, the more dangerous failure mode is quieter: distributions drift, tails dominate user experience, and accuracy degrades long before average metrics look suspicious. This appendix collects the reference calculations and statistical tools for reasoning about the data path as a systems engineer: data gravity napkin math, format and serialization costs, the algebraic primitives that create pipeline blowups, and drift metrics that compare full distributions rather than means. In D·A·M terms, it isolates the data axis and shows how volume, movement, and distribution constrain algorithm behavior and machine utilization.

How to Use This Appendix

This appendix is designed as a reference. Reach for it when debugging “slow training,” “low accelerator utilization,” or “production accuracy drift” calls for the quickest path from symptom to measurement.

Conventions used here follow the book-wide notation (for example, we reserve \(B\) for batch size and use \(\text{BW}\) for bandwidth).

• When data will not move: Start with table 1 and the transfer-time equation in section 1.1.
• When the accelerator is starving: Use table 2 and the layout discussion in section 1.1.3.
• When pipelines explode in cost: Use the primitives in section 1.1.4, especially join-induced shuffles.
• When “average looks fine” but users complain: Use section 1.2.1 and session-level tail probability.
• When accuracy drifts silently: Use section 1.2.2 to compare full distributions.

The data landscape can feel like a zoo of formats, encodings, and edge cases, but this appendix focuses on the structural constants that constrain every ML system.

With those reference points in place, the appendix begins with the physical constraints of data engineering and then moves to the statistical monitoring that keeps ML systems healthy. From storage formats that determine I/O throughput to drift metrics that detect silent failures, these foundations connect directly to the data pipelines in Data Engineering and the operational monitoring in ML Operations.

Data Engineering Foundations

Understanding hardware constraints is only half the battle; we must also shape our data to fit them. Data engineering applies the principles of the memory hierarchy to storage formats and pipeline design, ensuring that the accelerator never starves. This process begins with recognizing that data is physical—it has volume, it takes time to move, and it requires energy to parse.

Napkin math: The physics of data gravity

Data gravity¹ is a metaphor grounded in transfer-time calculations. Unlike compute, which gets faster every year, the speed of light is fixed and network bandwidth is a finite resource. When datasets grow large enough, their practical inertia grows—moving them can cost more time and energy than moving computation to where the data already lives.

¹ Data Gravity: Coined by Dave McCrory in 2010 to describe how large datasets attract services and applications toward them, much as massive bodies attract smaller ones in physics (McCrory 2010). The analogy is apt: the “escape velocity” required to move a petabyte-scale dataset is often measured in weeks.

McCrory, Dave. 2010. Data Gravity – in the Clouds.

The ideal line-rate transfer time is \(T = D_{\text{vol}} / \text{BW}\) (for the large volumes here, latency is negligible; the full equation appears in Bandwidth vs. latency). Table 1 gives lower bounds; protocol overhead and endpoint limits increase them:

Table 1: The Cost of Inertia: Why we often “ship compute to data” rather than “ship data to compute.” Network columns are lower bounds at sustained line rate. The truck entry is an illustrative elapsed-time model for 1 PB: 8 hours to load, 32 hours in transit, and 8 hours to unload, excluding scheduling and provisioning overhead.

Data Volume	1 Gbps	10 Gbps	100 Gbps	Truck
1 TB	2.2 hours	13.3 minutes	80 seconds	N/A
100 TB	9 days	22.2 hours	2.2 hours	N/A
1 PB	3 months	9 days	22.2 hours	2 days

The cost of serialization

Even after data arrives at the machine, we face one final hurdle: the serialization tax.² As table 2 shows, many engineers meticulously optimize their accelerator kernels while ignoring the CPU overhead of decoding data. Parsing text-based formats like JavaScript Object Notation (JSON) or CSV is extremely CPU-intensive, often leaving the accelerator idling while the CPU struggles to convert strings into floating-point numbers.

² Serialization: From Latin serialis (forming a series). The process of converting in-memory data structures into a byte stream for storage or transmission, and the reverse (deserialization). In ML pipelines, the choice of serialization format can dominate end-to-end training time; Data Engineering covers pipeline design strategies that minimize this overhead.

Table 2: Serialization Overhead: These illustrative values show why binary columnar formats can outperform row-based text, but they are not portable benchmarks. Arrow is an in-memory/IPC format; Parquet is a compressed storage format.

Format	Decoding Speed (MB/s)	Relative CPU Decode Cost	Suitability
CSV/JSON	~100 MB/s	High	Inspection/interchange
Protobuf	~300 MB/s	Medium	RPC/Messages
Parquet	> 1,000 MB/s	Low	Columnar storage

Row vs. columnar formats

The choice of file format determines the “physics” of how the data is read. Row-oriented formats such as CSV and JSON store data record-by-record, so reading the age field still requires parsing each row. This suits appended logs but not analytics on selected features. Parquet stores compressed column chunks on disk, while Arrow provides a columnar in-memory and IPC format. Parquet readers can use projection pushdown to fetch selected column chunks, and both formats support vectorized processing. Compare the two arrangements in figure 1 to see why columnar access avoids scanning unnecessary bytes.

Figure 1: Storage Layouts: Row-oriented formats pack data together by record (good for transactions). Column-oriented formats pack data by feature (good for analytics).

This layout difference becomes a systems issue whenever accelerator utilization depends on data loading speed.

Systems Perspective 1.1: The accelerator starvation problem

The choice of file format determines whether a system is I/O bound or compute bound. As table 2 shows, the serialization tax compounds with storage layout: row-oriented formats force full-row scans while columnar formats enable projection pushdown, reading only the bytes the model needs. The result is that the “Data Movement” term in the iron law can silently become the bottleneck that leaves expensive accelerators idling.

The algebra of data

Feature engineering turns raw records into the columns a model can learn from, and that transformation is usually a dataflow problem before it is a modeling problem. A pipeline first narrows the population, then chooses the features, then attaches context from other tables. Those three moves correspond to three Structured Query Language (SQL) primitives, and their computational cost determines whether the feature job stays local, scans unnecessary bytes, or turns into a network shuffle.

1. Selection (\(\sigma\)): Filtering rows (for example, WHERE age > 30).
   • Cost: \(\mathcal{O}(\log N + K)\) if an index lookup returns \(K\) rows; \(\mathcal{O}(N)\) for a full scan.
2. Projection (\(\pi\)): Selecting columns (for example, SELECT age).
   • Cost: Selected chunks plus decoding and metadata in columnar formats. This row-format example reads 1 KB for a 4-byte integer, wasting 99.6 percent of the read.
3. Join (\(\bowtie\)): Combining tables. The most expensive operation.
   • Shuffle Join: Both tables are partitioned by key and exchanged over the network.
     • Cost: Reshuffling both 1 TB tables moves about ~2 TB; partitioning and locality can reduce this.
   • Broadcast Join: One small table is sent to all workers.
     • Cost: \(S(W-1)\) bytes; the \(S\)-byte table must fit on each of \(W\) workers.

Understanding data formats, serialization costs, and algebraic primitives tells us how to move data efficiently. Even a perfectly engineered pipeline, however, can silently fail if the data it carries changes character over time. Detecting that change—and quantifying how much it matters—requires a different set of tools: probability and statistics.

Probability and Statistics

Once data is flowing through our pipelines, we need mathematical tools to ensure its quality and consistency. Probability and statistics provide the language for monitoring system health, detecting the silent failures of data drift, and managing uncertainty.

Systems Perspective 1.2: Why statistics matters for systems

Average latency can look fine while users still experience failures because systems live in the “long tail.” Statistics gives us the tools to measure uncertainty, detect drift, and handle numerical stability—three capabilities that separate robust production systems from fragile ones.

Distributions and the long tail

In systems, the mean is often misleading. Latency distributions are often long-tailed, and user-visible services are commonly governed by high-percentile behavior rather than mean response time (Dean and Barroso 2013). A “P99” (99th percentile) latency of 500 ms means 1 percent of requests experience that tail latency; with many requests per session, the fraction of users who see at least one slow request can be much higher. At scale (1M users), even a 1 percent affected-user rate would affect 10,000 users.

Napkin Math 1.1: The median experience

For \(N_{\text{session}} = 100\) independent requests, each with a 0.01 chance of being slow, the probability of at least one P99-tail event is \(\Pr(\text{Slow}) = 1 - (0.99)^{100} \approx 1 - 0.366 = 63.4\%\).

Computed: \(\Pr(\text{Slow})\) = 63.4 percent. Under this model, a 100-request session is more likely than not to include an event beyond P99. Correlated slowdowns can change this probability (Dean and Barroso 2013; DeCandia et al. 2007).

DeCandia, Giuseppe, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels. 2007. “Dynamo: Amazon’s Highly Available Key-Value Store.” Proceedings of Twenty-First ACM SIGOPS Symposium on Operating Systems Principles, 205–20. https://doi.org/10.1145/1294261.1294281.

Dean, Jeffrey, and Luiz André Barroso. 2013. “The Tail at Scale.” Communications of the ACM 56 (2): 74–80. https://doi.org/10.1145/2408776.2408794.

Measuring drift (divergence)

Because distributions have long tails where the most dangerous failures hide, simple metrics like “mean shift” are insufficient. We need tools that compare the entire shape of the distribution. Detecting drift between the serving distribution (\(P_t\)) and training distribution (\(P_0\)) requires measuring the “distance” between distributions.

KL divergence³ measures the expected extra coding cost of approximating \(P_t\) with \(P_0\) (Kullback and Leibler 1951): \[ \mathcal{D}_{\text{KL}}(P_t \lVert P_0) = \sum_x P_t(x) \log \frac{P_t(x)}{P_0(x)} \]

³ KL Divergence: Named after Solomon Kullback and Richard Leibler, who introduced it in 1951. Also called relative entropy, it quantifies the expected extra information needed to encode samples from one distribution using a code optimized for another; for drift monitoring in this volume, the canonical direction is \(\mathcal{D}_{\text{KL}}(P_t \lVert P_0)\). With natural logarithms, as in the example above, the unit is nats; with base-2 logarithms, the unit is bits. In production ML, KL divergence is the theoretical backbone of many drift-detection metrics.

Kullback, Solomon, and Richard A. Leibler. 1951. “On Information and Sufficiency.” The Annals of Mathematical Statistics 22 (1): 79–86. https://doi.org/10.1214/aoms/1177729694.

Napkin Math 1.2: Worked example: KL divergence for drift detection

Scenario: A sentiment classifier was trained on data where 60 percent of reviews were positive, 30 percent negative, and 10 percent neutral. After deployment, the serving distribution shifts to 45 percent positive, 40 percent negative, and 15 percent neutral.

Training distribution \(P_0\): [0.60, 0.30, 0.10]. Serving distribution \(P_t\): [0.45, 0.40, 0.15].

\[\begin{gather*} \mathcal{D}_{\text{KL}}(P_t \lVert P_0) = 0.45 \log\frac{0.45}{0.60} + 0.40 \log\frac{0.40}{0.30} + 0.15 \log\frac{0.15}{0.10}\end{gather*}\] \[\begin{gather*} = 0.45 \times (-0.2877) + 0.40 \times (0.2877) + 0.15 \times (0.4055) \\[-1pt] = -0.1295 + 0.1151 + 0.0608 = 0.0464\text{ nats}\end{gather*}\] \[\begin{gather*} \mathcal{D}_{\text{KL}}(P_0 \lVert P_t) = 0.60 \log\frac{0.60}{0.45} + 0.30 \log\frac{0.30}{0.40} + 0.10 \log\frac{0.10}{0.15}\end{gather*}\] \[\begin{gather*} = 0.60 \times 0.2877 + 0.30 \times (-0.2877) + 0.10 \times (-0.4055) \\[-1pt] = 0.1726 + (-0.0863) + (-0.0405) = 0.0458\text{ nats}\end{gather*}\]

Notice the asymmetry: \(\mathcal{D}_{\text{KL}}(P_0 \lVert P_t) \neq \mathcal{D}_{\text{KL}}(P_t \lVert P_0)\). The Population Stability Index (PSI) symmetrizes this:

\[\begin{gather*} \text{PSI} = \sum (P_{0,i} - P_{t,i}) \log\frac{P_{0,i}}{P_{t,i}} = (0.15)(0.2877) + (-0.10)(-0.2877) + (-0.05)(-0.4055) \\[-1pt] = 0.04315 + 0.02877 + 0.02027 = 0.0922 \end{gather*}\]

Since PSI = 0.0922 < 0.2, this drift does not yet cross the illustrative manual-review threshold used here. The right response is to keep the feature on the monitoring dashboard and look for corroborating drift signals rather than retraining from this statistic alone.

Because KL divergence is asymmetric (\(\mathcal{D}_{\text{KL}}(P_0 \lVert P_t) \neq \mathcal{D}_{\text{KL}}(P_t \lVert P_0)\)), practitioners often use PSI, a symmetric heuristic. A PSI above 0.2 may prompt review, but thresholds depend on sample size, binning, smoothing, and zero handling; retraining requires model-quality evidence.

Both KL divergence and PSI are grounded in a deeper framework—information theory—which provides the units and bounds that make these metrics principled rather than ad hoc.

Information theory for systems

A training run can consume more examples, run longer, and still stop improving if the added data carries too little useful signal. At that point, the bottleneck is not only compute or bandwidth; it is the amount of information the data pipeline delivers to the learner. Information roofline (the destination) treats that data quality limit as a physical constraint, and information theory provides the units for reasoning about it.

Entropy (\(H\))⁴ is the average uncertainty in a distribution, defined as \(H(X) = -\sum p(x) \log p(x)\). The log base sets the unit: natural logs give nats, while \(\log_2\) gives bits. A uniform distribution has maximum entropy only over a fixed finite support. This connects directly to KL divergence above: \(\mathcal{D}_{\text{KL}}\) measures the excess coding cost of using the wrong distribution.

⁴ Entropy: From Greek entropia (transformation), the term was adopted by Shannon in 1948 to quantify information content (Shannon 1948). In systems terms, Shannon entropy is a lower bound on average lossless code length. Practical prefix codes approach it, and block codes can approach it asymptotically. With \(\log_2\), the unit is bits; natural logs give nats. These bounds inform compression ratios and data-pipeline sizing.

Shannon, Claude E. 1948. “A Mathematical Theory of Communication.” Bell System Technical Journal 27 (3): 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x.

Information Density is used here as a book-level heuristic for useful learning signal per unit of storage, not as a standard information-theory quantity.

Signal-to-Noise Ratio (SNR) is the ratio of signal power to noise power. In ML, the book uses low SNR qualitatively for data in which task-relevant structure is weak relative to noise; adding compute cannot compensate for missing task-relevant signal.

Logits and numerical stability

Neural networks output logits⁵ (unnormalized scores), not probabilities. We convert them using Softmax:

⁵ Logit: From log + unit, coined by Joseph Berkson in 1944. The logit function is the inverse of the logistic (sigmoid) function: \(\text{logit}(p) = \log(p/(1-p))\). In deep learning, “logits” refers more loosely to the raw, unnormalized output of the final linear layer before any activation function is applied. \[ \text{Softmax}(z_i) = \frac{e^{z_i}}{\sum e^{z_j}} \]

⁶ Log-Sum-Exp: Implemented as torch.logsumexp in PyTorch and scipy.special.logsumexp in SciPy. It relies on the identity \(\log\left(\sum e^{x_i}\right) = a + \log\left(\sum e^{x_i - a}\right)\), where \(a = \max(x_i)\). Shifted values \(x_i - a\) are \(\le 0\), ensuring exponentials never overflow.

The problem is that if \(z_i\) is large (for example, 100), the exponential \(e^{z_i}\) overflows common training and inference formats such as FP32, BF16, and FP16. FP64 can represent this specific value, but production ML kernels rarely use FP64 for softmax. The solution is to compute in log-space: the “Log-Sum-Exp” trick allows us to compute \(\log\left(\sum e^{z_j}\right)\) without ever calculating the massive exponentials directly, preserving numerical precision.⁶

A small worked example shows why this trick matters in practice, even for large but realistic logit values:

Example 1.1: Log-sum-exp in action

Setup: A three-class classifier outputs logits \(z =\) \(\lbrack 100, 101, 102 \rbrack\).

Without the trick (naive softmax): \[ \text{$\exp(100) \approx 2.7 \times 10^{43}$},\quad \text{$\exp(101) \approx 7.3 \times 10^{43}$}, \quad \text{$\exp(102) \approx 2.0 \times 10^{44}$} \]

These numbers are representable in FP64 but overflow FP32 (max \(\approx 3.4 \times 10^{38}\)). With FP16 (max \(\approx 65{,}504\)), even \(e^{12}\) overflows. Raw logits rarely reach magnitude 100, but large intermediate values do arise (for example, unscaled attention scores or overconfident outputs), and because FP16 and BF16 are the standard training precisions, a naive softmax risks overflow often enough that the stable form is used by default.

With the trick: Subtract \(a = \max(z) =\) 102: \[\begin{gather*} \text{$z - a =$ $\lbrack -2, -1, 0 \rbrack$} \\ \text{$\exp(-2) \approx 0.135$,}\quad \text{$\exp(-1) \approx 0.368$,}\quad \text{$\exp(0) = 1.0$} \end{gather*}\]

Sum = 1.503. LogSumExp = 102 \(+ \log(1.503) = 102.408\).

Softmax: \(\lbrack 0.135/1.503,\; 0.368/1.503,\; 1.0/1.503 \rbrack = \lbrack 0.090,\; 0.245,\; 0.665 \rbrack\)

Systems insight: The exponentiated shifted values and the resulting softmax probabilities are in \([0, 1]\)—no overflow risk, even in FP16.

These worked examples now give us enough machinery to check the full chain, from physical data movement to distribution shift and numerical stability.

Checkpoint 1.1: Check your understanding

1. A training pipeline reads 500 GB of CSV data over a 10 Gbps link. Estimate the transfer time. Now estimate how long it would take if the data were stored as Parquet and only 20 percent of columns were needed—what changes and why?
2. Your model’s average latency is 50 ms, but P99 is 800 ms. Assuming independent requests, what is the probability that a fifty-request session includes a request beyond P99? Does average latency describe user experience under this model?
3. Explain why KL divergence is asymmetric and why this matters when choosing a drift metric for production monitoring. When would you prefer PSI over raw KL divergence?

The tools in this section—tail-aware metrics, drift divergences, information-theoretic bounds, and numerical stability tricks—give us the vocabulary to diagnose data-related failures quantitatively rather than anecdotally.

Summary

Key Takeaways: Data as a physical constraint

• Data has physical inertia: Transfer time scales linearly with volume and inversely with bandwidth, making petabyte-scale transfers expensive enough that locality often wins. Design pipelines around data locality rather than data movement.
• Serialization format is a first-order decision: Columnar binary formats can decode much faster than text for suitable workloads, but the factor varies with schema, compression, implementation, storage, and hardware, affecting accelerator utilization.
• Algebraic primitives shape I/O cost: Selection, projection, and join have radically different I/O costs. Joins in particular can require moving both input tables across the network; choosing between shuffle and broadcast joins depends on relative table sizes.
• Average metrics hide tails: High request counts increase the chance that a session encounters tail latency; quantify this with an explicit dependence model rather than treating P99 as a worst case.
• Drift detection compares distributions: KL divergence and PSI help detect input-distribution shift before or alongside accuracy metrics, but neither alone establishes an accuracy change.
• Numerical stability is mandatory: The log-sum-exp trick and log-space computation prevent overflow in softmax and loss calculations, making them essential building blocks of any training or inference pipeline.