Foundation Principles

Machine learning systems exhibit a deceptively simple heuristic: complexity often moves rather than disappearing, although good design can remove accidental complexity. Complexity flows among the three domains of the D·A·M taxonomy: Data as information, Algorithm as logic, and Machine as physics. Simplifying one domain can burden the others. A hand-crafted feature pipeline reduces algorithmic complexity but demands more data engineering effort. A larger model absorbs messy data but shifts complexity onto the hardware that must train and serve it. This Conservation of Complexity heuristic motivates everything in this book. The quantitative bounds, models, and principles introduced throughout the book describe constraints that emerge from where complexity currently resides.

Architectures, frameworks, and optimizations succeed only when they respect constraints imposed by hardware, mathematics, and information theory. Just as civil engineers cannot ignore gravity, ML engineers cannot ignore the physical laws that govern data, computation, and system throughput. Part I establishes these constraints and models: not best practices that evolve with frameworks or opinions that differ between teams, but the physics and models of ML engineering. The first constraint starts with data itself, where the familiar boundary between program and input begins to disappear.

Principle 1: The Data-as-Code Principle

Principle: Data functions as the source code of the ML system. A change to the training dataset can alter the system’s learned behavior. \[ \text{Learned Behavior} \approx f(\text{Training Data}, \text{Training Procedure}) \]

Implication: Data engineering requires the same rigor as software engineering. Datasets should be versioned (like Git), unit-tested (data quality checks), and debugged. Deleting a row of training data can alter the rebuilt model just as deleting a line of code can alter a compiled artifact.

If data functions as source code, then it is not merely a logical artifact—it also has physical properties that constrain system architecture. Unlike code, which can often be copied and distributed cheaply, a large dataset can resist movement over a constrained network, and its scale can change where computation should happen.

Principle 2: The Data-Gravity Principle

Principle: As data volume (\(D_{\text{vol}}\)) increases, moving data can cost more in latency, bandwidth, or energy than moving compute when the network is constrained. \[ \text{Cost to transfer dataset} \gg \text{Cost to transfer query or code} \quad \text{in this regime} \]

Implication: Large datasets can become the gravitational center of the architecture. Systems may move compute to data by shipping queries or code to the storage layer rather than moving data to compute by repeatedly downloading large datasets.

Together, these two principles establish that data can be both the logical program and a physical anchor of an ML system. With these foundations in place, Part I builds the conceptual framework: from the discipline’s origins and core metrics, through the physical constraints that create the deployment spectrum, to the lifecycle that manages complexity across stages, and finally to the engineering practices that treat data with the rigor it demands.