Policy Training

6 Policy Training

Purpose

Physical AI stack focus for this chapter.

Why can a policy improve its training score while becoming less reliable on the machine?

A learned policy acquires its behavior from the experience its training supplies, and its weights keep no record of which experience that was. Demonstrations can omit the recovery from a displaced fixture, a simulator can reward a fast approach that a real contact surface cannot tolerate, and a model pretrained on other robots can behave plausibly in conditions no one has tested on this machine. In each case the training score rises while the dependency stays hidden, so ranking candidates by score can favor the one that exploited its data or its solver. Training also fixes how long the policy takes to propose an action, and every improvement gathered on the machine itself is paid for in heat and component life. An engineer who does not understand these mechanisms cannot choose a source of experience that supports the behavior the task needs, cannot state the conditions in which a checkpoint may be trusted, and discovers each hidden dependency only when the machine meets it. Because an admitted action crosses the causal boundary and its consequences cannot be undone by another gradient update, the limits of a policy’s evidence must be established before it is allowed to propose motion.

Learning Objectives

Compare training regimes, from behavioral cloning to post-training a pretrained generalist, by the experience they supply and the physical dependencies they leave in the policy
Diagnose the separate risks of averaging valid action choices and drifting beyond demonstrated states
Calculate whether an action decoder fits its proposal deadline and identify the required execution safeguards
Evaluate simulator mismatch and distinguish parameter variation from missing mechanisms or unavailable observations
Bound real-machine adaptation by physical exposure, residual authority, and regression evidence
Construct a policy manifest that confines a candidate’s declared operating region to its documented support

6.1 Dual Failure Modes of Policy Training

A robot arm executing a learned trajectory can track every waypoint cleanly across hundreds of simulation episodes and still arrive at its first physical deployment with no evidence about whether it will fracture a latch. In classical robotics, control design begins with an explicit analytical model of the plant: engineers write down kinematic equations, inertia matrices, stability margins, and torque limits where they can be systematically inspected and verified. A learned neural policy, in contrast, derives its behavior entirely from its training distribution, setting network weights from empirical experience to map physical observations to proposed control actions. That experience may arrive as teleoperated demonstrations, recorded operational logs, rollouts inside a parallel physics simulator, trial-and-error interactions on physical hardware, or pretrained representations imported from foundation models. Although empirical data makes complex manipulation tractable, it also embeds unstated dynamic assumptions directly into the model weights, leaving no explicit record of which operating regimes the policy can safely navigate.

The central engineering question in embodied policy training is therefore not which regime achieves the highest optimization score, but which physical behaviors the training experience supports, and at what operational cost. Four quantities decide the answer. The first is coverage, because a policy is supported only in the states its experience actually visited, which is why the provenance record of The Dataset Schema lists its known absences beside its demonstrated cells. The second is inference latency, because a policy that evaluates too slowly misses its control deadline, however good its proposed trajectories are. The third is hardware cost, since experience gathered on the machine consumes machine hours, heats motor windings, and wears drivetrains. The fourth is fidelity, because synthetic experience from simulation scales across parallel accelerators but is only as faithful as the integrator and contact solver that produced it. A checkpoint’s policy manifest (section 6.8) records all four, and its declared operational design domain (ODD) is the part of the task’s scenario table that the checkpoint’s evidence supports. Two candidate latch policies show why a clean training curve does not establish physical viability.

Consider the cage-door latch task introduced in Endogenous Experience, where two learned controllers both succeed under nominal warehouse conditions yet each carries a hidden dependency that physical hardware can expose, for different mechanical reasons. Both controllers share the same real-time execution model: Action Chunking with Transformers (ACT) (Zhao et al. 2023), a chunk-policy architecture that generates open-loop action sequences in a single decoder pass. The first candidate, latch-bc-01, was trained via supervised behavioral cloning on teleoperated door-opening demonstrations. The second candidate, latch-sim-02, is a distilled student policy: a reinforcement learning teacher was first trained inside a massively parallel physics simulator with full access to privileged state variables, after which latch-sim-02, initialized from latch-bc-01, learned through distillation to replicate the teacher’s actions using only onboard sensor streams (section 6.5.1).

Both candidates inherit the absence list of the baseline archive. Its demonstrations characterize only the nominal strike plate at a guarded approach velocity, so a displaced plate and a faster approach are both recorded as known absences, and each candidate meets one of them. While behavioral cloning must interpolate between these discrete demonstrations to form coherent trajectory proposals, interactive learning in simulation can explore novel state-action pairs. However, the validity of that synthetic exploration depends entirely on how accurately the simulated mechanics mirror the physical plant. Selecting a training regime is therefore an architectural decision that dictates which unverified assumptions must be empirically audited before a checkpoint is permitted to command actuators.

In this illustrative comparison, at the nominal strike plate and the guarded approach speed of 0.03 m/s, both policies open the door. Their failure modes diverge once physical deployment departs from that single nominal operating point:

Missing recovery experience: When a sagging door places the strike plate off nominal, latch-bc-01 encounters observations outside its training distribution. Because behavioral cloning minimizes supervised prediction error across the recorded demonstrations, the training loss supplies no direct supervision at unvisited recovery states, although updates to shared weights can still change the actions predicted there. When confronted with unfamiliar observations, the network still emits action setpoints, but those ungrounded commands drive the manipulator further off course. Trajectory errors compound at each 25 ms setpoint tick until the gripper binds against the strike plate’s chamfered edge. The root bottleneck is the absence of corrective recovery trajectories in the training distribution, rather than model capacity.
Misleading simulated contact: In contrast, latch-sim-02 is at risk on the nominal strike plate itself. Its simulator models contact dynamics using a compliant penalty spring¹ (\(\NormalContactForce = \PenaltyContactStiffness \PenetrationDepth + \PenaltyContactDamping \PenetrationRate\)), where surface penetration \(\PenetrationDepth\) and penetration rate \(\PenetrationRate\) govern the contact force. Because this numerical spring yields far more compliance than a physical steel plate, the simulated impact at 0.10 m/s peaks well under the latch’s 100 N limit. The teacher’s training reward paid for opening the door quickly and carried no term for contact force, so nothing in training charged for an aggressive approach, and the optimizer exploited the contact model to shave cycle time off each episode. The reinforcement learning teacher converges on an aggressive plunge, and policy distillation faithfully transfers that shortcut into the student. Whether the plunge breaks the real latch depends on how stiff the arm side of the contact is, a quantity nothing in training measured.

¹ Kelvin-Voigt Contact: A linear spring and damper in parallel produce a force jump at first touch, because the damping term acts on the full approach velocity at zero penetration, and a pulling, “sticky” force as the bodies separate, because the damping term then outweighs the vanishing spring term. Hunt and Crossley (1975) scaled the damping with penetration to remove both artifacts. A policy trained against the linear model therefore meets its least faithful forces at contact onset and at release, the two events a latch opening depends on.

Hunt, Kenneth H., and F. R. Erskine Crossley. 1975. “Coefficient of Restitution Interpreted as Damping in Vibroimpact.” Journal of Applied Mechanics 42 (2): 440–45. https://doi.org/10.1115/1.3423596.

A contact-force budget demonstrates why the simulated benchmark score cannot be trusted on hardware. In the rigid-arm, constant-drive estimate, the physical strike plate’s stiffness sets the force rise. At the unbraked approach velocity of 0.10 m/s, force reaches the 100 N limit in 2.5 ms after touch. The tripwire fires at 0.375 ms; the subsequent 2 ms response places brake onset at 2.375 ms, leaving only 0.125 ms before the constant-drive ramp reaches the limit. That interval is not a stopping certificate. Section 6.4.2 derives the timeline alongside the passive impact forces; sanctioning the fast plunge would require physical deceleration evidence that this simulation-only training process cannot establish.

Both failures expose critical dependencies hidden from validation loss and episodic reward: behavioral cloning depends on demonstrated recovery distributions, while simulation-to-real transfer depends on faithful contact and stiffness dynamics. Selecting a policy training regime therefore requires matching the physical properties of the training source to the mechanical demands of the operational task. Whichever regime produced it, a learned checkpoint sits above the proposal boundary as an unprivileged proposal generator. Achieving safe and reliable manipulation requires action proposals that respect the machine’s actuator dynamics, contact deadlines, and physical safety envelopes as rigorously as they pursue the nominal task objective, and comparing the five regimes that can generate those proposals clarifies what each source of experience buys and what it leaves unverified.

↩︎ The proposal boundary, under which a learned proposer holds no authority until an independent permission path admits its command, is established in The Machine in Five Levels.

6.2 Policy Training Regimes

When latch-bc-01 binds against a sagged strike plate and latch-sim-02 plunges into the nominal plate at 0.10 m/s, neither failure can be repaired simply by training the same network for more epochs on the same dataset. Fixing an unvisited recovery state or an exploited contact solver requires changing the experience the learner sees. Training a control policy for the latch task can draw on five sources of experience: teleoperated door openings, logged archives of imperfect or scripted trajectories, a physics simulator of the door and cage, direct physical trial on the arm itself, and a generalist foundation checkpoint pretrained across multi-robot corpora. Each source buys distinct sensorimotor capabilities at a characteristic physical cost and leaves its own failure mode behind (table 1). Behavioral cloning, batch offline reinforcement learning, and supervised post-training extract control policies from recorded datasets without runtime exploration, whereas online trial, conducted in simulation or on the physical mechanism, generates new interactions under closed-loop feedback. Comparing how those regimes evaluate and update actions begins with the reward quantities that score closed-loop behavior over an episode.

Table 1: Comparative trade-off matrix across policy training regimes: The five regimes compared are behavioral cloning, offline reinforcement learning, sim-to-real reinforcement learning, residual RL/fine-tuning, and post-training of a pretrained generalist policy. Columns separate the capability sought, the resources that determine its cost, and the remaining failure mechanism. Training remedies and physical execution safeguards have different roles; neither supplies the other’s evidence.

Policy Training Regime	Core Purchased Capability	Data & Compute Budget	Sample Efficiency & Scale	Hardware Safety & Wear Risk	Dominant Failure Mode & Interceptor
Behavioral Cloning (BC)	Imitates attributable expert actions	Expert collection, curation, and supervised optimization	Limited by coverage and operator availability	Optimization adds no physical exploration; collection still incurs exposure	Mode averaging and missing recovery. Multimodal decoding and new expert labels address training; qualified state and action limits bound execution.
Offline / Batch RL	Uses rewards to select behavior from heterogeneous logs	Logged transitions with rewards and state-action coverage	Reuses recorded experience without new online trials	Optimization adds no exploration; the archive inherits collection risks	Unsupported action values. Conservative objectives reduce optimism; independent action limits remain necessary.
Sim-to-Real RL	Explores behavior through simulated interaction; a privileged teacher can be distilled into a deployable student	Parallel rollout compute plus physical calibration and transfer tests	Throughput depends on task, solver, and hardware	Simulation avoids training wear; physical probing and deployment still incur exposure	Solver exploitation and omitted mechanisms. Paired probes qualify the simulator; approach and force limits need their own response evidence.
Residual RL & Real Fine-Tuning	Adapts to local physical mismatch	Machine interaction, resets, cooling, and regression testing	Physical trials make sample efficiency consequential	Contact loads and thermal exposure accumulate during adaptation	Local improvement with regression elsewhere. Independent residual gates limit authority; combined-controller tests assess the change.
Pretrain & Post-Train	Starts from a generalist checkpoint’s broad competence and adapts it to the task with demonstrations or reward	A base checkpoint trained elsewhere, task data, and fine-tuning compute	Task data needs shrink when the base already represents related skills	Supervised post-training adds no exploration; RL post-training adds the exposure of its own rollouts	Borrowed competence mistaken for evidence, and forgetting. Support comes only from the task data join; regression tests cover what post-training may disturb.

6.2.1 The latch task as a reward problem

A reward function that pays a policy for opening the cage door and penalizes each control step for elapsed time without charging for contact force will train the arm to slam into the strike plate, because every millisecond shaved off the approach increases the episode’s score. Even behavioral cloning, which uses no reward during training, has its failures measured in the quantities that a reward problem defines: the states a policy visits, the return it earns over an episode, and the value of one action against another. Posing the latch mechanism as a discrete-time optimal control problem makes those quantities concrete and shows how an omitted force penalty or a discount factor mis-scaled against the control rate distorts learned behavior.

A Markov decision process (MDP)² describes a task by its state \(\DecisionProcessState_{t}\), its action \(\ContinuousActionVector_{t}\), a transition \(\EnvironmentTransitionDistribution(\DecisionProcessState_{t+1} \mid \DecisionProcessState_{t}, \ContinuousActionVector_{t})\) that the physics supplies, and a scalar reward \(\Reward(\DecisionProcessState_{t}, \ContinuousActionVector_{t})\) that the engineer writes. For the latch, the state holds the gripper pose, the plate position, and the measured contact force. The action is the setpoint the chunk policy emits every 25 ms. An episode is one opening, 8 s of autonomous contact-phase motion, or 320 decisions. A reward that states what the task values and what it must avoid is \[\Reward_t = \DoorOpenReward\,\mathbb{1}[\text{door open at } \ControlStepIndex] - \ExcessForcePenaltyWeight \max\left(0, \LatchContactForce_t - \ContactTripwireForce\right) - \StepTimePenalty, \tag{1}\] where \(\ContactTripwireForce\) is the 15 N contact tripwire, the force threshold that triggers the machine’s qualified collection response (set below the latch’s damage limit), \(\ExcessForcePenaltyWeight\) is a weight in reward per newton of excess force, and \(\StepTimePenalty\) charges each step for slowness. The weights make the terms commensurable, and choosing them is a design decision that the policy manifest records. Dropping the force term, as the simulator teacher of latch-sim-02 did, leaves only the time term to shape the approach, and the time term rewards every millisecond an aggressive plunge saves. On hardware the success indicator must itself be measured, by a latch sensor or a learned classifier, and every error in that measurement is a surface the optimizer can exploit. When engineers introduce auxiliary rewards to guide exploration toward that success indicator, adding a potential difference \(\DiscountFactor\ShapingPotential(\NextDecisionProcessState) - \ShapingPotential(\DecisionProcessState)\) provably leaves the optimal policy unchanged, and without further assumptions about the task it is the only added term that does (Ng et al. 1999). Any other shaping term can change what is optimal.

² Markov Decision Process: Bellman (1957) formulated sequential decisions under uncertainty with the dynamic-programming recursion that equation 2 restates, and Howard (1960) supplied policy iteration to solve them. The formalism came from operations research and optimal control decades before reinforcement learning adopted it, which is why the cost-to-go translation below is an identity rather than an analogy, and why a control engineer’s tools for that recursion apply to a learned policy’s value function.

Bellman, Richard. 1957. Dynamic Programming. Princeton University Press.

Howard, Ronald A. 1960. Dynamic Programming and Markov Processes. Technology Press of the Massachusetts Institute of Technology.

Ng, Andrew Y., Daishi Harada, and Stuart J. Russell. 1999. “Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping.” Proceedings of the Sixteenth International Conference on Machine Learning (ICML), 278–87.

A policy \(\Policy(\ContinuousActionVector \mid \DecisionProcessState)\) is a distribution over actions, and its return from step \(\ControlStepIndex\) is \(\DiscountedReturn_t = \sum_{j \ge 0} \DiscountFactor^\DiscountedReturnStepOffset \Reward_{\ControlStepIndex+\DiscountedReturnStepOffset}\) with discount factor \(\DiscountFactor \in [0, 1)\). Because \(\DiscountFactor\) applies once per decision, its horizon is measured in control steps, not seconds. At \(\DiscountFactor =\) 0.99 and one decision every 25 ms, the door-open reward earned at the end of an opening reaches the first decision with weight 0.04, and the effective horizon \(1/(1-\DiscountFactor)\) of 100 steps spans only 2.5 s. For the terminal success to keep half its weight across the whole opening, \(\DiscountFactor\) must be at least 0.9978. Changing the setpoint rate without re-deriving \(\DiscountFactor\) therefore changes the task horizon in seconds. Training a policy means choosing \(\ModelParameters\) to maximize the objective \(\ExpectedReturnObjective(\ModelParameters) = \mathbb{E}_{\Policy}[\DiscountedReturn_0]\), the return expected over whole episodes that start from the task’s initial states. It is a return to maximize, where the \(\Glyph{J}(\pi)\) of equation 5.1 was an expected cost to minimize.

Two expectations of the return organize every reinforcement learning method. The state value \(\PolicyRemainingReturn(\DecisionProcessState) = \mathbb{E}_\pi[\DiscountedReturn_t \mid \DecisionProcessState_{t} = \DecisionProcessState]\) is the return expected from \(\DecisionProcessState\) when following \(\pi\). The action value \(\PolicyHistoryActionReturn(\DecisionProcessState, \ContinuousActionVector) = \mathbb{E}_\pi[\DiscountedReturn_t \mid \DecisionProcessState_{t} = \DecisionProcessState, \ContinuousActionVector_t = \ContinuousActionVector]\) is the return expected after taking \(\ContinuousActionVector\) first and following \(\pi\) thereafter. Both obey the one-step Bellman relation \[\PolicyHistoryActionReturn(\DecisionProcessState, \ContinuousActionVector) = \Reward(\DecisionProcessState, \ContinuousActionVector) + \DiscountFactor\, \mathbb{E}_{\NextDecisionProcessState \sim \EnvironmentTransitionDistribution,\ \SuccessorActionVector \sim \pi}\left[ \PolicyHistoryActionReturn(\NextDecisionProcessState, \SuccessorActionVector) \right], \qquad \PolicyRemainingReturn(\DecisionProcessState) = \mathbb{E}_{\ContinuousActionVector \sim \pi}\left[ \PolicyHistoryActionReturn(\DecisionProcessState, \ContinuousActionVector) \right]. \tag{2}\] The advantage \(\PolicyHistoryAdvantage(\DecisionProcessState, \ContinuousActionVector) = \PolicyHistoryActionReturn(\DecisionProcessState, \ContinuousActionVector) - \PolicyRemainingReturn(\DecisionProcessState)\) measures whether an action does better or worse than the policy’s usual choice from that state. At the plate, an approach has positive advantage only if its expected return exceeds the policy’s usual choice. A force penalty can make a plunge less attractive, but the sign still depends on the reward weights, completion probability, and baseline policy.

Readers from control have met these objects under other names. With stage cost \(\ell = -\Reward\), the value is the negated cost-to-go, \(\CostToGo(\DecisionProcessState) = -\ExpectedRemainingReturn(\DecisionProcessState)\), and equation 2 is the dynamic-programming recursion. (Control texts write the cost-to-go as \(\Glyph{J}\); this chapter reserves \(\ExpectedReturnObjective(\ModelParameters)\) for the expected return a learner maximizes.) For linear dynamics in the plant state \(\MechanicalPlantState\) and quadratic cost, that recursion has the closed-form solution \(\CostToGo(\MechanicalPlantState) = \MechanicalPlantState^\top \LqrCostToGoMatrix \MechanicalPlantState\) that the linear-quadratic regulator computes from a Riccati equation (Linear quadratic regulator and algebraic Riccati equations). Reinforcement learning keeps the recursion and changes the estimation problem: the dynamics are unknown, so \(\ExpectedRemainingReturn\) and \(\ExpectedHistoryActionReturn\) are fitted from sampled transitions instead of solved from a model.

These quantities organize policy learning along two axes, what is estimated and where the data comes from. Fitting \(\ExpectedHistoryActionReturn\) over an immutable logged archive to select high-value actions is offline reinforcement learning (section 6.2.7). Generating fresh rollouts from \(\Policy\), using each batch for a few updates, and shifting probability toward positive-advantage actions is on-policy policy-gradient optimization (section 6.3). Between the two, value-based methods also run online and off-policy. The machine keeps collecting experience, but each transition enters a replay buffer and is reused across many value updates, which is how the demonstration-seeded methods of section 6.3.1 learn in hours. The value-fitting and policy-gradient derivations behind these approaches appear in Value Functions and Continuous Reinforcement Learning. Before paying the reward-design or physical trial cost of those methods, however, the simplest training regime bypasses value estimation altogether and fits a policy directly to the expert setpoints the collection campaign already recorded.

6.2.2 Behavioral cloning and its action targets

Behavioral cloning avoids designing a reward for the nominal opening by framing policy training, as ALVINN first did for road following (Pomerleau 1989), as supervised regression or classification over a collected dataset of expert state-action pairs \(\DemonstrationDataset = \{(\MultimodalObservation_{t}, \ContinuousActionVector_t)\}\). On a physical machine, however, the four-stream action taps record both the operator’s mapped setpoints and the setpoints the permission path actually admitted: \[\IntentDemonstrationDataset = \left\{ \left(\MultimodalObservation_{t}, \NominalCommandWindow\right) \right\} \quad \text{vs.} \quad \VerifiedDemonstrationDataset = \left\{ \left(\MultimodalObservation_{t}, \CertifiedCommandWindow\right) \right\}\] The mapped setpoint \(\NominalControlCommand\) is the record at the mapped-command tap \(\MappedCommandTap\), and the admitted setpoint \(\CertifiedControlAction\) is the record at the enforced-command tap \(\EnforcedCommandTap\). The four-stream action tap tuple develops what each tap teaches a policy. In brief, cloning admitted setpoints couples the weights to one permission path’s intervention profile and tuning—so an uncurated policy can learn to steer straight into the safety clamps and rely on the hardware filter to bend its commands back onto the latch—whereas cloning mapped setpoints keeps intent separate from enforcement at the price of more frequent runtime interventions, and neither tap yields an expert target without curation. The latch campaign took the mapped command as its target wherever the permission path admitted it unchanged, so latch-bc-01 clones \(\NominalControlCommand\) on those steps and treats every modified command as an attributed intervention.

Whichever tap supplies the target, the training loss compares the proposed setpoints with the recorded ones, over a single step or over the \(\ActionChunkHorizon\)-step sequence that section 6.2.5 calls a chunk. Because this loss penalizes only the discrepancy between the commanded action and the recorded demonstration at observed states, the training objective does not evaluate task completion, terminal cost, or the stability of the closed-loop dynamical system. It fits actions to observations only, and that point-by-point fit breaks down at a single observation whenever demonstrators solve the same physical geometry in more than one valid way.

6.2.3 Mode averaging when demonstrations split

A mobile manipulator trained with least-squares regression on demonstrations that pass left of a central structural column half the time and right of it the other half will steer straight down the center line into the column. Each human teleoperator trajectory in the recorded archive \(\DemonstrationDataset\) is dynamically valid and collision-free, and not a single demonstration drives into the obstacle, yet a deterministic point-regression policy proposes the one path that every demonstrator avoided.

Under empirical mean squared error (\(\ell_2\)) loss, a deterministic behavioral cloning network cannot represent both modes for the same observation \(\MultimodalObservation_{t}\) and instead optimizes: \[\MeanSquaredErrorCloningLoss(\ModelParameters) = \mathbb{E}_{(\MultimodalObservation_{t}, \ContinuousActionVector) \sim \DemonstrationDataset} \left[ \left\| \Policy(\MultimodalObservation_{t}) - \ContinuousActionVector \right\|_2^2 \right].\] The theoretical minimizer of this objective is the conditional expectation of the action distribution (derived formally in Mode averaging and the decoders that avoid it): \[\pi^*(\MultimodalObservation_{t}) = \arg\min_{\CandidateActionVector} \int \|\CandidateActionVector - \ContinuousActionVector\|_2^2\,\DemonstrationArchiveActionDistribution(\ContinuousActionVector \mid \MultimodalObservation_{t})\,\DerivativeOperator\ContinuousActionVector = \mathbb{E}[\ContinuousActionVector \mid \MultimodalObservation_{t}] = \int \ContinuousActionVector\,\DemonstrationArchiveActionDistribution(\ContinuousActionVector \mid \MultimodalObservation_{t})\,\DerivativeOperator\ContinuousActionVector. \tag{3}\] In conventional computer vision, predicting the conditional mean of ambiguous pixel targets generates blurry, averaged textures; in natural language processing, it produces diffused semantic embeddings. In Physical AI, however, the conditional expectation of two collision-free evasion commands (such as \(\LeftEvasionAction = [\BaseForwardSpeed, +\YawRate]^\top\) and \(\RightEvasionAction = [\BaseForwardSpeed, -\YawRate]^\top\), where \(\BaseForwardSpeed\) is forward linear speed and \(\YawRate\) is angular yaw rate) evaluates to: \[\mathbb{E}[\ContinuousActionVector \mid \MultimodalObservation_{t}] = \frac{1}{2}\LeftEvasionAction + \frac{1}{2}\RightEvasionAction = \left[ \BaseForwardSpeed, 0.0 \right]^\top.\] The regression mean proposes forward motion with zero lateral evasion. If admitted unchanged, that proposal heads straight into the central obstacle.³

³ Jensen’s Inequality: The averaged action fails even where it happens to be admissible, because a nonlinear plant does not map the mean action to the mean outcome. In general \(\mathbb{E}[\PlantOutcomeFunction(\ContinuousActionVector)] \ne \PlantOutcomeFunction(\mathbb{E}[\ContinuousActionVector])\), and for a convex \(\PlantOutcomeFunction\) Jensen’s inequality fixes the sign of the gap. Contact and friction are the most nonlinear parts of a manipulator’s plant, so a regression mean that looks like a compromise between two good demonstrations can produce a contact state that neither demonstration visited.

This invalid proposal is not caused by an optimization pathology, insufficient model capacity, or dataset scarcity. It is the exact minimizer of a point-regression loss over a multimodal distribution. Viewed as a function of the proposed action, the expected loss is a bowl with a single bottom at the mean, and that bottom sits in the zero-density gap between the two demonstrated modes (\(\DemonstrationArchiveActionDistribution(\mathbb{E}[\ContinuousActionVector\mid\MultimodalObservation_{t}] \mid \MultimodalObservation_{t}) \approx 0\)). The statement concerns the optimal output, not whether optimizing the network’s weights is convex.

Definition 6.1: Mode averaging

Mode averaging is the failure in which a policy trained with a point-regression loss on demonstrations that contain several valid actions for one observation proposes their conditional mean, an action that lies between the demonstrated modes and that no demonstrator chose.

Significance: More data, capacity, or training moves the output toward the mean, not away from it. Two equally weighted evasions at \(\pm\YawRate\) average to zero yaw, a proposal aimed at the obstacle both demonstrations avoided, and it reaches the permission path as a smooth, confident command whose consequence cannot be undone once admitted.
Distinction: Mode averaging concerns the action chosen at one observation; compounding error concerns the states that successive actions reach. A multimodal decoder removes the first and does nothing for the second, which needs recovery experience.
Common pitfall: Reading a low validation loss as evidence against averaging. The mean minimizes squared error on held-out demonstrations drawn from the same modes, so a regression head can score best on its split and still propose the invalid middle action at every bifurcation.

Changing the regression penalty does not by itself represent alternative actions. Mean absolute error (\(\ell_1\)) admits conditional medians, while Huber loss limits the influence of large residuals, so neither guarantees a unique, coherent route through a multimodal demonstration distribution. Instead, the decoder must represent the alternatives, and execution must maintain a consistent choice across successive observations.

Several decoder families represent these alternatives at different computational costs. Mixture Density Networks (MDNs)⁴ parameterize a Gaussian Mixture Model (GMM): \(\DemonstrationArchiveActionDistribution(\ContinuousActionVector \mid \MultimodalObservation_{t}) = \sum_{\MixtureComponentIndex=1}^{\MixtureComponentCount} \pi_\MixtureComponentIndex(\MultimodalObservation_{t}) \GaussianDistribution\left(\ContinuousActionVector \mid \MixtureComponentMean(\MultimodalObservation_{t}), \MixtureComponentCovariance(\MultimodalObservation_{t})\right)\), in which \(\pi_\MixtureComponentIndex\) is the weight of component \(\MixtureComponentIndex\) (not the policy), and \(\MixtureComponentMean\) and \(\MixtureComponentCovariance\) are its mean (not a friction coefficient) and covariance. They produce mixture parameters in one forward pass, but training must prevent covariance collapse (\(\MixtureComponentCovariance \to \ZeroMatrix\)) and loss of less frequent modes, which makes the component count and covariance representation consequential for high-dimensional chunks (\(\ChunkActionDimension = \ActionDimension \times \ActionChunkHorizon\)). Energy-based models instead score each candidate with a learned energy and must search for a low-energy action, so their iteration budget, rather than a head of fixed size, must fit the proposal deadline.

⁴ Mixture Density Network: Bishop (1994) proposed it for inverse problems, whose inputs admit several valid outputs, and demonstrated it on the inverse kinematics of a robot arm, where one hand position has two joint solutions and least-squares regression returns their invalid average. The mobile manipulator’s two routes around the column are the same problem, met three decades later in action space, which is why the remedy is still a change of output distribution rather than a larger network.

Bishop, Christopher M. 1994. Mixture Density Networks. NCRG/94/004. Neural Computing Research Group, Aston University.

Ho, Jonathan, Ajay Jain, and Pieter Abbeel. 2020. “Denoising Diffusion Probabilistic Models.” Advances in Neural Information Processing Systems (NeurIPS) 33: 6840–51.

Generative architectures, including diffusion models (Ho et al. 2020) and flow matching (Lipman et al. 2022), offer another way to represent \(\DemonstrationArchiveActionDistribution(\ContinuousActionVector \mid \MultimodalObservation_{t})\) by training a model to refine a noisy action candidate toward the learned distribution rather than returning its conditional mean. Sampling can then select a coherent demonstration mode (\(\LeftEvasionAction\) or \(\RightEvasionAction\)). Although representing multiple modes resolves the averaging problem, a generated candidate can still violate clearance or actuator constraints, so the permission path must check it before execution.

A mode boundary in a single-step Markov policy can also produce high-frequency temporal chattering. In a 50–100 Hz feedback loop, microscopic sensor noise drives consecutive observations back and forth across the mode boundary, causing a single-step controller mapping \(\MultimodalObservation_{t} \to \ContinuousActionVector_{t}\) to alternate between opposing actuator efforts (for example, +10 N·m and −10 N·m). This alternation excites drivetrain resonances, accelerates gear tooth fatigue, and drives motor winding temperatures toward their thermal limit. Even when a multimodal decoder commits to one route and avoids chattering across the boundary, however, a second imitation failure emerges as soon as a small tracking error pushes the robot outside the states the demonstrators visited.

6.2.4 Compounding errors and recovery experience

A policy can choose a coherent route around the column and still drift into states its demonstrations never covered. Mode averaging concerns the action selected at one observation; compounding error concerns the observations created by successive actions. Keeping these failures separate prevents a richer action decoder from being mistaken for recovery experience. Cloning fits actions only at the states the demonstrators visited, whereas deployment visits the states the policy’s own actions create, and a fixed archive holds no example of the states the policy’s execution errors reach. In continuous physical state spaces, this decoupling produces compounding covariate shift, which is the second law (endogenous data, principle 2) acting on a fixed archive.

Recovery and Compounding Error bounds the price of this shift: fixed-archive cloning admits a worst-case imitation cost that grows as \(\BehaviorCloningHorizon^2\SingleStepImitationError\) over an \(\BehaviorCloningHorizon\)-step horizon (equation 5.1). That bound counts accumulated task cost, not displacement, and the physical drift behind it depends on what the action commands and on the feedback below the policy. A persistent error in a commanded velocity integrates once into position, so the drift grows linearly in time; a persistent error in a commanded torque integrates twice, so it grows quadratically. A velocity-level policy also sits above a joint velocity loop that rejects load disturbances between its decisions, whereas a torque-level policy has no such loop beneath it and must supply every correction itself. Offline validation on demonstration splits measures prediction accuracy on the demonstrated distribution; recovery after displacement requires additional closed-loop evidence.

The regime that supplies that recovery experience is interactive imitation learning, of which Dataset Aggregation (DAgger) (Ross et al. 2011) is the classic form. In DAgger, an expert labels the states the learner itself visits and the policy retrains on the aggregate. Intervention and Recovery Data develops DAgger, with its guarantee of a cost that grows linearly in the horizon (equation 5.8) and the recoverability constant \(\DeviationDisadvantage\) on which that guarantee depends. The constant bounds how much a single deviation can raise the expert’s remaining cost, so it is an assumption about the plant, not a property of the learner. On physical hardware, this theoretical recoverability depends directly on joint torque authority and contact friction limits.

On hardware, collecting those labels requires a recoverable rollout and an expert who has enough information to choose a correction. Because a camera log may support a path label while leaving contact force or joint torque ambiguous, live supervision supplies richer feedback, yet the machine continues moving during human reaction and control handoff. Consequently, intervention must begin within the available stopping margin, and the handoff must avoid discontinuous setpoints.

↪ The bumpless transfer that keeps a human takeover from stepping the command is developed in Transfer Without Discontinuity.

Expert availability also does not imply physical recoverability. The machine may lack enough actuator effort or frictional grip to return from a deviation. The simplified force balance \(\GeneralizedInertiaMatrix\GeneralizedAccelerations = \JointTorque + \ExternalContactTorque\) is Newton’s second law written per joint: \(\GeneralizedCoordinates\) denotes joint positions, \(\GeneralizedInertiaMatrix\) inertia, \(\JointTorque\) actuator torque, and \(\ExternalContactTorque\) the torque that contact forces exert about the joints; other forces are omitted here for clarity, and Rigid-body dynamics and continuous collision checks develops the full equation. Actuator limits require \(|\CommandedJointTorque_i| \le \JointTorqueLimit\) on every joint axis, while available friction bounds tangential force by \(\|\TangentialContactShear\| \le \CoulombFrictionCoefficient \NormalContactForce\), with normal force \(\NormalContactForce\) and friction coefficient \(\CoulombFrictionCoefficient\). These constraints can leave no action that restores the task, and then \(\DeviationDisadvantage\) is unbounded and DAgger’s linear guarantee no longer applies. Interactive aggregation is therefore most useful where learner deviations can be contained, labeled, and repeated within the operator-time and hardware-exposure budget. The state space divides into three regions (figure 1): the tube of demonstrated states where cloning trains, a band just outside it where an expert label on a learner-visited state still restores the task, and an unrecoverable region beyond, where actuator and friction limits leave no restoring action and every further label is vacuous. Interactive aggregation buys experience only in the middle band.

Schematic state-space plot with task progress on the horizontal axis and deviation from the demonstrated path on the vertical axis. A shaded band surrounds a solid expert path. A dashed learner path leaves the band and continues upward into a red region at the top. At one point just outside the band, a marker labeled DAgger label has an arrow pointing back toward the band. — Figure 1: **Drift out of the demonstrated states**: Behavioral cloning trains only inside the shaded tube the expert visited. Once the learner leaves it, each action is taken from an unfamiliar state and the error compounds. DAgger labels a learner-visited state outside the tube, which supplies recovery experience while a return is still possible. Past the red boundary, actuator and friction limits leave no action that restores the task, so no label can help. The sketch is schematic; its axes carry no units.

For the latch, these distinctions select a concrete next training step. The stipulated follow-up campaign in Intervention and Recovery Data supplies latch-recovery-02, a versioned archive of expert recovery from the positively displaced plate at the guarded approach. The recovery candidate latch-bc-recovery-02 trains on the original nominal demonstrations and the stabilized, reviewed segment of each correction; the enforcer’s blend and the operator’s re-orientation after takeover stay masked. Learner commands that preceded takeover remain failure evidence unless an expert supplied replacement labels; they do not become imitation targets merely because the episode eventually succeeded. A multimodal chunk decoder can preserve a coherent back-off and re-seating sequence, while the additional demonstrations supply the recovery states that changing the decoder alone could not provide.

↪ Causal intervention masking, which admits only reviewed recovery segments as imitation targets, is enforced in Causal masking and recovery eligibility.

The training decision is complete, but the improvement claim remains to be tested. Entire collection sessions are held out for nominal and recovery checks, so neighboring frames from one run do not appear on both sides of the split. Training diagnostics check whether the candidate represents the demonstrated correction; qualification must then test whether it completes the door task under the declared perturbation without regressing nominal behavior. The candidate carries that request to Qualifying a recovery update, including task completion, cycle time, proposer intervention, and protective outcomes as separate measures. This is a stipulated engineering branch, not a report of collected data or measured improvement. The negative displacement and faster approach remain unsupported, and even on the supported path, emitting setpoints one tick at a time leaves the arm vulnerable to the mode-boundary chattering of section 6.2.3 unless the decoder predicts a multi-step action sequence in one pass.

6.2.5 Action chunking and multi-step trajectory prediction

Action chunking (definition 3.2) suppresses both high-frequency actuator chattering and single-step mode oscillation by predicting an entire sequence of future actions simultaneously rather than re-querying the decoder at every 25 ms control tick. Instead of regressing a single action \(\ContinuousActionVector_{t}\) from current observation \(\MultimodalObservation_{t}\), a chunking policy predicts a setpoint sequence \(\ActionChunk_t \in \mathbb{R}^{\ActionChunkHorizon \times \ActionDimension}\) across a prediction horizon of \(\ActionChunkHorizon\) future timesteps and \(\ActionDimension\) action dimensions, which the execution path then interpolates into continuous motion and checks for feasibility. In supervised behavioral cloning, the policy minimizes a multi-step trajectory loss over demonstrated trajectories: \[\ChunkImitationLoss(\ModelParameters) = \mathbb{E}_{(\MultimodalObservation_{t}, \ActionChunk_t^*) \sim \DemonstrationDataset} \left[ \sum_{i=0}^{\ActionChunkHorizon-1} \ell_\HuberLossThreshold\left( \Policy(\MultimodalObservation_{t})_i - \ContinuousActionVector^*_{t+i} \right) \right],\] where \(\ActionChunk_t^* = [\ContinuousActionVector_t^*, \dots, \ContinuousActionVector_{t+\ActionChunkHorizon-1}^*]^\top\) represents the recorded demonstration chunk and \(\Policy(\MultimodalObservation_{t})_i\) is the \(\ChunkStepIndex\)-th predicted future setpoint. Actions are expressed in declared normalized coordinates. For a vector residual \(\ActionResidualVector\), the chunk objective uses the componentwise sum \(\ell_\HuberLossThreshold(\ActionResidualVector) = \sum_{j=1}^{\ActionDimension}\ell_\HuberLossThreshold(\NormalizedActionResidual_j)\), where \(\NormalizedActionResidual_j\) is the matching component of \(\ActionResidualVector\), with equal component weights; the scalar Huber penalty below acts on each normalized prediction error \(\NormalizedActionResidual_j\), not on the reward of section 6.2.1: \[\ell_\HuberLossThreshold(\NormalizedActionResidual) = \begin{cases} \frac{1}{2} \NormalizedActionResidual^2 & \text{for } |\NormalizedActionResidual| \le \HuberLossThreshold, \\ \HuberLossThreshold \left(|\NormalizedActionResidual| - \frac{1}{2} \HuberLossThreshold\right) & \text{otherwise.} \end{cases}\] (ACT (Zhao et al. 2023), the architecture of both latch policies, uses an \(\ell_1\) reconstruction penalty in place of \(\ell_\HuberLossThreshold\) and adds a conditional variational autoencoder (CVAE) term \(\LatentKlWeight \KlDivergence(\LatentPosteriorDistribution(\LatentStyleVariable \mid \ActionChunk_t^*, \MultimodalObservation_{t}) \parallel \GaussianDistribution(\LatentStyleVariable))\) against a standard normal prior, which lets a latent variable absorb differences in style across demonstrations.)

⁵ Huber Loss: Huber (1964) introduced it as an estimator of location that tolerates a small fraction of grossly wrong observations, quadratic near the center and linear in the tails, and showed that it minimizes the worst-case variance over such contaminated distributions. Teleoperation tremor and dropouts are exactly that contamination, and \(\HuberLossThreshold\) sets, in the declared normalized action coordinates, where an error stops counting as noise and starts counting as a gross error.

Huber, Peter J. 1964. “Robust Estimation of a Location Parameter.” The Annals of Mathematical Statistics 35 (1): 73–101. https://doi.org/10.1214/aoms/1177703732.

Because teleoperated setpoints can contain outliers caused by tremor, tracking errors, or dropouts, squared loss gives a single large residual disproportionate influence over the fit. Huber loss⁵ is quadratic for small residuals (\(|\NormalizedActionResidual| \le \HuberLossThreshold\)) and linear beyond them, so its gradient saturates at magnitude \(\HuberLossThreshold\) and no single outlier can dominate the fit. Even so, the threshold acts on prediction residuals rather than on the velocity, acceleration, or jerk of the resulting trajectory, which means Huber loss alone establishes neither smooth motion nor actuator feasibility; those properties require explicit constraints and checks on the proposed chunk.

Predicting \(\ActionChunkHorizon\) future actions at once creates opportunities for more coherent execution, provided training and runtime checks preserve them:

Temporal and Dynamic Consistency: Jointly predicting adjacent actions lets the model learn their relationships. The resulting sequence still requires checks for abrupt changes and actuator limits.
Preserving Modal Commitment: A chunk can represent one coherent choice, such as passing left of the column, across adjacent actions. Predicting a sequence does not itself ensure that the decoder selects a single valid mode.
Receding-Horizon Replanning and Temporal Ensembling: The deployed policy queries a fresh chunk every \(\ChunkExecutionHorizon\) steps (\(\ChunkExecutionHorizon < \ActionChunkHorizon\)), permitting feedback correction before the prediction horizon ends. Blending overlapping predictions is optional and must preserve a coherent mode and feasible seam.

The modal commitment that the chunk objective buys in training can be undone at execution. If stochastic sampling selects the left route at one query and the right route at the next, blending the overlapping chunks across the seam averages the two routes and resurrects the mode averaging that action chunking was chosen to eliminate (Black, Galliker, et al. 2025).

Black, Kevin, Manuel Y. Galliker, and Sergey Levine. 2025. “Real-Time Execution of Action Chunking Flow Policies.” Advances in Neural Information Processing Systems (NeurIPS).

↩︎ Temporal ensembling of overlapping chunks, and real-time chunking, which freezes the committed prefix and generates the rest of the next chunk to agree with it, are established in Action chunking.

Zhao, Tony Z., Vikash Kumar, Sergey Levine, and Chelsea Finn. 2023. “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware.” Robotics: Science and Systems (RSS). https://doi.org/10.15607/RSS.2023.XIX.016.

Chunking changes how often the policy replans, but does not supply recovery states absent from the demonstrations. Executing \(\ChunkExecutionHorizon\) actions between queries reduces the number of query boundaries to \(\QueryBoundaryCount = \BehaviorCloningHorizon/\ChunkExecutionHorizon\) while leaving disturbances within each execution window to the lower-level feedback system. This count alone does not convert the imitation-learning cost bound \(\mathcal{O}(\BehaviorCloningHorizon^2 \SingleStepImitationError)\) into a task-survival curve, the probability that an episode reaches the goal without entering an unrecoverable state, plotted against the horizon. ACT (Zhao et al. 2023) and receding-horizon diffusion (Chi et al. 2025) change temporal execution; DAgger (Ross et al. 2011) changes the training distribution by obtaining labels on learner-visited states. These are separate ways to improve behavior, and their closed-loop benefits must be measured on the task. Neither the drift sketch of figure 1 nor the imitation-learning cost bound establishes a task-survival probability or ranks learning algorithms.

Increasing dataset size, model capacity, or tuning can reduce the one-step error rate on evaluated observations and therefore lower the chance of a first policy error during a fixed horizon. That improvement does not characterize what happens after an error. An external disturbance, uncalibrated payload, or voltage sag can also move the machine into weakly supported states without a policy disagreement. Recovery data, measured dynamics, and independent runtime limits determine whether either event remains within the physical envelope.

Drift off the demonstrated states raises no alarm of its own, so detecting it before physical contact cannot rely on the offline training loss. A feature-distance or uncertainty monitor can flag unfamiliar observations, but its threshold needs validation against missed detections and false alarms. For the latch, an independently measured plate position and approach condition provide a more direct check of the declared cells. A purely geometric monitor is weaker. One that accepts any state inside the convex hull of the demonstrated coordinates, a convex stand-in for the support hull of Collection Policy Coverage, still admits intermediate states that no demonstration visited, and nothing guarantees that those states are physically admissible or dynamically stable. Predicting an entire \(\ActionChunkHorizon\)-step chunk in a multimodal decoder solves the single-step averaging problem, but only if the hardware can finish evaluating that decoder before the next chunk is due.

6.2.6 Decoder latency and proposal deadlines

Selecting a coherent route around an obstacle or into a latch does no good if the decoder finishes computing the trajectory after the actuator deadline has already passed. A decoder converts the model’s representation into an action or action sequence; iterative decoders repeatedly refine a candidate before returning it. Their equations describe how a candidate is generated, while the execution decision depends on the wall-clock time those dependent model evaluations consume. The central comparison is between measured proposal latency and the deadline of the buffer that feeds the permission path.

The decoders that fix mode averaging buy that fix with time. A generative head escapes the regression mean by sampling one demonstrated route, but it reaches its sample through a chain of dependent refinements, and each refinement is another model evaluation that must finish before the next proposal is due (figure 2). The proposal then still has to cross from the application processor to the safety microcontroller’s own buffer before the permission path can check it, so the decoder’s time comes out of a budget it shares with transport and admission.

↪ The placement of the multi-rate proposal and permission paths on the application processor and the safety microcontroller is developed in Two Paths and the Resources They Share.

Figure 2: Generative policy architectures and action chunk denoising: (a) Sampling represents alternative demonstrated routes instead of their direct-regression mean. (b) Iterative refinement produces a candidate action chunk whose clearance and actuator feasibility still require checking. (c) On the machine’s schedule, proposals renew every 50 ms while setpoints play every 25 ms, and the bars compare the deployed ACT decoder with an illustrative diffusion head that leaves far less of each renewal period to spare. An admitted chunk crosses the black-channel link into the safety MCU’s own double buffer and is checked on every permission tick before the drives act. Buffering alone guarantees neither freshness nor freedom from starvation.

The decoder families themselves, from binned and compressed action tokens to parallel heads and iterative action experts, are introduced in Learned Representations. What matters for training is their dependency structure. A diffusion policy (Chi et al. 2025) or a flow-matching expert (Lipman et al. 2022; Black et al. 2024) refines a candidate chunk through evaluations that each consume the previous candidate, and an autoregressive vision-language-action policy (Kim et al. 2024) generates its action tokens one after another, so in each case the dependent evaluations run in sequence. None of these constructions guarantees collision-free or actuator-feasible output, and the diffusion and flow-matching objectives are developed in Continuous trajectory diffusion and flow matching. Some post-training recipes remove the token sequence from the run-time path altogether by training a separate action expert (section 6.2.8). For the latch decision, the relevant consequence is how many dependent evaluations yield adequate action quality, and the quantity that decides the execution arrangement is complete proposal latency, perception included, measured against the actual proposal deadline.

Lipman, Yaron, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nicklas, and Matt Le. 2022. “Flow Matching for Generative Modeling.” arXiv Preprint arXiv:2210.02747.

The number and dependency of decoder evaluations constrain proposal latency; the implementation and loaded hardware determine its measured value. The total wall-clock duration of a single policy inference cycle \(\ProposerInferenceTime\) is the sum of perceptual feature extraction and iterative generative trajectory sampling: \[\ProposerInferenceTime = \ObservationEncodingTime(\MultimodalObservation_{t}) + \sum_{k=1}^{\DecoderEvaluationCount} \DecoderPassTime\left(\NoisedActionChunk, \DiffusionStepIndex\right) \le \CompletionDeadline, \tag{4}\] where \(\ObservationEncodingTime\) is the visual backbone and proprioceptive encoding latency, \(\DecoderEvaluationCount\) is the number of dependent decoder evaluations (denoising or ODE solver steps), \(\DecoderPassTime\) is the latency of one forward pass of the action head (one denoising or solver step, not the whole-model evaluation time \(\ProposerInferenceTime\) of equation 3.4), \(\NoisedActionChunk\) is the candidate chunk that the \(\DiffusionStepIndex\)-th evaluation refines, and \(\CompletionDeadline\) is the real-time execution deadline.

The encoder runs once per proposal, whereas each dependent decoder evaluation adds work. Action chunking prices that work for each decoder family on the machine’s processor, where re-reading a large action head on every evaluation can dominate it. Training decides the two quantities those prices depend on, the head and its evaluation count \(\DecoderEvaluationCount\), so the question here is whether a head chosen for its multimodal fit also meets the latch’s deadlines.

The latch policies avoid repeated evaluations: latch-bc-01 decodes each chunk with the machine’s 80M-parameter ACT network in one pass, within the 40 ms p99 inference target. Consider instead replacing its decoder with an illustrative diffusion head, the kind that multimodal latch demonstrations might motivate, sampled in \(\DecoderEvaluationCount =\) 16 steps by the denoising diffusion implicit model (DDIM) sampler, a deterministic sampler whose step count sets the number of dependent evaluations. Stipulate a visual encoding latency of \(\ObservationEncodingTime =\) 8 ms and a total inference duration \(\ProposerInferenceTime \approx\) 48 ms once all evaluation work is included. These stipulated inputs define the comparison below; a measurement of the real head replaces them, and that measurement can only sit at or above the weight-streaming floor of The memory wall.

Only the 4-step flow decoder beats the setpoint; DDIM leaves 2 ms of its renewal.

Two deadlines apply, the setpoint step for a decoder that must supply every tick itself and the renewal period for one that runs beside an admitted prefix, and the lease rule of Coupling the distance, lease, and supply budgets adds an allowance for renewal jitter. That allowance belongs to the rule’s compute side, the renewal period plus jitter that the lease must exceed. Transport across the black-channel link, the permission tick, and the age of the evidence at admission are charged on its other side, against the stopping budget, so the slack computed below is the decoder’s alone; a slower decoder pays on that side too, because its inference time ages the evidence it is admitted on.

Napkin Math 6.1: The renewal slack

Problem: Can the stipulated diffusion head replace the one-pass ACT decoder of latch-bc-01, either synchronously or asynchronously, and how much of a stretched renewal can each decoder absorb?

Variables:

Setpoint step \(\SetpointStepInterval =\) 25 ms (chosen) and renewal period \(\ChunkRenewalPeriod =\) 50 ms (derived, two steps)
ACT inference 40 ms at p99, including vision (illustrative target)
Diffusion head: \(\DecoderEvaluationCount =\) 16, \(\ObservationEncodingTime =\) 8 ms, \(\ProposerInferenceTime =\) 48 ms (illustrative, stipulated)
Flow-matching candidate: \(\DecoderEvaluationCount =\) 4 at the same per-evaluation cost (illustrative)
Renewal-jitter allowance 5 ms (chosen, lease rule)

Math:

Per-evaluation cost (equation 4): \(\DecoderPassTime =\) (48 ms \(-\) 8 ms)/16 \(=\) 2.5 ms.
Synchronous step: The diffusion head’s 48 ms exceeds the 25 ms step. The flow candidate needs 8 ms \(+\) 4 \(\times\) 2.5 ms \(=\) 18 ms and fits it.
Renewal slack: ACT leaves 50 ms \(-\) 40 ms \(=\) 10 ms; the diffusion head leaves 50 ms \(-\) 48 ms \(=\) 2 ms.
Stretched renewal: 48 ms \(+\) 5 ms \(-\) 50 ms \(=\) 3 ms past the renewal for the diffusion head, while ACT ends at 40 ms \(+\) 5 ms, inside the period.

Result: The diffusion head can run only asynchronously, and a single jitter-sized stretch carries it past its renewal.

Systems insight: Renewal slack, rather than mean inference latency, dictates real-time schedulability. When a decoder’s slack drops below the jitter allowance, a single tail-latency stretch carries it past its renewal, and the active prefix keeps playing on evidence that grows older with every tick. The lease contract, not the neural decoder, then sets the hard safety bound on how long an admitted prefix may continue actuating the arm against aging sensor evidence.

The notebook separates sampler choice from execution authority. The flow candidate becomes a viable synchronous choice only if action quality, scheduling, transfer, and admission also fit, and a faster decoder does not remove the permission check. Its evaluation count is the quantity to compare, not its step count. A higher-order solver can use fewer integration steps while performing more network evaluations; a standard fourth-order Runge-Kutta (RK4) solver evaluates four stages per step, so compare that total work with the single-stage alternative before choosing it. Step distillation, which trains a sampler to reach the same chunk in fewer evaluations, or a different solver is useful when it reduces measured latency without losing the recovery behavior the latch needs.

The slower candidate can still generate chunks asynchronously against the 50 ms renewal cadence while the execution path plays an admitted prefix, provided loaded timing evidence confirms the schedule. Because generating an action chunk supplies candidate trajectory setpoints without conferring authorization to execute that entire window unchecked, the lease selected in Chunk Lease Sizing from the Stopping Budget bounds active permission separately from renewal cadence and evidence age. Execution renews only upon receiving fresh sensor evidence paired with a valid replacement proposal, so stamping a new timestamp header over stale sensor buffers cannot extend execution authority, and an invalid new proposal leaves an active prefix in place only while that prefix’s own state, evidence, and lease remain valid; once the lease expires, the execution path invokes a state-dependent, physically qualified fallback rather than an unspecified emergency stop.

↪ The ladder of resident protective responses from which that fallback is selected is developed in The Fallback Ladder.

On the renewal schedule the two decoders differ in slack, not in mean latency (figure 3). When a stretched diffusion run misses its renewal boundary, nothing in the decoder notices; the chunk lease is the only mechanism that bounds how long the previous prefix keeps playing.

Timing strip over 100 milliseconds with five rows. The top row marks setpoint ticks every 25 milliseconds; dashed lines mark renewal boundaries every 50 milliseconds. Below the setpoint row, bars show ACT inference with a short slack gap before each renewal, diffusion inference with a shorter gap, and diffusion inference stretched so that a red segment extends past the renewal boundary. The bottom row shows the lease interval extending beyond the first renewal. — Figure 3: **Decoder latency on the renewal schedule**: Setpoints play every 25 ms and proposals renew every 50 ms. The deployed ACT decoder finishes with 10 ms to spare; the illustrative diffusion head leaves 2 ms, and a 5 ms stretch pushes it 3 ms past the renewal. The lease bounds how long the active prefix may play if a renewal is missed.

Checkpoint 6.1: Mode choice, drift, and decoder deadlines

Cloning fails in two separate ways, and the decoder chosen to repair one of them must still meet the machine’s schedule.

Diagnosing the failure

Mode averaging vs. compounding error: A cloned navigation policy sometimes steers straight at the column. A cloned latch policy, on other runs, drifts until the gripper binds on a sagged door. Can you say which symptom a multimodal decoder addresses, which one needs recovery demonstrations, and why neither remedy supplies the other?
Recoverability: Can you explain why DAgger’s linear guarantee fails at a state where actuator torque limits and the friction cone leave no restoring action, however many expert labels are collected?

Checking the deadline

Evaluation budget: With the stipulated 8 ms encoder and 2.5 ms per evaluation, what is the largest evaluation count that fits the 25 ms setpoint step, and the largest that fits the 50 ms renewal while leaving the 5 ms jitter allowance free?
Renewal vs. lease: Can you explain why a new proposal header cannot renew stale observations, and what the execution path must do when the lease expires with no valid replacement?

6.2.7 Offline reinforcement learning and out-of-distribution actions

The decoder budget settles how fast a policy trained on recorded data can propose actions; the next question is what happens when the recorded archive mixes clean door openings with hesitant teleoperation, scripted routines, and aborted trials. Behavioral cloning copies every trajectory in such a log with equal supervised weight, good and bad alike, yet running standard Q-learning (Watkins and Dayan 1992) on the same static archive to cherry-pick the high-reward segments triggers a worse failure: the value network invents high returns for actions never tried in the log, and the deployed policy greedily selects ungrounded maneuvers that stall a joint or strike a fixture boundary. Offline reinforcement learning (Batch RL) applies the return and value quantities of section 6.2.1 to a fixed transition log \(\DemonstrationDataset = \{(\DecisionProcessState_{t}, \ContinuousActionVector_t, \Reward_t, \DecisionProcessState_{t+1})\}\) without active hardware exploration, using rewards to stitch together high-performing composite trajectories from disjoint, imperfect demonstrations while suppressing out-of-distribution (OOD) action overestimation (Kumar et al. 2020).

In standard Q-learning, the algorithm estimates the value of future states by asking its value network to predict the highest possible return it could achieve from the next step, which is equation 2 with a maximum over next actions in place of the policy’s expectation. That maximum is taken over the entire continuous action space. For actions that were never executed in the logging dataset, the network has received no direct value-fitting target and often outputs erroneously large value estimates due to generalization variance, predicting a high return for an untried physical maneuver. The algorithm greedily selects these spurious peaks, propagating over-optimistic value errors backwards until the deployed policy commands unverified motions. Because the distinction from cloning is the selection rule \(\pi(\DecisionProcessState) = \arg\max_{\ContinuousActionVector} \ExpectedHistoryActionReturn(\DecisionProcessState, \ContinuousActionVector)\), offline learning needs a way to limit that optimism without obtaining new transitions from the machine.

To reduce value overestimation without querying the physical environment, Conservative Q-learning (CQL) adds a conservative penalty to value fitting (Kumar et al. 2020). The objective below has two parts. The bracketed term discourages optimism about unsupported actions, and the squared term fits the reward observed in logged transitions together with estimated future reward. Here \(\ContinuousActionSpace\) is the action space, \(\ExpectedHistoryActionReturn\) the value function being fitted, \(\TargetActionReturnEstimate\) a fixed target estimate during this update, and \(\ConservativePenaltyWeight\) the penalty weight: \[\min_\ExpectedHistoryActionReturn \ConservativePenaltyWeight \cdot \mathbb{E}_{\DecisionProcessState \sim \DemonstrationDataset} \left[ \ln \int_{\ContinuousActionSpace} \exp(\ExpectedHistoryActionReturn(\DecisionProcessState, \ContinuousActionVector))\,\DerivativeOperator\ContinuousActionVector - \mathbb{E}_{\ContinuousActionVector \sim \DemonstrationDataset}\left[ \ExpectedHistoryActionReturn(\DecisionProcessState, \ContinuousActionVector) \right] \right] + \frac{1}{2} \mathbb{E}_{(\DecisionProcessState, \ContinuousActionVector, \Reward, \NextDecisionProcessState) \sim \DemonstrationDataset} \left[ \left( \ExpectedHistoryActionReturn(\DecisionProcessState, \ContinuousActionVector) - \EmpiricalBellmanOperator^* \TargetActionReturnEstimate(\DecisionProcessState, \ContinuousActionVector) \right)^2 \right],\] where \(\EmpiricalBellmanOperator^* \TargetActionReturnEstimate(\DecisionProcessState, \ContinuousActionVector) = \Reward(\DecisionProcessState, \ContinuousActionVector) + \DiscountFactor \max_{\SuccessorActionVector} \TargetActionReturnEstimate(\NextDecisionProcessState, \SuccessorActionVector)\) is the empirical Bellman target operator and \(\ConservativePenaltyWeight > 0\) balances value conservatism against Bellman error minimization.⁶

⁶ Scope of the CQL Bound: Under its data-coverage, optimization, and function-approximation assumptions, CQL constructs a conservative estimate whose expected value for a policy lower-bounds that policy’s true value (Kumar et al. 2020). This is not a pointwise bound for every state-action pair or every dataset, and expected return does not certify physical safety.

Kumar, Aviral, Aurick Zhou, George Tucker, and Sergey Levine. 2020. “Conservative q-Learning for Offline Reinforcement Learning.” Advances in Neural Information Processing Systems (NeurIPS) 33: 1179–91.

This conservative penalty reshapes value estimates across the continuous action space rather than establishing an absolute physical execution boundary:

The logarithmic integral term \(\ln \int_{\ContinuousActionSpace} \exp(\ExpectedHistoryActionReturn(\DecisionProcessState, \ContinuousActionVector))\,\DerivativeOperator\ContinuousActionVector\) acts as a soft-maximum that actively pushes down on Q-values across the entire action space, aggressively depressing values in unvisited action regions where function approximation variance would otherwise create spurious positive spikes.
The dataset expectation \(\mathbb{E}_{\ContinuousActionVector \sim \DemonstrationDataset}[\ExpectedHistoryActionReturn(\DecisionProcessState, \ContinuousActionVector)]\) pushes up on Q-values exclusively for state-action pairs executed during the logging campaign.
When \(\ConservativePenaltyWeight = 0\), the conservative penalty disappears and policy extraction uses \(\arg\max_{\ContinuousActionVector} \ExpectedHistoryActionReturn(\DecisionProcessState, \ContinuousActionVector)\) without that correction. With \(\ConservativePenaltyWeight > 0\), unsupported actions face a value penalty, but they are not categorically excluded. Physical action limits and checks of operating conditions remain obligations of the independent permission path.

Implicit Q-learning avoids the overestimation at its source by never evaluating an action outside the dataset; it estimates the value of the best logged actions with an expectile regression (Kostrikov et al. 2021). In robotics, offline learning now serves mostly as a starting point. A policy pretrained on the log is fine-tuned online, and calibrating its value estimates during pretraining keeps the first online updates from unlearning it (Nakamoto et al. 2023).

Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. 2021. “Offline Reinforcement Learning with Implicit Q-Learning.” arXiv Preprint arXiv:2110.06169.

Nakamoto, Mitsuhiko, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. 2023. “Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning.” Advances in Neural Information Processing Systems 36, 62244–69. https://doi.org/10.52202/075280-2719.

Applied to the latch, the retained archive holds successes and labeled tripwire failures, both rewardable, so a conservative objective could reweight nominal behavior toward the cleaner openings. It cannot supply displaced-plate competence. Those cells are known absences among the successes and the failures alike, and no offline objective extracts behavior from states the log never visited. When the task archive itself is sparse, teams often try to borrow that missing competence from a foundation model pretrained across other robots.

6.2.8 Pretrained generalist policies and post-training

Whereas latch-bc-01 starts from randomly initialized weights and learns only from the latch campaign, a generalist policy starts from pooled multi-robot data and is then post-trained on demonstrations of the target task. Because a model pretrained on hundreds of thousands of episodes from other robots can move smoothly toward a displaced strike plate, it is easy to mistake broad visual and motor familiarity for verified support on this arm, door, and contact surface. The reference manipulator already carries such a checkpoint as its intent model.

↩︎ The intent model, the slower of the machine’s two learned proposers, is established in The Embodied Brain.

Open X-Embodiment Collaboration, Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, et al. 2024. “Open X-Embodiment: Robotic Learning Datasets and RT-X Models.” IEEE International Conference on Robotics and Automation (ICRA), 6892–903. https://doi.org/10.1109/ICRA57147.2024.10611477.

That checkpoint is OpenVLA-7B, a 7B-parameter vision-language-action model trained on 970K robot episodes from the Open X-Embodiment collection (Kim et al. 2024; Open X-Embodiment Collaboration et al. 2024). Chunk-level generalists are built the same way (table 3). Post-training may update every weight, a low-rank adapter, or only a new action head, and the base may first be retrained on the target machine’s action space.

Pretraining transfers broad visual and motor competence across varied domains. A base network trained on thousands of grasps, door mechanisms, and cluttered tabletop scenes often requires substantially fewer task demonstrations and demonstrates greater resilience to varying illumination, visual distractors, and object geometries than a policy trained from scratch. Pretraining supplies no operational evidence for this plant, however, and the scenario-table join of Choosing a tractable coverage table shows exactly where the gap lies. The scenario table for this door on this machine has 6 cells. The pretraining episodes were collected on other arms, fixtures, and grippers under embodiment contracts the table does not describe, so none of them joins to any latch cell. The post-trained candidate therefore inherits only the campaign’s own support, 1 cell out of 6 with 5 known absences. It may execute the nominal cell more reliably and may even propose plausible maneuvers at the displaced plate, but those proposals are borrowed competence, not evidence. Its declared ODD, the region the policy manifest of section 6.8 computes from the join and the simulator omissions, gains no cell from pretraining.

Three critical dependencies follow, each requiring explicit tracking in the policy manifest. First, the base checkpoint is an external dependency whose underlying pretraining data rarely permits cryptographic digest verification, as its data mixture, filtering heuristics, and training run were permanently fixed by the upstream provider. The manifest therefore records the base checkpoint by the cryptographic digest of its weights and flags its pretraining provenance as a known unknown within the base_checkpoint field of table 4. Second, the base model’s native action space rarely aligns with the target machine. A generalist checkpoint emitting normalized end-effector velocity deltas on an asynchronous clock must be re-headed or mapped into discrete joint setpoints every 25 ms, exposing the pipeline to the normalization fallacy analyzed in Embodiment contracts and the normalization fallacy. Third, fine-tuning risks catastrophic forgetting. Gradient updates fitting the latch demonstrations can degrade broader capabilities that the base previously mastered but the target task never exercises. The same hazard returns, on a smaller scale, when a policy is later fine-tuned on the machine itself (section 6.6); here it is magnified across a large, poorly documented behavioral repertoire. Post-training need not stop at imitation: a pretrained policy can also be improved with reinforcement learning on its own experience, in simulation or on the machine, and section 6.6.2 compares those methods by what each one freezes.

The decoder question also carries over, because post-training often changes the head that runs at deployment. A vision-language backbone can be adapted on discrete action tokens while a separate continuous action expert is trained beside it, with the expert’s gradient stopped before it reaches the backbone, so the newly initialized expert does not disturb the pretrained weights; this knowledge insulation trained faster and generalized better than letting the expert’s gradient flow into the backbone (Driess et al. 2025). The run-time decoder is then the one-pass or few-step action expert, not the token sequence. A study of post-training recipes likewise found that action chunking with a plain \(\ell_1\) regression head matched iterative decoders on its benchmark while decoding far faster (Kim et al. 2025). Either way, the deadline arithmetic of section 6.2.6 governs the post-trained head just as it governs the latch’s own.

Driess, Danny, Jost Tobias Springenberg, Brian Ichter, Lili Yu, Adrian Li-Bell, Karl Pertsch, Allen Z. Ren, et al. 2025. “Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better.” arXiv Preprint arXiv:2505.23705.

Supervised behavioral cloning, action chunking, batch offline RL, and generalist post-training all generate action proposals beyond the state-action support of their logging archives. A low training loss or an inflated Q-value does not constitute physical evidence of safety or success. When an archive lacks the recovery trajectories that a physical perturbation demands, the system must generate new closed-loop experience through direct environment interaction.

6.3 Reinforcement Learning and Environment Interaction

When an archive lacks recovery demonstrations for a displaced strike plate, letting the robot discover a recovery on its own by trial and error means paying for every failed attempt in motor winding heat (\(\WindingJouleHeat = \int \TorqueProducingCurrent(\TimeInstant)^2 \ElectricalResistance\,d\TimeInstant\)), gearbox fatigue reversals, jammed latch cams, and human reset labor. Yet when expert corrections are unavailable or a task demands strategies no demonstrator can supply, closed-loop trial is the only way to generate experience in the states the learner’s own actions reach. By executing actions on the physical system and observing the resulting state transitions, the policy experiences the consequences of its own exploratory deviations. In online policy search,⁷ the agent updates its parameter weights from the scalar advantage estimates accumulated across its own rollouts: \[\nabla_\ModelParameters \ExpectedReturnObjective(\ModelParameters) = \mathbb{E}_{\RolloutStateSequence, \RolloutActionSequence \sim \PolicyTrajectoryDistribution} \left[ \sum_{t=0}^\Time \nabla_\ModelParameters \ln \Policy(\ContinuousActionVector_{t} \mid \DecisionProcessState_{t}) \EstimatedHistoryAdvantage_t \right],\] where \(\EstimatedHistoryAdvantage_t\), a scalar and not the bold action chunk \(\ActionChunk_t\) of section 6.2.5, estimates the advantage \(\PolicyHistoryActionReturn(\DecisionProcessState_{t}, \ContinuousActionVector_{t}) - \PolicyRemainingReturn(\DecisionProcessState_{t})\) introduced in section 6.2.1, quantifying whether an exploratory action yielded higher or lower return than the baseline policy expected. Because transitions are drawn from the state occupancy distribution \(\StateDistribution(\DecisionProcessState)\) induced by the currently deployed policy, closed-loop execution directly exposes the agent to compounding drift and provides immediate evaluative feedback on attempted recoveries. A positive advantage (\(\EstimatedHistoryAdvantage_t > 0\)) reinforces the exploratory action relative to the baseline, whereas a negative advantage (\(\EstimatedHistoryAdvantage_t < 0\)) penalizes it. For this feedback to drive convergence, the reward formulation and sensory observations must make the underlying physical failure modes visible. While learning on policy-generated states resolves the covariate shift between expert demonstrators and the learner, it does not immunize the system against distribution shift caused by novel disturbances, unmodeled environment changes, or drifting hardware dynamics.

⁷ Policy Gradient: Williams (1992) derived the REINFORCE estimator, which recovers the gradient of expected return from sampled actions through the likelihood-ratio identity without requiring an explicit physics model. However, estimator variance scales steeply with the trajectory horizon. Practical implementations subtract a baseline, typically a learned value function that converts sampled returns into advantages, yet still require the extensive interaction budgets evaluated below.

Williams, Ronald J. 1992. “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning.” Machine Learning 8 (3–4): 229–56. https://doi.org/10.1007/BF00992696.

Because each physical rollout dissipates heat in motor windings, cycles actuator gearboxes, and consumes unrecoverable calendar time during resets, an engineering budget for physical training must treat thermal ceilings, mechanical wear, and operational logistics as first-order constraints alongside GPU compute. An algorithm that requires fewer physical interactions is almost always the better choice on hardware, even if each parameter update demands far more computation.

Closed-loop exploration on hardware test benches is billed primarily in calendar time. For an experimental campaign of \(\TrialCount\) trials, where each rollout runs for duration \(\TrialRolloutTime\) and requires a reset duration \(\TrialResetTime\), distributed across \(\ParallelRigCount\) parallel rigs, the total wall-clock duration is \[\TrialCampaignWallTime = \frac{\TrialCount\,(\TrialRolloutTime + \TrialResetTime)}{\ParallelRigCount}. \tag{5}\] Consider the door-opening benchmark, where the arm regulates contact forces over an active horizon of \(\TrialRolloutTime\) = 8 s (the autonomous contact phase of an opening rather than a full teleoperated episode). Training vision-based policies from scratch with reinforcement learning typically demands on the order of \(10^6\) trials; for instance, QT-Opt learned closed-loop grasping from over 580,000 real grasp attempts (Kalashnikov et al. 2018). A baseline scenario of \(\TrialCount = 10^6\) exposes the logistical bottleneck. Between successive trials, an automated fixture, faster than the re-latch fixture used during demonstration collection, must re-latch the mechanism and return the arm to its initial configuration. However, when exploratory trajectories jam the latch cam or trigger joint overcurrent trips, automated reset sequences stall. A human technician must intervene to dislodge the latch, re-align the door, and re-home the joint encoders. In addition, each opening cycles every joint drivetrain through several bidirectional load reversals.

↩︎ The fixture re-latch that paces demonstration collection, with the yield and component wear it costs, is established in Operational yield and mechanical fatigue budgeting.

Napkin Math 6.2: The reset wall

Problem: A from-scratch reinforcement learning campaign of \(\TrialCount = 10^6\) latch openings runs on the machine. How long does it take with automated and with manual resets, what does a farm of rigs buy, and what does the drivetrain absorb regardless of the rig count?

Variables:

Trials \(\TrialCount =\) 1.0 × 10⁶ (illustrative order of magnitude for learning from scratch)
Rollout \(\TrialRolloutTime =\) 8 s (illustrative contact phase of one opening)
Resets \(\AutomatedResetTime =\) 4 s and \(\ManualResetTime =\) 30 s (illustrative)
Rigs \(\ParallelRigCount = 1\), or 14, the largest simultaneous arm count of figure 4 (measured)
Load reversals per opening \(\TrialLoadReversalCount =\) 4 (illustrative)

Math:

Execution only (equation 5 with no reset): 1.0 × 10⁶ \(\times\) 8 s \(=\) 8.0 × 10⁶ s, or 92.6 days of round-the-clock operation.
Automated resets: 1.0 × 10⁶ \(\times\) (8 s \(+\) 4 s) \(=\) 1.2 × 10⁷ s, or 138.9 days.
Manual resets: 1.0 × 10⁶ \(\times\) (8 s \(+\) 30 s) \(=\) 3.8 × 10⁷ s, or 439.8 days; the reset is 78.9 percent of every trial.
Farm: 439.8 days \(/\) 14 \(=\) 31.4 days, about a month.
Wear: 1.0 × 10⁶ \(\times\) 4 \(=\) 4.0 × 10⁶ drivetrain load reversals, for any \(\ParallelRigCount\).

Result: Manual resets extend the campaign to more than a year on a single machine, and even a 14-rig farm requires an entire month of continuous operation.

Systems insight: Reset overhead dominates active execution time, and scaling out hardware rigs partitions the calendar without mitigating cumulative mechanical wear. Only reducing the interaction count \(\TrialCount\) relieves both calendar duration and physical fatigue. Consequently, algorithmic sample efficiency, rather than hardware parallelism, determines whether on-machine reinforcement learning is economically viable.

Adding parallel rigs scales floor space, capital cost, electrical power infrastructure, and supervisory labor in direct proportion to \(\ParallelRigCount\). A grasp-learning farm shows what that scale looks like (figure 4): up to 14 arms, each with its own gripper, bin, and camera, ran day and night, and the cluster collected over 800,000 grasp attempts over two months. The farm was not running reinforcement learning. It used self-supervised grasp outcomes to train an offline success predictor, which an external servo loop queried to guide real-time grasping motions (Levine et al. 2018). That is why it serves here as a picture of scale rather than of an algorithm. Whatever the learner, collecting close to \(10^6\) physical trials takes a room of arms running for months and pays the wear that comes with it, and QT-Opt, which did learn its grasping by reinforcement learning, paid the same price for its 580,000 attempts.

Two stacked photographs of a row of white articulated robot arms on tables, one under daylight and one in a darkened room. Each arm has a two-finger gripper above a metal bin of household objects, with a camera on a post above and behind it. — Figure 4: **Physical data-collection farm**: Between six and fourteen manipulators ran concurrently to collect over 800,000 grasp attempts over two months, using self-supervision to train an offline grasp-success predictor (Levine et al. 2018). While hardware parallelism compresses calendar duration, aggregate system wear and operational costs continue to scale with the total trial count: each station adds capital depreciation, gearbox and bearing wear, motor winding thermal stress, and technician labor to clear dropped objects and mechanical jams. Attribution: Levine et al. (2018).

Mechanical component wear often becomes the binding constraint long before calendar limits are reached. The cumulative 4.0 × 10⁶ load reversals demanded by a from-scratch campaign must be absorbed by actuator drivetrains regardless of how many rigs divide the trials.⁸ Those same million contact events progressively gall the latch cam and strike plate, wear out compliant elastomer gripper pads, and cycle internal wiring harnesses toward fatigue failure.

⁸ Bearing Life vs. Command Reversals: ISO 281 models rolling-bearing fatigue life under continuous rated loads and rotational velocities. Precision transmission manufacturers specify wave-generator bearing fatigue life (\(\RatedFatigueLife\)) in operating hours under nominal operating conditions. However, converting high-frequency commanded torque reversals into consumed gearhead life requires an empirical, unit-specific duty-cycle model.

In purely numerical simulation, an exploratory branch that destabilizes costs nothing more than a discarded buffer and an environment reset. On physical hardware, exploration answers to the first law, irreversibility (principle 1), because a trial that goes wrong delivers its energy before any gradient step can take it back. Unconstrained exploratory torque commands can slam mechanisms into rigid fixtures, strip the precision teeth of harmonic drives, and heat motor stators beyond their 155 °C Class F insulation limit.

↩︎ The Class F winding limit and the insulation breakdown it guards against are established in Thermal capacity and insulation breakdown thresholds.

Physical degradation systematically alters the plant throughout the training campaign. Standard Markov Decision Process formulations assume that the transition probability kernel \(\EnvironmentTransitionDistribution(\DecisionProcessState_{t+1} \mid \DecisionProcessState_{t}, \ContinuousActionVector_{t})\) remains strictly stationary. On physical hardware, this assumption fails. Repetitive impacts burnish contact surfaces, backlash in precision gearboxes expands, and structural fasteners loosen under continuous cyclic vibration. Consequently, the transition dynamics late in a campaign, \(\WornPlantTransitionDistribution(\DecisionProcessState_{t+1} \mid \DecisionProcessState_{t}, \ContinuousActionVector_{t})\), differ measurably from the pristine dynamics measured at trial \(\TrialIndex = 1\). The physical plant mutates under the mechanical work of collecting its own training data, forcing the optimizer to track a drifting target.

Reward formulations can also inadvertently incentivize destructive hardware behaviors. In trial-based policy search, the scalar reward \(\Reward(\DecisionProcessState, \ContinuousActionVector)\) defines the optimization objective, while the action parameterization, lower-level controllers, and independent hardware interlocks define the admissible operating space. If neither the reward nor those controllers account for contact forces, torque slew rates, or thermal buildup, the policy optimizer will exploit those unpenalized degrees of freedom to maximize task return. Engineers mitigate these tendencies by penalizing excessive effort, jerk, and temperature: \[\ShapedTotalReward(\DecisionProcessState, \ContinuousActionVector) = \TaskReward(\DecisionProcessState, \ContinuousActionVector) - \TorqueEffortPenaltyWeight \|\JointTorque_t\|_2^2 - \TorqueSlewPenaltyWeight \left\| \frac{\JointTorque_t - \JointTorque_{t-1}}{\FiniteChange \TimeInstant} \right\|_2^2 - \ThermalPenaltyWeight \max\left(0, \WindingTemperature - \WindingTemperatureLimit\right).\] For example, an unregularized tracking reward often encourages the policy to bang current limits back and forth at high frequency, exciting mechanical resonances and generating severe thermal dissipation. Similarly, an unpenalized contact policy may brace rigidly against a mechanical stop to artificially constrain kinematic degrees of freedom. While the penalty weights above (\(\TorqueEffortPenaltyWeight, \TorqueSlewPenaltyWeight, \ThermalPenaltyWeight\)) trade task progress against effort, torque slew, and thermal rise, soft reward penalties cannot guarantee safety; hard physical stops and active current limits in the motor drive firmware must enforce strict operating envelopes, and practical on-machine training must still cut the million-trial budget by three orders of magnitude before wear and reset labor exhaust the workcell.

6.3.1 Demonstration-seeded reinforcement learning on the machine

Modern real-world reinforcement learning systems circumvent the million-trial reset wall by seeding exploration with prior demonstrations, trading raw trial count for dependencies on initial demonstration support, human intervention gating, and reward-classifier accuracy. For example, human-in-the-loop sample-efficient reinforcement learning (HIL-SERL) achieves high success rates across precision assembly, dynamic manipulation, and dual-arm tasks after only 1 h to 2.5 h of physical training per task (Luo et al. 2025). Four mechanisms compress the required trial count \(\TrialCount\). First, demonstration seeding initializes the replay buffer with 20 to 30 teleoperated trajectories, ensuring that early gradient updates begin within competent task regimes. Second, aggressive off-policy data reuse, implemented via Reinforcement Learning with Prior Data (RLPD) (Ball et al. 2023), replays each recorded transition across multiple gradient updates rather than limiting reuse to a recently collected rollout batch. Third, human-gated interventions truncate failing episodes; an operator takes over when the policy destabilizes, routing the corrective maneuver directly into the replay buffer as in human-gated DAgger (Kelly et al. 2019). Finally, an autonomous binary success classifier trained on labeled visual examples replaces brittle, hand-engineered reward instruments.

Ball, Philip J., Laura Smith, Ilya Kostrikov, and Sergey Levine. 2023. “Efficient Online Reinforcement Learning with Offline Data.” Proceedings of the 40th International Conference on Machine Learning, Proceedings of machine learning research, vol. 202: 1577–94.

Kelly, Michael, Chelsea Sidrane, Katherine Driggs-Campbell, and Mykel J. Kochenderfer. 2019. “HG-DAgger: Interactive Imitation Learning with Human Experts.” IEEE International Conference on Robotics and Automation (ICRA), 8077–83. https://doi.org/10.1109/ICRA.2019.8793698.

At the latch task’s per-trial duration with automated fixture resets, the 2.5 h upper ceiling permits at most 750 physical rollouts. Under these latch timing assumptions, that budget contains a factor of 1,300 fewer trials than the illustrative from-scratch campaign, with 3,000 load reversals rather than 4.0 × 10⁶, shifting the operational burden of equation 5 from months to hours. However, each sample-efficiency mechanism introduces assumptions that must be explicitly tracked in the policy manifest (section 6.8): demonstrations constrain the policy only within the support of the expert’s initial distribution, human interventions generate authority-record transitions whose pre-takeover states serve as negative failure indicators rather than positive imitation targets, and the success classifier acts as a learned perceptual sensor whose systematic false positives—such as mistaking a partially jammed cam for an open door—the policy optimizer will exploit.

↪ The authority record, in which the permission path logs every takeover and barrier trip, and from which intervention episodes are curated, is developed in The Authority Record.

Direct on-hardware reinforcement learning remains practical for low-dimensional mechanisms characterized by trivial reset mechanics, negligible mechanical wear, and broad operating safety margins. When a task’s projected interaction budget outstrips the physical plant’s thermal endurance, component fatigue life, or available calendar time, engineers must offload exploration to simulation. Where simulation is impractical, the remaining alternatives are improving sample efficiency through richer demonstrations, restricting the operational task scope, or incorporating stronger inductive priors. When adopting simulation, raw parallel rollout throughput must always be evaluated in tandem with physical contact fidelity.

Offloading exploration to simulation alters the verification contract between policy training and physical reality. While a virtual environment eliminates mechanical wear, thermal dissipation, and physical reset latency during training, the resulting policy becomes vulnerable to every unmodeled dynamic, relaxed contact constraint, and numerical solver artifact in the simulation pipeline. If the simulator omits sensor transport latency or softens stiff contact constraints to stabilize numerical integration, the policy can exploit those mathematical conveniences to achieve artificially high returns. When transferred to physical hardware, these simulator-specific artifacts manifest as control chattering, limit cycles, or catastrophic instability. Sim-to-real transfer validation must therefore rigorously benchmark the specific physical contact dynamics, friction regimes, and execution latencies required for real-world task success.

6.4 The Simulator as a System Component

A reinforcement learning policy trained on door latching learned an aggressive, unbraked plunge because its simulated strike plate yielded far more than the physical mechanism. Stripping away the simulator during physical deployment does nothing to unlearn that behavior. The physics simulator is not an external sandbox; it is an active component of the training pipeline whose equations, contact approximations, and timing assumptions dictate the exact experience supplied to the optimizer. Accepting simulated task success as evidence of hardware readiness requires measurable, verifiable error bounds on every one of those assumptions.

Like any physical subsystem, a simulator requires a formal interface contract stated in quantities that engineers can measure and budget. This interface contract specifies observation dimensions, action bounds, simulation time step \(\FiniteChange \TimeInstant\), transport latency between command issuance and physical actuation, actuator torque and current limits, sensor noise distributions, reset distributions, and termination conditions. For instance, if a simulator executes an observation loop at 100 Hz with an assumed zero-delay sensor channel and Gaussian noise of variance \(\SensorNoiseStandardDeviation^2 = 1.0 \times 10^{-4}\text{ (rad/s)}^2\), the policy tunes its feedback gains to those exact signal statistics. An unmodeled 15 ms bus transport delay on the physical machine then removes \(\CrossoverFrequency \PureTransportDelay\) of phase at the loop’s crossover frequency \(\CrossoverFrequency\) (equation 4.1), and the loop goes unstable whenever that loss exceeds the phase margin the learned gains left. OpenAI’s Dactyl project faced this contract at full scale, with a 24-DoF Shadow Hand in a motion-capture cage and its MuJoCo twin (figure 5) (Andrychowicz et al. 2020). The team calibrated the simulator against the physical hand and then randomized physical parameters around those calibrated values, so that the policy met the remaining discrepancies as variation it had already seen in training.

Two side-by-side panels. On the left, headed real-world environment, a photograph of an aluminum frame cage with many cameras mounted on it and a robot hand at its center. On the right, headed simulation environment, a rendering on an orange background of a robot hand holding a lettered cube beside a purple cylindrical housing. — Figure 5: **Sim-to-real transfer setup**: OpenAI’s Dactyl apparatus, a 24-DoF Shadow Hand inside a motion-capture cage beside the MuJoCo digital twin it was trained in (Andrychowicz et al. 2020). Transfer is bought with domain randomization (section 6.5), training across randomly varied simulator parameters such as friction, damping, actuator gains, masses, dimensions, and lighting, together with modeled backlash and action delay. A recurrent policy with memory could identify and adapt to the variation it met, so transfer did not depend on making the simulator exact. Attribution: Andrychowicz et al. (2020).

War Story 6.1: Sim-to-real hardware breakage (2018)

Context: OpenAI trained a reinforcement learning policy entirely in a randomized MuJoCo simulation to reorient a block in a 24-DoF Shadow Dexterous Hand, whose joints are driven through tendons by motors in its base (Andrychowicz et al. 2020).

Mechanism: The simulated hand applied torque directly at its joints, while the physical hand is tendon-actuated and has substantial backlash. The authors smoothed every action with an exponential moving average, in simulation and on the robot, to avoid abrupt changes in the action signal that could harm the physical hand.

Impact: The authors reported frequent hardware breakages during physical evaluation, concentrated at the wrist vertical joint under heavy load. Each repair altered tendon tension, joint friction, and sensor calibration, shifting the physical plant across trials and lengthening the evaluation campaign.

Response: Physical ablations revealed the mechanisms enabling successful transfer. Policies trained without domain randomization, or without modeling unmodeled hardware traits like drivetrain backlash and action delay, achieved far fewer consecutive rotations. Furthermore, mechanically locking the fragile wrist pitch joint produced substantially higher physical reliability.

Systems lesson: A simulator that neither wears nor breaks cannot price hardware exposure into the reward. The limits that protected the hand, an action filter and a locked joint, were imposed outside the learned policy, and every repair changed the plant on which the policy was being evaluated.

When a simulator uses an inaccurate rotor inertia, assigning \(\RotorInertia = 2.0 \times 10^{-3}\text{ kg}\cdot\text{m}^2\) instead of the true \(2.5 \times 10^{-3}\text{ kg}\cdot\text{m}^2\), the underlying equations of motion still contain the acceleration-dependent inertial-torque term, with the wrong magnitude. The parameter error alters the size of the response, but the causal relationship holds. In contrast, omitting an entire physical mechanism, such as drivetrain backlash, structural compliance, or bus communication jitter, removes that governing physical constraint entirely. An omitted delay or compliance represents an unmodeled physical degree of freedom. For instance, a simulator that treats robot joints as perfectly rigid permits instantaneous torque reversals. When commanded on a physical machine, those instantaneous reversals shock the drivetrain, load gear teeth beyond their fatigue limits, and excite destructive structural resonances.

Whether a simulator’s modeling compromises are acceptable depends on two fundamental properties: throughput (the rate of experience generated per wall-clock hour) and contact fidelity (how faithfully the contact model reflects physical strike-plate dynamics). Pure throughput motivates teams to adopt simulation in the first place.

6.4.1 Massively parallel simulation and on-policy learning

Simulation substitutes GPU compute time for physical wall-clock time specifically to make on-policy methods such as proximal policy optimization (PPO) (Schulman et al. 2017) practical without months of continuous machine wear. Because PPO collects rollouts with the current policy, takes several gradient steps on that batch, and then discards the data—replacing the batch before policy drift makes its policy-ratio correction less reliable—it requires a simulator that produces fresh rollouts in bulk. To maintain training stability across those gradient steps, PPO constrains policy drift using the clipped surrogate objective of equation 6: \[\PpoClippedSurrogateObjective(\ModelParameters) = \EmpiricalTimestepAverage_t\left[\min\left(\PolicyProbabilityRatio_t(\ModelParameters)\,\EstimatedHistoryAdvantage_t,\ \operatorname{clip}\left(\PolicyProbabilityRatio_t(\ModelParameters), 1 - \RatioClippingWidth, 1 + \RatioClippingWidth\right)\EstimatedHistoryAdvantage_t\right)\right], \qquad \PolicyProbabilityRatio_t(\ModelParameters) = \frac{\Policy(\CognitivePolicyAction_{t} \mid \PhysicalWorldState_{t})}{\pi_{\RolloutPolicyParameters}(\CognitivePolicyAction_{t} \mid \PhysicalWorldState_{t})}, \tag{6}\] where \(\PolicyProbabilityRatio_t(\ModelParameters)\) measures the probability ratio between updated and rollout policies, and \(\RatioClippingWidth\) clips the surrogate incentive for favorable probability-ratio changes; it does not impose a hard trust-region bound on the updated policy.

GPU-resident simulation achieves this scale by maintaining the state of thousands of parallel environments directly in accelerator memory. Physics solvers advance all environments concurrently using batched GPU compute kernels, streaming observation tensors directly into the policy network without crossing the PCIe bus to host CPU memory (Makoviychuk et al. 2021). Policy inference evaluates concurrently across the entire population, turning an environment step into a deterministic sequence of GPU kernel launches rather than thousands of separate CPU process calls.

Rudin and colleagues trained an agile quadruped locomotion policy using this architecture with 4,096 parallel simulated robots, each gathering 24 environment steps per update cycle to form a batch of 98,304 transitions. Completing 1,500 policy updates required less than 20 min on a single workstation GPU (Rudin et al. 2022). This run processed 1.47 × 10⁸ total policy steps, sustaining an aggregate throughput of at least 1.23 × 10⁵ steps per second and accumulating 34.1 days of simulated locomotion at the policy’s 50 Hz control frequency.

At reported throughput, simulation turns 138.9 days of automated trials into 43.4 min.

Applying that throughput to the latch scenario illustrates the appeal. The from-scratch training budget from section 6.3 requires 3.2 × 10⁸ policy steps. At the GPU simulator’s reported rate, this training run completes in 43.4 min, compared to 138.9 days on an automated physical test fixture.

While this thousandfold speedup explains why modern robotic reinforcement learning relies heavily on simulation, raw step throughput can easily mislead. The quadruped benchmark measured locomotion across smooth heightfield terrain. The authors noted that simulation execution time varied by a factor of two over the course of training: as the robots dispersed and ceased colliding with one another, computational cost dropped precipitously because narrow-phase contact detection and linear complementarity solving dominate simulator runtime. Manipulating a latch involves complex multi-body interactions (an articulated door, a sliding cam, and a high-stiffness strike plate) where contact resolution operates at its slowest, and rendering camera observations introduces additional memory and compute bottlenecks. Furthermore, published throughput benchmarks often sacrifice fidelity for speed. For example, the 2024 launch benchmark for one GPU physics simulator sparked controversy because it evaluated throughput by issuing a single action followed by open-loop stepping, disabling robot self-collisions, and using aggressive, low-accuracy solver settings (Tao 2024). High step rates provide no guarantee of physical validity. Training an optimizer against an unvalidated contact model simply drives the policy to exploit the simulator’s numerical artifacts, as demonstrated by the latch simulator’s contact response.

Tao, Stone. 2024. Simulation Speed Benchmark Likely Has Significant Issues Leading to Overstated Numbers. GitHub issue #181, Genesis-Embodied-AI/genesis-world.

6.4.2 The sim-to-real contact exploit

When a physics simulator softens a steel strike plate into a compliant penalty spring to keep its numerical integrator stable at large timesteps, a reinforcement learning policy will exploit that artificial cushion by plunging the gripper into the latch at 0.10 m/s without braking. Contact mechanics govern every critical phase of door manipulation: the physics solver must map surface separation, material penetration, friction, and relative velocity into normal and tangential forces observable by the policy. While Rigid Body Contact Mechanics details rigid-contact complementarity formulations (Stewart and Trinkle 1996) and the time-stepping and compliant approximations that simulators substitute for them, the immediate engineering criterion for training the latch policy is whether these approximations preserve the contact forces and timing boundaries that govern safe approach speeds.

Stewart, David E., and Jeffrey C. Trinkle. 1996. “An Implicit Time-Stepping Scheme for Rigid Body Dynamics with Inelastic Collisions and Coulomb Friction.” International Journal for Numerical Methods in Engineering 39 (15): 2673–91. https://doi.org/10.1002/(SICI)1097-0207(19960815)39:15<2673::AID-NME972>3.0.CO;2-I.

Rigid-body physics engines typically model contact using one of two paradigms: constraint-based complementarity formulations, which enforce non-penetration by solving for constraint impulses, or penalty formulations, which permit geometric interpenetration and generate restoring forces proportional to penetration depth. In practice, real-time solvers approximate both approaches, and numerical tuning parameters alter the resulting dynamics. In the latch simulator, the penalty contact computes an instantaneous normal restoring force from penetration depth \(\PenetrationDepth\) and penetration velocity \(\PenetrationRate\) via \(\NormalContactForce = \PenaltyContactStiffness \PenetrationDepth + \PenaltyContactDamping \PenetrationRate\). Artificially softening this contact interface alters the relationship between approach velocity, contact compliance, and peak impact force. The calculation below models this interaction using an undamped kinetic-to-strain energy balance, demonstrating how penalty spring compliance distorts the resulting force profile.

The manipulator drives the latch tool into the strike plate with an effective moving mass of 9 kg at the tool center point. This value is a stipulation, not a derivation. The registry takes the mass an arm presents at its tool as half its structural mass, the same order as the third of its mass that a uniform link pivoting at its base presents at its tip, and this scenario drops the 3 kg payload capacity because the gripper holds no external object at impact. In a physical manipulator, reflected inertia varies with arm posture, and drivetrain elasticity partially decouples motor rotor inertia during rapid impact transients (Haddadin et al. 2009). Consequently, effective impact mass is an empirical parameter that physical paired probing must calibrate. Equating initial kinetic energy to elastic strain energy at maximum deflection, \(\tfrac{1}{2} \Mass \SpeedAtDetection^2 = \UnbrakedImpactForce^2 / (2\ContactStiffness)\), yields the peak impact force for an unpowered mass \(\Mass\) impacting a linear spring of stiffness \(\ContactStiffness\) at velocity \(\SpeedAtDetection\): \[\UnbrakedImpactForce = \SpeedAtDetection\sqrt{\ContactStiffness\,\Mass}. \tag{7}\] On physical hardware, the compliance does not reside solely in the strike plate. The robot’s rubber finger pads, mechanical linkages, and elastic joints sit directly between the effective link mass and the strike plate, acting as a series spring chain. Because the contact force transmits equally through each element in series, total mechanical deflection is additive: \(\FiniteChange \ContactNormalDeflection = \NormalContactForce/\LatchPlateStiffness + \NormalContactForce/\RobotStructuralStiffness\). The coupled assembly responds with an effective series stiffness: \[\EffectiveSeriesStiffness = \left(\frac{1}{\LatchPlateStiffness} + \frac{1}{\RobotStructuralStiffness}\right)^{-1}. \tag{8}\] In the limit of an infinitely rigid manipulator (\(\RobotStructuralStiffness \to \infty\)), the effective stiffness collapses to \(\EffectiveSeriesStiffness = \LatchPlateStiffness\), establishing an upper bound on impact force. In contrast, the penalty simulator implements a standalone compliant spring, whose artificial stiffness depends entirely on the maximum allowable penetration depth.

Haddadin, Sami, Alin Albu-Schäffer, and Gerd Hirzinger. 2009. “Requirements for Safe Robots: Measurements, Analysis and New Insights.” The International Journal of Robotics Research 28 (11–12): 1507–27. https://doi.org/10.1177/0278364909343970.

Napkin Math 6.3: The soft-spring exploit

Problem: The simulator’s penalty contact lets the gripper sink 2.4 mm into the plate at the fast approach. What peak force does the modeled contact produce, what peak can the plate impose in the rigid-arm limit, and at what robot-side stiffness does the passive model cross the latch force limit?

Variables:

Moving mass \(\Mass =\) 9 kg (stipulated from the arm’s datasheet mass)
Fast approach \(\SpeedAtDetection =\) 0.10 m/s and guarded approach 0.03 m/s (illustrative)
Penalty penetration \(\PenaltyPenetrationDepth =\) 2.4 mm at the fast approach (illustrative)
Plate stiffness \(\LatchPlateStiffness =\) 4.0 × 10⁵ N/m and latch limit \(\LatchForceLimit =\) 100 N (illustrative)

Math:

Simulated spring: The penetration fixes \(\PenaltyContactStiffness = \Mass \SpeedAtDetection^2/\PenaltyPenetrationDepth^2 =\) 1.6 × 10⁴ N/m, so equation 7 gives \(\SimulatedUnbrakedImpactForce = \Mass \SpeedAtDetection^2/\PenaltyPenetrationDepth =\) 9 kg \(\times\) (0.10 m/s)\(^2 /\) 2.4 mm \(=\) 37.5 N, under the limit.
Rigid robot: With \(\ContactStiffness = \LatchPlateStiffness\), \(\UnbrakedImpactForce = \SpeedAtDetection\sqrt{\ContactStiffness\,\Mass} =\) 0.10 m/s \(\times\) (4.0 × 10⁵ N/m \(\times\) 9 kg)\(^{1/2}\) \(=\) 189.7 N, 1.9× the limit.
Break-even chain: The limit is reached at \(\EffectiveSeriesStiffness^* = (\LatchForceLimit/\SpeedAtDetection)^2/\Mass =\) 1.11 × 10⁵ N/m, which equation 8 turns into \(\RobotStructuralStiffness^* = (1/\EffectiveSeriesStiffness^* - 1/\LatchPlateStiffness)^{-1} =\) 1.54 × 10⁵ N/m.
Guarded approach: A rigid robot coasting in at 0.03 m/s peaks at 56.9 N.

Result: In this passive impact model, the fast approach exceeds the force limit if the robot side is stiffer than 1.54 × 10⁵ N/m and remains below it if softer; the guarded approach remains below the limit even with a rigid robot.

Systems insight: A spring 25.6× softer than the plate placed the training result on the safe side of a threshold that the machine’s own compliance decides. No training score reveals whether the physical robot sits above or below \(\RobotStructuralStiffness^*\); empirical hardware measurement provides the only valid ground truth.

The teacher’s reward kept only the task and time terms of equation 1, and the simulated contact placed the plunge on the compliant side of the damage boundary, so the plunge cost the optimizer nothing, shortened every episode, and earned maximum reward. (Had the force term been present, the simulated 37.5 N peak would have crossed the 15 N tripwire and drawn a penalty, but the real plate’s peak would still have gone unseen.) Consequently, the policy converged on an aggressive, unbraked plunge. In the absence of an empirical measurement of \(\RobotStructuralStiffness\) for this specific arm, gripper, and fingertip assembly, the actual physical outcome remains undetermined. Determining true system behavior requires conducting a quasi-static force-deflection test at the tool center point. Until physical characterization confirms whether effective stiffness falls below \(\RobotStructuralStiffness^*\), accepting the policy’s simulated success as proof of safety is unwarranted.

Past about 1.54 × 10⁵ N/m of arm stiffness, the passive model predicts a force peak above the latch limit.

While the energy balance model analyzes an unpowered mass coasting into an obstacle, an actively powered drive pushing forward represents the critical failure mode that safety supervisory logic must interrupt. Contact force follows Hooke’s law, \(\NormalContactForce = \ContactStiffness\,\FiniteChange \ContactNormalDeflection\). Assuming the actuator drives the mechanism into the plate at constant velocity \(\SpeedAtDetection\), penetration depth scales as \(\FiniteChange \ContactNormalDeflection = \SpeedAtDetection \TimeInstant\), driving a linear force ramp \(\NormalContactForce(\TimeInstant) = \ContactStiffness \SpeedAtDetection \TimeInstant\) starting from the instant of physical contact. Setting \(\NormalContactForce(\TimeInstant) = \LatchForceLimit\) defines the contact yield deadline: \[\ContactYieldDeadline = \frac{\LatchForceLimit}{\ContactStiffness \SpeedAtDetection}, \tag{9}\] representing the available time before exceeding the mechanical structural threshold under constant-velocity, linear-elastic conditions. (A complete stopping analysis must additionally account for controller braking trajectories and actuator force slew rates.) Assuming an infinitely rigid manipulator yields a conservative lower bound on this deadline by using the strike plate stiffness \(\LatchPlateStiffness\) directly.

Under these conditions, a fast approach at 0.10 m/s reaches the latch limit of 100 N in 100 N \(/\) (4.0 × 10⁵ N/m \(\times\) 0.10 m/s) \(=\) 2.5 ms after touch. The 15 N tripwire fires at 0.375 ms. Sampling, intervention computation, and drive response then consume 2 ms, placing brake onset at 2.375 ms. Only 0.125 ms remains before the counterfactual constant-drive ramp reaches the force limit. At the guarded approach of 0.03 m/s, the tripwire fires at 1.25 ms and braking begins at 3.25 ms, leaving 5.083 ms before the 8.333 ms touch-relative limit (figure 6). Neither remaining interval establishes that the actual braking trajectory stays below the limit.

The plate-only deadline leaves too little demonstrated braking margin to credit the reactive force monitor at the fast approach, so protection must act before contact until the contact response is qualified. An independently monitored deceleration envelope, \(\SafeApproachSpeedEnvelope(\StoppingBoundaryClearance) \le \sqrt{2 \CredibleBrakingDeceleration \StoppingBoundaryClearance}\), is the stopping inequality of Delay consumes stopping clearance applied to the gripper. Here \(\StoppingBoundaryClearance\) is the clearance to a stopping boundary placed a short standoff before the plate, after sensing and actuation latencies are charged, and \(\CredibleBrakingDeceleration\) is the credible deceleration of the arm at the tool, not the base’s value on the machine card. Because \(\SafeApproachSpeedEnvelope\) falls to zero at that boundary, the envelope forbids an unbraked arrival at the plate. Crossing the standoff is a separate act. A guarded contact mode, qualified on its own, takes over inside it and holds the gripper at or below the guarded approach speed of 0.03 m/s until touch. That mode needs its own evidence about approach speed, force rise rate, loop latency, and deceleration, and a force tripwire works only inside those validated bounds. This contact deadline, not the reward used in training, is what confines the policy’s declared ODD to the guarded approach speed (section 6.8).

Line plot of force against time since touch. Dashed fast and solid guarded constant-drive ramps rise from the origin through a horizontal tripwire toward the latch limit. Separate vertical markers locate fast brake onset at 2.375 milliseconds and guarded brake onset at 3.25 milliseconds. Brackets run from each onset to its ramp's force-limit crossing, with a short 0.125 millisecond fast interval and a longer guarded interval. A red band lies above the force limit. — Figure 6: **Contact force against brake onset**: Counterfactual constant-drive ramps rise as \(\NormalContactForce = \ContactStiffness \SpeedAtDetection \TimeInstant\) from touch. At the fast approach, the tripwire fires at 0.375 ms and braking can begin at 2.375 ms, only 0.125 ms before the ramp reaches the latch limit. At the guarded approach, brake onset is 3.25 ms, leaving 5.083 ms. Each onset includes the same 2 ms response after its own tripwire crossing. Plate stiffness alone gives the shortest deadlines; the lines do not model actual braking trajectories.

Engineering teams must budget contact simulation fidelity in terms of observable physical quantities, such as contact onset timing, peak normal force, total transferred impulse, slip distance, and separation release delay, rather than abstract solver convergence metrics. Discrepancies between simulation and reality frequently manifest in both directions. For example, a penalty solver that resolves an impact within a single integration step concentrates momentum transfer into an impulse spike of extreme amplitude and sub-millisecond duration. On real hardware, viscoelastic fingertip deformation and joint compliance distribute that momentum transfer over dozens of milliseconds. In such cases, total integrated impulse may match closely between simulation and reality even as peak force differs by an order of magnitude. A policy trained to trigger reflexive retraction upon sensing an impulse spike will fail on the physical robot, because the attenuated real-world signal never crosses the detection threshold, causing the motor drive to stall against its maximum current limit.

⇄ Contrast: The 2.4 mm of penalty-spring penetration is compliance the solver invents, whereas the backlash and torsional compliance of the real drivetrain, analyzed in Transmission compliance, contact impedance, and tooth fracture, are compliance a rigid-body solver omits.

Systems Perspective 6.1: The contact fidelity invariant

Never trust a simulated contact policy whose success depends on instantaneous momentum absorption or zero-delay friction engagement; a policy that relies on penetrating the digital surface loads the physical workpiece with force the simulator never showed.

The operational relevance of specific physical omissions depends directly on the task dynamics. A policy executing uninterrupted, high-velocity motions may tolerate crude Coulomb friction models because joint velocities never linger near zero. Conversely, a policy tasked with precise force regulation or static holding will destabilize into violent limit-cycle oscillations if the simulator ignores static friction thresholds, stick-slip transitions, and actuator deadbands.

Validating simulation fidelity requires paired probing. The systems engineer executes identical open-loop actuation trajectories on both the physical robot and its digital twin, quantifying trajectory divergence against explicit error budgets. For the latch assembly, this procedure commands identical velocity profiles into the simulated and physical strike plates, measuring discrepancies directly in milliseconds and newtons. Whenever empirical divergence violates the error margins demanded by the robot’s operating envelope, the simulator cannot be trusted to train that specific capability. That audit relies on knowing which differential equations the simulator solved; when the simulator is itself a neural network trained on video or trajectory logs, even the list of omitted mechanisms disappears.

6.4.3 Learned world models as simulators

A policy trained or evaluated inside a learned world model can score near-perfect task completion by exploiting hallucinated physics where training data were sparse—pushing through a contact transition that the neural predictor smooths over because it rarely saw high-force impacts, with no analytical equation available for an engineer to audit. Whereas physics-based simulators are built from explicit differential equations and mechanical models whose simplifications can be enumerated, a world model derives system dynamics from data by training a neural network to predict future observations (or compact latent states) conditioned on the current state and applied control actions. Contemporary physical AI architectures employ learned world models across three distinct operational roles:

Latent imagination: Policies train via reinforcement learning within the simulated latent representations of a world model. For example, DayDreamer and DreamerV3 trained quadruped locomotion directly from physical robot interaction, mastering rolling, standing, and walking behaviors in several hours without hand-crafted physics simulators (Wu et al. 2023; Hafner et al. 2025).
Zero-shot planning: Action-conditioned video models plan trajectory sequences directly. For instance, V-JEPA 2, post-trained on uncurated robotic manipulation video, guided pick-and-place manipulation across physical robot arms in novel laboratory environments without prior domain calibration (Assran et al. 2025).
Synthetic data generation and evaluation: Scaled generative video models synthesize diverse visual trajectories for offline imitation learning and act as simulated environments to evaluate candidate policies (NVIDIA, Agarwal, et al. 2025).

Wu, Philipp, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. 2023. “DayDreamer: World Models for Physical Robot Learning.” Proceedings of the 6th Conference on Robot Learning, Proceedings of machine learning research, vol. 205: 2226–40.

Unlike a physics engine, a neural world model provides no explicit list of the phenomena it leaves out. Prediction errors concentrate wherever training distribution coverage is sparse, and no analytical derivation exists to reveal which dynamic constraints are violated. Consequently, training data generated purely through world model rollouts must be cataloged as unqualified in the policy manifest, whose simulation_provenance field captures the model architecture, parameter checkpoint hash, and dataset lineage. Simulated success inside a learned world model cannot expand the policy’s declared ODD until empirical paired probing on physical hardware verifies the specific contact forces, transport latencies, and tracking tolerances required by the task. Treating a world model’s internal rollout predictions as verified evidence mistakes model hallucination for physical reality.

Because structural simulation assumptions, numerical integrator time steps, and contact approximations become encoded in the policy’s weights during training, the simulation configuration is an immutable part of the model’s lineage. Every deployable policy artifact must trace its complete provenance through the simulation_provenance metadata block (section 6.8). Deploying policy weights without explicit simulator provenance severs the trained model from the kinematic limits, noise distributions, and solver thresholds under which it was derived. A scalar task completion rate cannot audit that lineage. The reality gap has to be taken apart into its dynamics, sensing, appearance, and timing channels, each measured on its own.

6.5 Physical Channels of the Sim-to-Real Gap

A robotic simulation rarely diverges from physical hardware in only one channel. In the cage-door latch mechanism, for instance, the simulator departs from reality in two fundamentally different ways: a penalty spring far softer than the physical strike plate, and an idealized command path that completely omits transport lag. Collapsing these discrepancies into a single scalar trajectory error blends them into an uninformative aggregate that obscures both mechanisms. Because errors in different physical channels carry different units, propagate along distinct paths within the closed loop, and trigger different failure modes, the sim-to-real mismatch must be treated as a coupled vector of physical components spanning dynamics, sensing, appearance, and timing. Sizing this vector requires an explicit accounting of physical errors against task tolerances, evaluated from paired command-response traces recorded under identical excitations. Each channel demands its own mitigation and runtime interceptor. Table 2 lists these interceptors across actuator dynamics, contact mechanics, sensing, and timing; appearance mismatch is sized separately and charted in panel (d) of figure 7.

Definition 6.2: Sim-to-real reality gap vector

Sim-to-real reality gap vector \(\RealityGapVector = [\DynamicsRealityGap, \SensingRealityGap, \AppearanceRealityGap, \TimingRealityGap]\) is the coupled, multi-channel decomposition of physical discrepancies between simulated and hardware environments spanning contact mechanics (\(\DynamicsRealityGap\)), sensor transduction and noise (\(\SensingRealityGap\)), visual appearance (\(\AppearanceRealityGap\)), and asynchronous transport latency and jitter (\(\TimingRealityGap\)).

Significance: Treating the “reality gap” as a scalar difficulty metric or pure visual domain mismatch causes engineers to overlook dynamical and temporal divergences. Discrepancies in actuator back-EMF, gearbox compliance, or bus jitter deplete feedback stability margins before the policy completes its first physical control step.
Distinction: An inaccurate parameter value (such as a 10 percent error in link mass) preserves the causal differential equations of motion, whereas an omitted mechanism (such as zero-delay communication or unmodeled Coulomb stick-slip) entirely severs physical feedback paths, leading policies to exploit non-physical simulator artifacts.
Common pitfall: Relying on domain randomization alone to bridge a large gap. Randomization covers only the mechanisms the simulator models; for a memoryless policy, indistinguishable parameter instances can favor a compromise or leave low-probability cases poorly served.

Dynamics mismatch measures discrepancies between commanded actuator effort and the resulting physical motion or contact force across the operating envelope. Its contact case is the impulse-spike example of section 6.4.2. A simulator with rigid links, zero backlash, and stiff contact predicts an abrupt force onset; real drivetrain compliance and drive delays soften and delay that rise; and a policy that keyed its braking to the sharp onset keeps driving into the hard stop. The mismatch has the opposite sign of the soft-spring exploit, where the simulator was softer than the real plate rather than stiffer, and the two together are why the dynamics channel has to be measured in both directions.

Sensing mismatch measures discrepancies between the true physical state of the machine and the observation vector delivered to the policy network. Simulated range and depth sensors typically yield clean, uniformly quantized measurements corrupted only by independent zero-mean Gaussian noise. Physical sensors, by contrast, suffer from optical calibration bias, structural noise, temperature-dependent drift, and sustained dropout bursts caused by specular reflections on polished surfaces. An uncalibrated range bias shifts the physical standoff distance where the controller transitions from rapid approach to gentle force regulation. A multi-frame dropout burst leaves the policy momentarily blind during high-speed descent, creating an unobserved window where collisions occur before feedback can intervene.

Image-space distance metrics such as Mean Squared Error (MSE), Peak Signal-to-Noise Ratio (PSNR), or Fréchet Inception Distance (FID) cannot reliably qualify appearance transfer. These summary statistics fail to weight visual discrepancies by policy sensitivity, that is, by the policy Jacobian \(\partial \CognitivePolicyAction / \partial \MultimodalObservation\) relating actions to observations. Consequently, a benign aggregate difference can mask a task-critical localization error. A photorealistic rendering with ray-traced lighting might register a high pixel MSE simply because of background shadows or wall textures that the policy’s visual encoder discards. Conversely, a sub-millimeter pixel shift along the silhouette of an assembly peg might alter the overall image MSE by less than 0.1 percent, yet that subtle shift moves the policy’s predicted insertion coordinate outside the mechanical chamfer tolerance, driving the peg directly into a rigid fixture shoulder. Appearance mismatch must therefore be quantified by evaluating policy-relevant perceptual representations (detected keypoints, surface normal tokens, or latent feature centroids) under controlled variations in physical geometry, illumination, material reflectance, and camera exposure.

Timing mismatch governs the temporal alignment between observation capture, policy evaluation, and actuation. Identical state transitions occurring at different moments in physical time yield entirely different closed-loop dynamics. A simulation that executes observation, inference, and actuation instantaneously at the beginning of each discrete integration step omits camera sensor exposure times, PCIe bus serialization, neural network tail latency, and EtherCAT fieldbus transport, along with their cycle-to-cycle jitter. Every millisecond of omitted transport delay translates directly into physical distance the mechanism traverses open-loop before corrective actuation begins. Unmodeled delay also introduces frequency-dependent phase lag. Near gain crossover, the added lag consumes the available open-loop phase margin. Whether the loop remains stable depends on the sampled plant, compensation, jitter, and total response; Discrete-time sampling and zero-order hold phase lag quantifies the delay contribution.

A single illustrative scenario, kept separate from the latch so that its numbers are not mistaken for latch measurements, shows the four channels diverging at once (figure 7). A linear drive approaches a surface at 0.20 m/s and regulates contact force to 10 N. In the dynamics channel, an unmodeled drive delay of 8 ms and a softer physical contact pad cut the force rise from 500 N/s in simulation to 250 N/s on hardware (a rate the force regulator sets as it ramps toward its target, far below the free-impact rate \(\ContactStiffness \SpeedAtDetection\) of the latch analysis), so the target force arrives at 40 ms instead of 20 ms, the drive overtravels by 3 mm, and a hard current-limit shutdown trips. In the sensing channel, a calibration bias of +3.2 mm triggers premature deceleration, and a specular dropout burst of 4 frames blinds the controller during final descent. In appearance, a pixel difference of only 5.5 percent hides a 6 mm lateral shift in the contact-patch estimate, more than the 2 mm chamfer tolerance. In timing, an illustrative p99 actuation delay of 49.5 ms carries the mechanism 9.9 mm past the point the simulator predicts with its idealized 0 ms observation-to-actuation delay. The simulator’s 20 ms integration step is a numerical parameter, not pipeline latency, and cannot be subtracted from the physical transport delay. Each channel fails in its own units (milliseconds of force rise, millimeters of bias, millimeters of patch shift, millimeters of unobserved travel). A scalar loss can report that hardware departs from simulation but cannot say which channel failed, and separating the channels is what lets table 2 give each one its own paired-trace acceptance test and its own permission-path interceptor.

Four-panel diagram comparing a dashed simulated trace with a solid hardware trace, with a legend for simulated, physical hardware, and limit or tolerance exceeded. Panel (a), dynamics, shows force rising later on hardware and overshooting a dashed force limit in red before a current-limit trip drops it to zero, while the simulated force settles at a target. Panel (b), timing, shows two sinusoids offset by an unmodeled delay T d. Panel (c), sensing, shows the hardware reading offset above the simulated one by a bias delta y, with a shaded dropout gap. Panel (d), appearance, shows the simulated contact-patch position inside a shaded chamfer tolerance band and the hardware position, in red, shifted outside it by delta z. — Figure 7: **Sim-to-real reality gap vector**: Illustrative traces for a separate linear-drive scenario distinguish dynamics (a), timing (b), sensing (c), and appearance (d). Each channel has its own units and acceptance criterion, and red marks only the points where hardware exceeds a limit or tolerance. These constructed traces demonstrate what paired measurements should compare; they are not measured qualification evidence for the latch.

Table 2: Multidimensional sim-to-real reality gap taxonomy: Decomposition of the dynamics, sensing, and timing channels of the reality gap, with dynamics split into actuator and contact rows; appearance is sized in the prose and in panel (d) of figure 7. Columns specify the unmodeled physical phenomenon, representative quantitative discrepancy, the latent control shortcut exploited by the optimizer, illustrative paired-trace acceptance criteria, and candidate mitigations. In the timing row, PCIe is the host’s internal expansion bus that carries camera and accelerator traffic, and EtherCAT is the Ethernet fieldbus that carries setpoints to the servo drives. The actuator row’s \(\StepResponsePhaseLag\) is the phase by which the physical step response lags the simulated one at the test frequency, positive for lag as in equation 4.1. The listed values are scenario assumptions, not measured latch discrepancies or universally qualified thresholds.

Reality Gap Dimension	Unmodeled Physical Mechanism	Nominal Simulation Spec vs. Real Hardware	Latent Policy Exploitation / Behavioral Failure	Paired-Trace Diagnostic Metric & Tolerance	Hardware & Permission-Path Interceptor
Actuator & Drivetrain Dynamics	Strain-wave torsional compliance and hysteresis, Coulomb stiction, velocity-dependent Stribeck friction, and back-EMF voltage saturation	Idealized direct torque source (\(\RotorInertia=0\), \(\SimulatedTransportDelay =\) 0 ms) vs. physical harmonic drive (\(\HarmonicDriveTorsionalStiffness =\) 6.7 × 10² N·m/rad, lost motion \(\BacklashAngle =\) 0.08\(^\circ\), bus delay \(\PureTransportDelay =\) 8 ms)	Policy commands high-frequency full-torque reversals (above 50 Hz), inducing mechanical chatter, gear fatigue, and windings driven past their derating threshold	Step-torque response discrepancy: phase lag \(\StepResponsePhaseLag \le\) 5\(^\circ\) at 10 Hz; settling time discrepancy \(\FiniteChange \StepSettlingTime \le\) 15 ms; overshoot \(\le\) 5%	Low-pass action filtering (\(\ActionFilterCutoffFrequency \le\) 20 Hz), motor current RMS thermal estimators, and actuator slew-rate limiters (\(\DerivativeOperator\CommandedJointTorque/d\TimeInstant \le \JointTorqueLimit/\FiniteChange \TimeInstant\))
Contact Mechanics & Compliance	Viscoelastic surface deformation, contact hysteresis, non-smooth stick-slip transitions, and finite restitution phase	Point-mass rigid contact via a complementarity solver (Rigid Body Contact Mechanics) or penalty spring (\(\PenaltyContactStiffness =\) 1 × 10⁶ N/m, restitution \(\PenaltyRestitutionDuration =\) 2 ms) vs. physical contact pad (\(\ContactStiffness =\) 8.0 × 10⁴ N/m, deformation span \(\ContactDeformationDuration =\) 18 ms)	Policy keys contact onset to the sharp simulated force spike, never sees that spike on the slower real rise, and keeps pushing until the drive reaches its current limit	Contact impulse error \(\FiniteChange \ContactImpulse \le\) 0.05 N·s; peak contact force discrepancy \(\FiniteChange \PeakContactForce \le\) 15 N; contact onset rise-time error \(\FiniteChange \ContactForceRiseTime \le\) 5 ms	Stop-before-contact governor (\(\SafeApproachSpeedEnvelope(\ClearDistance) \le \sqrt{2 \CredibleBrakingDeceleration \ClearDistance}\)); separately qualified guarded contact mode, force-derivative clamp (\(d\NormalContactForce/d\TimeInstant \le\) 1.0 × 10⁴ N/s), compliant mechanical flexures
Sensor Transduction & Noise Profiles	Optical specular dropout, CMOS rolling-shutter distortion, ambient lux-dependent noise, and force-torque gauge thermal drift	Synthetic RGB-D with zero-bias Gaussian noise (\(\SensorNoiseStandardDeviation =\) 0.5 mm, zero dropout) vs. physical structured-light depth (+3.2 mm bias, 4-frame dropout bursts, drift 0.4 N per 10 °C)	Policy triggers premature deceleration above surface or remains blind during multi-frame dropouts, driving end-effector past hard fixture boundaries into stall overcurrent	Distribution distance between simulated and real sensor streams (largest gap between their cumulative distributions) \(\KolmogorovSmirnovStatistic \le\) 0.05; spatial feature activation centroid error \(\le\) 2 mm; zero sustained dropout frames (\(\SustainedDropoutFrameCount = 0\))	Multi-sensor fusion (IMU/encoders + vision), optical dropout holdover filters, thermal drift autocalibration, independent proximity break-beams
Temporal Latency & Bus Asynchrony	Variable camera exposure times, PCIe/EtherCAT bus serialization, neural inference tail latency (p99), and non-deterministic OS scheduling	Synchronous discrete-time lockstep (\(\FiniteChange \TimeInstant =\) 20 ms, zero transport latency) vs. asynchronous physical pipeline (\(\TotalPipelineLatency =\) 35 ms at p50, 49.5 ms at p99, jitter \(\pm\) 6.2 ms)	Severe latency-induced feedback destabilization; policy experiences underdamped oscillations, hunting limit-cycles, and unobserved travel during latency spikes (9.9 mm at p99 relative to zero simulated observation-to-actuation delay)	End-to-end command-response latency jitter \(\FiniteChange \Latency \le\) 2 ms; worst-case observation age \(\ObservationAge \le\) 25 ms; zero deadline overruns at the 25 ms step	Real-time kernel scheduling, action buffer delay queue injection in simulation, state extrapolation that compensates for the known delay

The first line of defense against dynamics mismatch is empirical measurement. System identification⁹ calibrates the simulator’s parameter values (link masses, friction coefficients, torque constants, and transport delays) against paired command-response traces recorded on the physical robot. When a physical subsystem resists accurate first-principles modeling, engineers can replace analytical equations with data-driven approximations. In quadrupedal locomotion work, Hwangbo et al. (2019) trained an actuator network that predicted physical joint torque directly from a rolling history of position errors and joint velocities, executing this neural network inside the simulation loop instead of an idealized torque actuator. Similarly, delta-action residual models learn state-dependent corrections from physical rollouts, adjusting simulated commands so that simulated trajectories track physical behavior (He et al. 2025). In the cage-door latch task, these techniques help reconcile actuator response and bus transport latency, yet they share one limit. A learned correction model is valid only across the empirical state distribution visited during data collection, and outside that distribution the model is uncharacterized. Consequently, any learned simulator correction must enter the policy manifest’s omission log with its own explicitly bounded domain of validity.

⁹ Identification Experiments: System identification treats the input excitation signal as part of the experiment design. Signals such as step functions, swept-frequency sines (chirps), and pseudo-random binary sequences (PRBS) must persistently excite the relevant mechanical and electrical modes, since nominal closed-loop operation rarely activates them (Ljung 1999). Paired physical probing follows this discipline: it drives both the simulator and physical hardware with identical open-loop excitation trajectories rather than relying on passive operational logs, where feedback regulation actively keeps the machine away from the boundary contact dynamics listed in the omission log.

Ljung, Lennart. 1999. System Identification: Theory for the User. 2nd ed. Prentice Hall.

He, Tairan, Jiawei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang, Jiashun Wang, Zhengyi Luo, et al. 2025. “ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills.” Robotics: Science and Systems XXI. https://doi.org/10.15607/RSS.2025.XXI.066.

¹⁰ Domain Randomization: The technique originated in computer vision, where Tobin et al. (2017) randomized textures, lighting, and camera poses so that physical scenes appeared as ordinary variations to a detector trained purely on synthetic renders. Peng et al. (2018) extended the concept to dynamics, randomizing link masses, friction coefficients, joint damping, and transport latencies for robotic manipulation. Appearance and dynamics differ in how a policy can exploit history. Appearance variation changes instantaneous visual observations, whereas dynamics variation changes how the physical plant responds to control inputs, a discrepancy that a policy with temporal memory can infer online.

Tobin, Josh, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. 2017. “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World.” 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 23–30. https://doi.org/10.1109/IROS.2017.8202133.

Peng, Xue Bin, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. 2018. “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization.” 2018 IEEE International Conference on Robotics and Automation (ICRA), 3803–10. https://doi.org/10.1109/ICRA.2018.8460528.

Repeatedly training against a single point estimate of physical parameters, such as an assumed strike-plate stiffness, encourages a policy to overfit to that exact numerical value. Domain randomization¹⁰ addresses parameter uncertainty by perturbing modeled simulation parameters across plausible distributions during training, forcing the policy to encounter varied physical responses (Tobin et al. 2017; Peng et al. 2018). However, sampling from a parameter distribution merely defines training weights; it does not constitute a formal certificate of robustness across the parameter set. How effectively a policy handles this diversity depends heavily on whether its architecture incorporates memory.

In standard reinforcement learning, a policy maximizes expected return within a single nominal simulation environment parameterized by the static vector \(\SimulationParameterVector_0\): \[\ExpectedReturnObjective(\ModelParameters; \SimulationParameterVector_0) = \mathbb{E}_{\RolloutStateSequence, \RolloutActionSequence \sim \PolicyTrajectoryDistribution(\cdot \mid \SimulationParameterVector_0)} \left[ \sum_{k=0}^\Time \DiscountFactor^\RolloutStepIndex \Reward(\PhysicalWorldState_{k}, \CognitivePolicyAction_{k}) \right].\] Domain randomization reformulates the physical environment as a distribution over plausible Markov Decision Processes parameterized by physical vector \(\SimulationParameterVector \sim \ProbabilityFunction(\SimulationParameterVector)\), where \(\SimulationParameterVector\) encompasses link masses, centers of mass, surface friction coefficients, joint damping, transport delays, and sensor noise variances. The domain randomization objective maximizes the expected return marginalized across this distribution: \[\DomainRandomizationObjective(\ModelParameters) = \mathbb{E}_{\SimulationParameterVector \sim \ProbabilityFunction(\SimulationParameterVector)} \left[ \ExpectedReturnObjective(\ModelParameters; \SimulationParameterVector) \right] = \int_{\SimulationParameterSet} \left( \mathbb{E}_{\RolloutStateSequence, \RolloutActionSequence \sim \PolicyTrajectoryDistribution(\cdot \mid \SimulationParameterVector)} \left[ \sum_{k=0}^\Time \DiscountFactor^\RolloutStepIndex \Reward(\PhysicalWorldState_{k}, \CognitivePolicyAction_{k}) \right] \right) \ProbabilityFunction(\SimulationParameterVector)\,\DerivativeOperator\SimulationParameterVector. \tag{10}\] Marginalizing over \(\ProbabilityFunction(\SimulationParameterVector)\) incentivizes policies to maintain performance across sampled physical configurations rather than collapsing to a single operating point. Nevertheless, optimizing expected return can mask catastrophic control failures occurring in low-probability regions of the distribution; high average reward does not imply uniform stability across the support \(\SimulationParameterSet\).

An alternative formulation adopts a minimax objective from robust control, optimizing performance against the worst-case modeled configuration: \[\max_\ModelParameters \min_{\SimulationParameterVector \in \SimulationParameterSet} \ExpectedReturnObjective(\ModelParameters; \SimulationParameterVector).\] Here, the inner minimization identifies the worst-case return within the bounded uncertainty set \(\SimulationParameterSet\), while the outer optimization searches for policy parameters \(\ModelParameters\) that maximize this floor. Minimax optimization targets the worst return within the modeled set, but this remains an optimization statement over modeled rewards rather than a physical stability proof. Neither distribution-averaging nor minimax optimization can protect against physical mechanisms that the simulator completely omits.

Two fundamental constraints govern domain randomization. First, hardware operating outside the sampled support (\(\HardwareParameterVector \notin \text{supp}(\ProbabilityFunction(\SimulationParameterVector))\)) is entirely unrepresented during training. Second, parameter randomization cannot compensate for structural omissions in the physics engine: no amount of variation in link mass or friction will expose a policy to the unmodeled lost motion of a harmonic drive or the preemptive jitter of a real-time operating system. Within the modeled support, the operational effect of broad randomization depends critically on whether the policy maintains temporal state. A memoryless feedforward network receives only the current observation \(\MultimodalObservation_{t}\). When different parameter instances produce indistinguishable observations but require different actions, the policy cannot adapt its choice to the active instance. Optimizing average return may then favor a compromise, and neither conservatism nor stability across the envelope is guaranteed. Conversely, a policy that conditions on a temporal history of observations and actions can infer physical plant dynamics online from the system’s transient response, effectively performing implicit system identification. For this reason, recurrent policies trained across randomized dynamics transfer with substantially higher fidelity than memoryless architectures (Peng et al. 2018), and memory is what lets a wider randomization range produce adaptation to the real plant rather than conservative detuning.

In engineering practice, parameter distributions are configured through two distinct strategies grounded in different forms of evidence. In calibration-first engineering, developers measure nominal masses, torque constants, bus latencies, and friction parameters directly on test benches. Randomization bounds are then derived from known physical mechanisms. For example, motor winding thermal limits determine the envelope of torque-constant variation, and surface degradation models govern friction bounds. These ranges represent verifiable physical claims about the robot and its environment. In contrast, automatic domain randomization (ADR) initializes the simulator at calibrated values and dynamically expands parameter ranges as the policy’s training performance at the distribution boundaries exceeds predefined success thresholds (OpenAI et al. 2019). ADR parameter bounds reflect the limits of what the reinforcement learning optimization can master rather than physical properties of the hardware, often expanding to unphysical extremes. The dexterity achieved by ADR in robotic in-hand manipulation relied on a recurrent policy learning to adapt online to these artificially broad dynamics. Regardless of the scheduling strategy, paired hardware traces must ultimately audit each channel against explicit error budgets before deployment; discrepancies exceeding tolerance require simulator revision rather than blind parameter widening. A randomization range expanded by an automated training curriculum represents an optimization scaffold, not an empirical characterization of the hardware, and therefore cannot expand the declared ODD.

Where domain randomization and simulation fidelity reach their physical limits, deterministic safety mechanisms must enforce system invariants. Every action proposed by a learned policy must pass through an independent permission path before reaching hardware actuators. Within this execution layer, hard force-derivative clamps, torque slew-rate limiters, and velocity-dependent stopping governors restrict commands to the envelope supported by verified sensing and braking capabilities.

↪ The Control Barrier Function filter that minimally corrects unverified policy proposals on every permission tick is developed in The Minimal Intervention Principle and Optimization Filters.

Decomposing the reality gap channel by channel divides the operating space in two. Inside the region where paired traces match simulation within verified tolerances, the simulated experience counts as support. Every unmodeled phenomenon, uncalibrated parameter, and unrepresented disturbance outside it becomes an entry in the omission log, the policy manifest’s record of what the simulator leaves out (section 6.8). The log therefore holds two kinds of entry, mechanisms the simulator omits and parameters no paired probe has yet calibrated, because an uncalibrated parameter leaves the simulated experience just as unsupported until a probe bounds it. For the latch, one of each heads the log: the strike-plate stiffness, a parameter the penalty spring sets far softer than the plate, and the transport latency of the actuator bus, a mechanism the simulator omits. No paired probe has yet closed either, so the high-speed contact states they govern fall outside what the simulation supports. Each entry also states a requirement for a runtime monitor or safety filter, which must detect when execution leaves the regime where the policy’s simulated training remains valid. Calibrating or randomizing simulator parameters still leaves a second gap: the simulator exposes exact contact forces, friction coefficients, and fixture offsets during training that the physical robot’s onboard sensors cannot directly measure at run time.

6.5.1 Teacher-student privileged state distillation and asymmetric actor-critic

While policy training can exploit privileged simulator state—such as ground-truth contact wrenches, friction coefficients, and exact strike-plate offsets—the onboard policy proposer must execute exclusively from the noisy, bandwidth-limited sensors installed on the physical machine. In actor-critic reinforcement learning, the actor proposes control actions while the critic evaluates expected future returns to compute policy gradients. Systems engineers bridge this sensor gap through two main architectures: asymmetric actor-critic, which grants privileged observations exclusively to the critic during training, and teacher-student distillation, which first trains a privileged teacher policy and subsequently trains a sensor-limited student policy to imitate the teacher’s actions. Both architectures require validating whether the runtime observation stream delivered to the student contains enough information to reconstruct or compensate for the missing state.

The core challenge stems from partial observability, which casts physical robot control as a partially observable Markov decision process (POMDP).¹¹ Identical sequences of measured joint positions and velocities can produce drastically different physical trajectories depending on whether a footpad grips dry concrete or slips on wet oil. A conventional critic \(\CriticReturnEstimate(\MultimodalObservation_{t})\) conditioned solely on instantaneous sensory observations cannot distinguish between these physical regimes, producing high-variance or biased value estimates. In simulation, exposing hidden contact wrenches \(\ContactWrench\), friction coefficients \(\CoulombFrictionCoefficient\), terrain profiles, and payload inertias directly to the critic stabilizes temporal-difference learning without burdening the onboard runtime actor.

¹¹ Partially Observable MDP: Åström (1965) formulated optimal control under incomplete state information, proving that an optimal controller must act on the conditional probability distribution of the state given the observation history (the belief state) rather than solely on the instantaneous observation. Reinforcement learning formalized this framework three decades later (Kaelbling et al. 1998). An onboard student network’s temporal observation history serves as a learned approximation of this belief state; its temporal window must span sufficient duration for physical contact and load dynamics to become observable in the sensor stream.

Åström, Karl Johan. 1965. “Optimal Control of Markov Processes with Incomplete State Information.” Journal of Mathematical Analysis and Applications 10 (1): 174–205. https://doi.org/10.1016/0022-247X(65)90154-X.

Kaelbling, Leslie Pack, Michael L. Littman, and Anthony R. Cassandra. 1998. “Planning and Acting in Partially Observable Stochastic Domains.” Artificial Intelligence 101 (1–2): 99–134. https://doi.org/10.1016/S0004-3702(98)00023-X.

Pinto, Lerrel, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. 2018. “Asymmetric Actor Critic for Image-Based Robot Learning.” Robotics: Science and Systems XIV. https://doi.org/10.15607/RSS.2018.XIV.008.

Lee, Joonho, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. 2020. “Learning Quadrupedal Locomotion over Challenging Terrain.” Science Robotics 5 (47): eabc5986.

Kumar, Ashish, Zipeng Fu, Deepak Pathak, and Jitendra Malik. 2021. “RMA: Rapid Motor Adaptation for Legged Robots.” Robotics: Science and Systems XVII. https://doi.org/10.15607/RSS.2021.XVII.011.

Asymmetric actor-critic and teacher-student distillation deploy privileged information through contrasting computational pipelines (Pinto et al. 2018; Lee et al. 2020; Kumar et al. 2021). Under asymmetric actor-critic, the actor receives only the onboard sensor history available at runtime, while the critic receives full privileged simulation state. Because the critic is discarded after training, the deployed actor architecture remains clean and lightweight. However, training an unprivileged actor directly against complex, contact-rich tasks can suffer from high sample complexity.

Distillation splits the work across the proposal boundary (figure 8). Everything privileged stays on the simulation side, and only the frozen student’s proposals, built from the machine’s own sensor history and renewed every 50 ms, cross to the permission path. In Phase 1, the privileged teacher policy \(\pi_{\text{teacher}}(\CognitivePolicyAction_{t} \mid \PrivilegedSimulatorState, \MultimodalObservation_{t})\) learns an effective control strategy via standard reinforcement learning, exploiting full access to contact forces and surface friction. In Phase 2, the student policy \(\pi_{\text{student}}(\CognitivePolicyAction_{t} \mid \ObservationHistoryWindow)\) is trained via supervised imitation learning (such as DAgger or behavioral cloning) to reproduce the teacher’s proposed actions using only a sliding history of onboard observations. While this two-phase approach simplifies reinforcement learning exploration, it introduces a transfer bottleneck. If the teacher’s control strategy relies on distinctions that cannot be reconstructed from the onboard observation history \(\ObservationHistoryWindow\), the student cannot replicate the teacher’s behavior. Privileged information accelerates training convergence, but it cannot make physically unobservable states identifiable on hardware.

Consider the cage-door latch mechanism. A student network equipped with a temporal history can compare commanded motor efforts against observed joint accelerations over recent timesteps, inferring changes in contact stiffness or friction through the resulting mechanical resistance. However, identical joint kinematics can arise from multiple distinct physical causes, and an unobserved geometric offset in the strike plate cannot be inferred simply because the simulation teacher possessed that knowledge. A temporal convolutional network or transformer encoder can output a learned latent embedding of the environment, but its accuracy must be audited against physical ground truth. Adding auxiliary loss functions to predict privileged parameters during distillation regularizes the representation, but an auxiliary loss cannot create physical information that is absent from the underlying sensor stream.

Figure 8: Teacher-student privileged state distillation: A schematic two-phase transfer pipeline. The teacher uses simulator state during training, and the student learns from onboard observations and their history. The training advantage does not establish that every privileged quantity is identifiable from those observations. On the machine, the frozen student renews a proposal every 50 ms and crosses the proposal boundary to the independent permission path; it still requires physical validation.

Selecting the observation history length \(\ObservationHistoryLength\) (distinct from the chunk horizon \(\ActionChunkHorizon\)) involves a trade-off in real-time estimator design. A window that is too short cannot filter high-frequency sensor noise or integrate enough kinematic evidence to distinguish sudden payload changes from transient disturbances. Conversely, an excessively long window increases onboard memory footprint, inflates neural inference latency, and risks diluting crucial transient signals, such as the exact moment of initial mechanical contact, under an accumulation of pre-contact free-space observations. System qualification for the cage-door latch policy must verify contact-onset braking dynamics under the chosen history length while accounting for real hardware sensing delays. Buffering a longer temporal history in memory does not inherently add latency to incoming sensor frames, but the temporal encoder must be structured so that the newest observation immediately influences the output action.

Checkpoint 6.2: Simulator fidelity and transfer

A simulator supplies cheap experience only for the mechanisms it models, and the evidence that it models them comes from the machine.

Sizing the gap

Parameter error vs. omitted mechanism: Can you classify the rotor-inertia error, unmodeled backlash, and a missing transport lag of this chapter, say which of them domain randomization can cover, and explain why the latch’s penalty stiffness, a parameter error, still sits in the omission log until a paired probe calibrates it?
Compliance estimate: Suppose the push test returned exactly the break-even robot-side stiffness of notebook 6.3. How long after touch would a drive that keeps pushing at the fast approach reach the latch limit, and how does that compare with the plate-only contact yield deadline and the contact loop’s response?

Crediting the result

Randomization range: Can you explain why a range widened by a success threshold never enlarges the declared ODD, while a range set from a measured mechanism can be checked against the plant?
Privileged information: Can you say which of the latch teacher’s privileged quantities the student can recover from its observation history, and why it cannot infer an unobserved plate offset merely because the teacher knew it?

Even after system identification, domain randomization, and privileged state distillation, residual discrepancies persist in high-precision, contact-rich operational regimes. For the latch, both omission-log entries remain open. While engineers can resolve certain localized kinematic and friction mismatches through fine-tuning directly on the physical fixture, physical training consumes hardware operational life through mechanical wear and reopens qualification claims that depend on changed proposer behavior, timing, interfaces, or physical conditions. Every updated policy checkpoint produced by on-robot training demands fresh qualification evidence before it can be cleared for autonomous operation.

6.6 Real-Robot Fine-Tuning and Residual Control

Fine-tuning a simulation-trained policy directly on a physical robot to absorb a small fixture offset can fix the local contact error while overheating the motor windings, spending gearbox fatigue life on exploratory impacts, and silently corrupting the free-space approach trajectories that had already passed qualification. Mechanical assemblies rarely match their models exactly: on a warehouse cage door, for example, the physical strike plate may sit at a small mounting offset relative to its nominal position in the simulation model. By computing gradient updates directly from physical rollouts, on-robot fine-tuning adapts policy parameters to the true structural compliance, contact dynamics, and bus latencies of the hardware. However, because end-to-end policy gradients optimize cumulative task return rather than identifying individual system parameters, they do not isolate or quantify specific physical mechanisms. Unmodeled simulation discrepancies, such as an idealized contact spring or an omitted communication transport lag, therefore remain open in the engineering omission log until dedicated system-identification probes characterize them.

Furthermore, physical adaptation exacts a concrete hardware toll. In simulation, sampling aggressive state transitions near the boundaries of stability costs only compute cycles. On physical hardware, the exploratory actions and gradient updates that correct a contact error consume mechanical fatigue life and generate substantial heat in the motor windings. Every exploratory rollout risks driving the mechanism into an unrecoverable contact jam or an over-current trip, and unconstrained parameter updates can silently degrade nominal, previously certified behaviors.

Consider adapting the baseline policy latch-sim-02 into latch-res-03, a residual candidate that adds a gated learned correction to the frozen latch-sim-02 setpoints. This adaptation is motivated by the physical reality of the hardware fixture: the physical strike plate sits 1.5 mm away from its modeled location in simulation. The physical mechanism must guide a spring-loaded latch past this plate, where non-smooth Coulomb friction, cam surface geometry, and spring preloads produce sharp stick-slip transitions. The policy outputs setpoints every 25 ms (40 Hz). To adapt its control commands to the strike plate’s contact dynamics, an on-robot fine-tuning routine collects 20,000 interaction steps, totaling 500 s of active contact and near-contact operation.

During exploration, stochastic perturbations can inject force transients faster than the low-level controller can reject them. If 1.5 percent of the sampled steps breach the 15 N contact tripwire, the fine-tuning run records 300 above-threshold samples. However, engineers must not mistake these raw sample counts for independent mechanical collisions. Consecutive over-threshold steps typically belong to a single sustained contact episode rather than multiple distinct impacts. Accurately counting mechanical shocks and evaluating their damage potential requires temporal event segmentation alongside continuous force-time telemetry.

Every mechanical impact transmits impulsive loads directly into the drivetrain. Translating that cumulative stress into an estimated expenditure of actuator service life requires an empirical, drive-specific load spectrum and fatigue life model. Actuator thermal dynamics impose a second, equally rigid operational boundary. Stochastic exploration dither elevates the root-mean-square current passing through the motor coils. Without explicit cooling intervals scheduled between training episodes, this continuous Joule heating can push motor windings to their thermal derating threshold within minutes.

Demonstrating improved performance on a specific contact fixture does not prove that the overall manipulation task remains sound. An optimization step yielding a parameter update \(\FiniteChange\ModelParameters\) produces an entirely distinct model checkpoint. Empirical evaluations gathered on the prior policy \(\ModelParameters_0\) remain historical observations of \(\ModelParameters_0\) alone; they do not substantiate safety claims for the updated weights. Gradient updates that successfully minimize contact forces on the strike plate can inadvertently distort free-space approach trajectories or degrade emergency stopping profiles in operational regions absent from the fine-tuning rollouts. Because physical trials consume valuable robot runtime and cause mechanical wear, requalification protocols must be strictly prioritized to verify both the targeted contact improvement and the baseline capabilities the update could destabilize.

Each updated checkpoint must carry its own explicit configuration manifest and regression evidence before deployment. Moreover, while optimization progresses on physical hardware, the safety permission path must actively evaluate every candidate action proposed by intermediate policies. Restricting where learned corrections may act keeps unconstrained weight updates from corrupting verified behaviors.

6.6.1 Residual policy architectures and bounded authority

Residual architectures prevent unconstrained weight updates from emitting nonzero drift commands in free space by freezing the base policy weights and multiplying the learned additive correction by an independently enforced binary gate: \[\NominalControlCommand(\DecisionProcessState) = \pi_{\text{base}}(\DecisionProcessState) + \ResidualGate(\DecisionProcessState)\,\FiniteChange\Policy(\DecisionProcessState), \tag{11}\] where \(\pi_{\text{base}}\) is the frozen base controller, \(\FiniteChange\Policy\) is a learned contact correction trained via reinforcement learning on the physical machine against the latch reward of equation 1, and \(\ResidualGate(\DecisionProcessState)\in\{0,1\}\) is an independently evaluated deterministic gate that equals one only within the certified contact envelope.

Both the base policy and the composite controller require explicit verification, because freezing an unverified base policy such as latch-sim-02 does not make it qualified for deployment. Instead, the role of the gate \(\ResidualGate(\DecisionProcessState)\) is structural: it forces the residual correction to evaluate identically to zero in free space and across all non-contact states, where an absence of reward gradients during training would still allow a function approximator to output nonzero values on out-of-distribution inputs. Before any command reaches physical actuators, the permission path verifies the validity of the current state estimate (the machine’s fused estimate of its own configuration, developed in Spatial Memory) and audits the proposed composite action.

Relying on a soft training loss, such as an action-norm penalty \(\|\FiniteChange\Policy(\DecisionProcessState)\|^2\), merely biases the optimizer toward modest corrections during nominal rollouts; it offers no deterministic runtime guarantee. In contrast, the independent gate explicitly revokes residual authority whenever the robot operates outside designated regimes. Even with an architectural gate, the verification suite must evaluate transient switching behaviors at gate boundaries, state-estimation latency, execution timing jitter, and joint torque limits under the combined controller.

Imposing an output magnitude ceiling, \(\|\ResidualGate(\DecisionProcessState)\FiniteChange\Policy(\DecisionProcessState)\|\le\ResidualCorrectionCeiling\), restricts only one dimension of authority. A production permission path must also enforce slew-rate limits on the correction delta, prevent joint limit violations, verify multi-axis dynamic feasibility, and invoke the qualified fallback when active execution loses valid state, evidence, or lease. A late replacement proposal does not revoke a still-valid active prefix; that prefix may continue only within its own validity conditions. Furthermore, the authorized residual envelope must lie strictly within the verified stability margin of the combined closed-loop system across its intended operating envelope; checking an isolated torque limit against an idealized, single-axis specification is never sufficient. For the latch fixture, for instance, the architecture activates residual corrections exclusively during confirmed physical contact, enforces a hard zero via the gate throughout the free-space approach, and clamps joint torque slew rates. Following each optimization step on hardware, an automated regression suite must re-verify free-space trajectory tracking, contact engagement forces, emergency stopping, thermal duty cycles, and drivetrain shock profiles before granting operational authority to the new checkpoint. An improperly verified residual, even when clipped in magnitude, can still induce limit-cycle oscillations or drive the actuator gearboxes into saturation. Freezing the base policy and gating an additive residual is the most localized way to refine a policy with rewards; other post-training regimes modify the decoder’s input noise or update every weight in the network, widening the regression scope accordingly.

6.6.2 Reinforcement learning after pretraining

The gated residual is one member of a broader family of post-training regimes. Every method that refines a pretrained policy using environmental rewards must specify which parameters are allowed to change—because updating shared weights to double throughput on one fixture can silently degrade baseline behaviors elsewhere—and that choice determines how much regression evidence is required to re-qualify the resulting checkpoint. Modern robotic systems use four primary post-training strategies (table 3):

Residual control freezes the base policy weights and learns an additive correction term, exactly as demonstrated by latch-res-03. This strategy has been successfully deployed to refine frozen chunked imitation policies for high-precision assembly, training the residual policy via PPO in simulation (Ankile et al. 2025). Residual architectures are straightforward to bound because independent hardware permission logic can directly gate, saturate, and rate-limit the additive output.
Noise-space steering freezes every parameter of an underlying diffusion or flow-matching policy, training an auxiliary policy only to predict the initial latent noise vector supplied to the reverse diffusion trajectory. Every proposed trajectory still passes through the frozen generative decoder, so steering is restricted to outputs that decoder can generate. This restriction does not establish that those outputs lie within demonstrated support or satisfy physical limits. Furthermore, the optimizer requires only black-box sampling access to the base policy (Wagenmaker et al. 2025). Here, the support of the base distribution constrains the proposed behaviors; the independent permission path must still determine whether those proposals may execute.
Advantage-conditioned post-training updates all network weights using supervised learning conditioned on advantage estimates. A learned critic evaluates the advantage of candidate actions across human demonstrations, autonomous rollouts, and real-time human interventions; the policy is then trained on this mixed corpus conditioned on target advantage and queried for high-advantage actions at run time. Applied to generalist robot manipulation models, this approach more than doubled throughput and halved failure rates on the reported physical manipulation tasks (Physical Intelligence et al. 2025). However, because gradient updates modify every weight in the network, all nominal behaviors remain subject to drift.
Policy-gradient fine-tuning in simulation applies reinforcement learning techniques (section 6.4.1) to update the weights of a pretrained vision-language-action policy using sparse or binary success rewards in massively parallel simulation (Li et al. 2025). Like advantage conditioning, this paradigm modifies all network parameters; in addition, it inherits every physical approximation and contact-solver idealization present in the underlying simulation environment.

Regression scope follows where the composite output can change and which execution assumptions the update affects. An independently enforced residual gate localizes the changed output, but regression still covers its active region, boundary transitions, combined closed-loop behavior, and timing and resource effects. Noise-space steering can change sampling throughout the base domain despite frozen decoder weights, so every affected operating mode requires regression unless a separate gate establishes a narrower scope. Full-weight updates can also alter behaviors across the entire operational envelope. A newly fine-tuned full-weight model demands complete end-to-end regression testing across all previously validated operational modes, and the system’s adaptation_wear_log must document every modified parameter group. Whether adaptation rollouts accumulate physical machine wear on hardware or incur modeling dependencies in simulation, each rollout must be recorded in the policy manifest. No model can legitimately claim expanded operational authority without concrete validation data recorded in the system audit log.

Table 3: State of practice in robot policy training, as of 2026: Named systems that instantiate the mechanisms this chapter teaches, dated by their cited publications. The systems will be superseded; the mechanism and manifest columns are the durable part of the table.

Regime	Named Systems	Mechanism Taught in This Chapter	What the Policy Manifest Must Record
Pretrain, then post-train	Octo (Octo Model Team et al. 2024); OpenVLA (Kim et al. 2024); \(\pi_0\) (Black et al. 2024); \(\pi_{0.5}\) (Black, Brown, et al. 2025); GR00T N1 (Bjorck et al. 2025)	Pretraining on pooled multi-robot data, then supervised post-training on the task	Base weight digest; pretraining mixture as a known unknown; action-space mapping
Post-training recipe	OpenVLA-OFT (Kim et al. 2025)	Action chunking with parallel decoding and an \(\ell_1\) regression head	Decoder latency of the new head against the proposal deadline
RL post-training on the machine	\(\pi^*_{0.6}\) with RECAP (Physical Intelligence et al. 2025)	Advantage-conditioned updates from autonomous experience and corrections	Full-weight change; corrections as authority-record entries; regression over base domain
RL post-training in simulation	SimpleVLA-RL (Li et al. 2025)	Policy gradient with a binary success reward in a parallel simulator	Simulator provenance and omission log
Frozen-policy steering	DSRL (Wagenmaker et al. 2025)	Reinforcement learning over the decoder’s input noise, base weights unchanged	Base digest unchanged; steering policy digest
Residual RL	ResiP (Ankile et al. 2025)	Residual trained by PPO on a frozen chunked imitation policy	Gate, magnitude, and slew limits; regression where the residual acts
Real-world RL from demonstrations	HIL-SERL (Luo et al. 2025)	Demonstrations, interventions, and a learned success reward with off-policy reuse	Success classifier as a learned sensor; interventions as authority-record entries
GPU-parallel simulation	Isaac Lab (NVIDIA, Mittal, et al. 2025); MuJoCo Playground (Zakka et al. 2025)	Batched, accelerator-resident physics feeding on-policy learning	Solver build and contact settings; throughput stated with its scene
World models	DreamerV3 (Hafner et al. 2025); V-JEPA 2 (Assran et al. 2025); Cosmos (NVIDIA, Agarwal, et al. 2025)	Learned dynamics for imagined rollouts, planning, synthetic data, and evaluation	Model weight digest and training data; generated experience unqualified

Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees Nair, et al. 2024. “Octo: An Open-Source Generalist Robot Policy.” arXiv Preprint arXiv:2404.08839.

Black, Kevin, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, et al. 2025. “\(\pi_{0.5}\): A Vision-Language-Action Model with Open-World Generalization.” Proceedings of the Conference on Robot Learning (CoRL), Proceedings of machine learning research, vol. 305: 17–40.

Bjorck, Johan, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, et al. 2025. “GR00T N1: An Open Foundation Model for Generalist Humanoid Robots.” arXiv Preprint arXiv:2503.14734.

Kim, Moo Jin, Chelsea Finn, and Percy Liang. 2025. “Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success.” Robotics: Science and Systems XXI. https://doi.org/10.15607/RSS.2025.XXI.017.

Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, et al. 2025. “\(\pi^{*}_{0.6}\): A VLA That Learns from Experience.” arXiv Preprint arXiv:2511.14759.

Li, Haozhan, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Zhaohui Yang, Kaiyan Zhang, Xuekai Zhu, et al. 2025. “SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning.” arXiv Preprint arXiv:2509.09674.

Wagenmaker, Andrew, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. 2025. “Steering Your Diffusion Policy with Latent Space Reinforcement Learning.” arXiv Preprint arXiv:2506.15799.

Ankile, Lars, Anthony Simeonov, Idan Shenfeld, Marcel Torne, and Pulkit Agrawal. 2025. “From Imitation to Refinement: Residual RL for Precise Assembly.” 2025 IEEE International Conference on Robotics and Automation (ICRA). https://doi.org/10.1109/ICRA55743.2025.11127442.

Luo, Jianlan, Charles Xu, Jeffrey Wu, and Sergey Levine. 2025. “Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning.” Science Robotics 10 (105): eads5033. https://doi.org/10.1126/scirobotics.ads5033.

NVIDIA, Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, et al. 2025. “Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning.” arXiv Preprint arXiv:2511.04831.

Zakka, Kevin, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, et al. 2025. “MuJoCo Playground.” arXiv Preprint arXiv:2502.08844.

Hafner, Danijar, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. 2025. “Mastering Diverse Control Tasks Through World Models.” Nature 640 (8059): 647–53. https://doi.org/10.1038/s41586-025-08744-2.

Assran, Mido, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, et al. 2025. “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning.” arXiv Preprint arXiv:2506.09985.

NVIDIA, Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, et al. 2025. “Cosmos World Foundation Model Platform for Physical AI.” arXiv Preprint arXiv:2501.03575.

Bounded residual authority restricts the influence of an exploratory policy during execution, but it cannot alone prevent mechanical damage or eliminate latent vulnerabilities introduced during earlier training phases. Every stage of policy synthesis, from demonstration ingestion and simulation contact mechanics to residual adaptation, leaves a distinct behavioral signature on closed-loop execution. When an embodied machine develops high-frequency chattering, thermal derating, or semantic drift, systems engineers can trace that failure behavior directly back to the training assumptions, solver compromises, and architectural trade-offs that created it.

6.7 Training Failure Modes and Permission-Path Interceptors

A physical failure in a learned policy is rarely arbitrary. When an end-effector collides with a strike plate, that command reflects optimization over a specific dataset, loss formulation, and simulator dynamics, and the two failures that opened the chapter can now be traced back to theirs. The assumptions baked into that optimization leave a structural footprint on policy behavior, one that engineers can trace and mitigate before hardware ever runs. The policy vulnerability inference chain makes this causal trace explicit across five links: (\(\text{Exposure} \to \text{Dependency} \to \text{Violated Assumption} \to \text{Observable Signature} \to \text{Permission-Path Interceptor}\)). Training exposure establishes a learned dependency; an unmodeled deployment condition violates that dependency; the violation produces a measurable signature in the commands or physical plant telemetry; and that signature names the specific runtime interceptor that the execution permission path requires. Reading the three latch checkpoints back through this chain shows how distinct optimization regimes produce different failure modes.

For latch-bc-01, behavioral cloning relies on the teleoperated openings. These demonstrations only support nominal plate alignment during the guarded approach; the data-collection campaign never demonstrated a displaced or misaligned plate. A sagging door frame directly violates that geometric dependency. The resulting failure signature is compounding covariate shift. Because behavioral cloning optimizes an open-loop supervised objective over expert trajectories, small drift errors compound at each control step. Individual motion proposals often remain physically plausible right up until the gripper mechanically binds against the frame. To prevent mechanical damage, a support monitor must evaluate runtime observations against the empirically supported demonstration cells and revoke policy authority outside them. Predeployment qualification follows directly from the known coverage gap that the scenario table of Choosing a tractable coverage table recorded: bench trials with deliberately displaced plates must confirm that the support monitor trips reliably before the mechanism binds.

For latch-sim-02, training inside a physics simulator instills strong dependencies on idealized contact dynamics and negligible actuation lag. A physical steel plate violates both assumptions, generating sharp contact forces and unmodeled transport delays that produce the overforce transients quantified in section 6.4.2. Paired contact probes expose this dynamical mismatch before hardware deployment. If the physical contact deadline leaves insufficient time for closed-loop deceleration, a reactive force tripwire arrives too late. Detecting a force excursion confirms that an impact occurred, but actuator inertia and mechanical compliance will continue driving the mechanism into the plate before the brakes can arrest motion. The interceptor is therefore the pre-contact stopping envelope of section 6.4.2, which guarantees a full stop at the standoff before the plate and hands engagement beyond it to the separately qualified guarded contact mode.

For latch-res-03, adaptation concentrates real-robot experience on the contact states of a single hardware rig. Localized fine-tuning targets tight-tolerance contact dynamics on that specific fixture, but it does not qualify the frozen base across the rest of the operational state space. Unlike full-weight fine-tuning, which can introduce catastrophic forgetting, this gated residual retains the base weights and holds corrections at zero in free space; its primary regression risks arise from contact corrections and gate-boundary transitions. Before updated weights receive operational authority, they must clear a frozen regression suite that evaluates both nominal latch engagement and free-space flight. In addition, an architectural residual gate must hold corrections at zero outside the designated contact phase, preventing fine-tuned offsets from destabilizing the free-space approach. Physical exploration during real-machine tuning (section 6.6) also imposes mechanical shock and motor winding heat; the runtime supervisor must monitor these thermal and structural thresholds continuously to ensure that task reward improvements do not mask hardware degradation.

Subsequent training stages do not automatically heal the unresolved dependencies of earlier stages. In the simulation-and-residual lineage (latch-sim-02 through latch-res-03), simulation artifacts persist until physical hardware evidence explicitly closes them. Conversely, the recovery-imitation branch adopts an alternative architectural path. The checkpoint latch-bc-recovery-02 incorporates targeted, human-guided recovery demonstrations for a named displaced-plate condition, expanding the operational envelope without relying on an uncalibrated simulator. Its remaining engineering uncertainties, chiefly closed-loop trajectory stability and potential regression on nominal openings, are made explicit through targeted qualification requests.

Ultimately, each inference chain connects a specific training assumption to an empirical test and an active execution limit. The policy manifest preserves these connections alongside the checkpoint’s source data, simulator configuration, optimization objective, and architectural constraints. By maintaining this end-to-end traceability, verification suites can systematically evaluate candidate policies against the exact physical vulnerabilities created by their training history.

6.8 The Policy Manifest

Handing a candidate checkpoint to an evaluation team with only its weight file and a training curve hides whether its behavior on a displaced plate comes from human recovery demonstrations, uncalibrated simulator rollouts, or pure network extrapolation. The candidate selection must distinguish nominal behavior supported by demonstrations, newly taught recovery, and contact behavior that still depends on unverified simulation. The failure chains of section 6.7 make that distinction concrete for each checkpoint; the record accompanying the candidate into evaluation preserves the reasoning, including the assumptions that remain open.

The policy manifest records how a checkpoint was produced: its training stages, the data support it inherits from the provenance records, its simulator versions and the omission log of mechanisms they leave out and mismatches paired probes have not closed, the wear incurred during adaptation, and the operating domain it declares (table 4). These fields state what evaluation must test and what runtime enforcement must accommodate. The runtime safety supervisor can check the manifest against its validated configuration before enabling a checkpoint. A declared boundary does not automatically generate a reliable detector or a safe clamp; those mechanisms require their own engineering and evidence.

The manifest separates optimization results from the conditions the policy may be asked to operate in. The declared ODD is the part of the ODD from Collection Policy Coverage that the policy manifest claims: the supported cells of the data join, minus every state the omission log excludes. High training reward or low offline loss does not show that the policy is safe outside it, so the declared ODD is computed from the join and the log, never from the training curve, and a convergent metric cannot stand in for clearance over regimes nobody characterized.

The separate latch-bc-recovery-02 manifest consumes the original provenance and the stipulated latch-recovery-02 archive, adding the positively displaced-plate cell at the guarded approach to the candidate domain with the expert recovery labels and held-out sessions identified by digest. Its qualification request asks whether the updated policy completes that recovery and preserves nominal service under the same physical limits, while the original archive remains unchanged and negative displacement and the fast approach remain excluded. Current provenance bounds the evidence already available, whereas collecting and qualifying new experience expands the machine’s useful competence.

Table 4: Policy manifest schema: Fields the policy manifest adds to the provenance records it consumes, recording policy synthesis stages, any pretrained base checkpoint, inference execution parameters, data support by digest, physics engine specifications and their omission log, hardware fine-tuning wear, and the declared ODD.

Record Schema Field	Recorded Quantities and Artifacts	Provenance & Immutability Guarantee	Downstream Verifier & Usage
`regime_sequence`	Array of optimization stages (\(\text{BC} \to \text{SimRL} \to \text{Distill} \to \text{Residual}\)), loss formulas, teacher privileged state dimensions (\(\ContactWrench, \SimulationParameterVector\)), student observation history length \(\ObservationHistoryLength\), reward weights, and stopping criteria hashes	Git commit SHA, PyTorch/JAX checkpoint UUIDs, parameter hash \(\text{hash}(\ModelParameters_k)\)	Tracing which regime introduced latent failure modes or high-gain chatter
`base_checkpoint`	Weight digest of any pretrained checkpoint the policy started from, its source, the action-space mapping applied, and the layers post-training changed; the pretraining mixture recorded as a known unknown when it cannot be cited by digest	Base weight digest; never counted as data support	Scopes forgetting regression tests; confirms that no supported cell is attributed to pretraining
`policy_inference_contract`	Model architecture class, parameter count, checkpoint weights hash \(\text{hash}(\ModelParameters)\), chunk prediction horizon \(\ActionChunkHorizon\), execution horizon \(\ExecutionHorizon\), decoder evaluations \(\DecoderEvaluationCount\), inference deadline \(\ChunkRenewalPeriod\), and chunk lease \(\ChunkLease\)	Cryptographic checkpoint hash \(\text{hash}(\ModelParameters)\), compiled engine digest	Parameterizes the runtime proposal buffer (Action chunking) and real-time lease monitor (Chunk Lease Sizing from the Stopping Budget)
`data_support`	Digests of the consumed provenance records and of the scenario table they were measured against; the `supported_cells` and `known_absences` of their occupancy join	Cites each provenance record by digest; no dataset field is copied	Upper bound on the declared ODD; parameterizes out-of-distribution support monitors
`negative_event_registry`	Authority-record entries (Intervention and Recovery Data) from training and adaptation rollouts	Cites each authority-record entry by digest; its pre-trigger trace is not copied	Isolating physical failure boundaries and preventing corrupted drift rollouts from training
`simulation_provenance`	Physics engine build, integrator method, timestep \(\FiniteChange \TimeInstant\), contact solver penalty parameters, and domain randomization ranges \(\ParameterRandomizationRanges(\SimulationParameterSet)\) with their source (measurement or curriculum); for a learned world model, its weight digest and training data	Physics engine binary hash, robot model asset tree hash (URDF/MJCF)	Reproducing the training physics and auditing paired-trace budgets
`omission_log`	Mechanisms each simulator leaves out (backlash, compliance, transport lag) and the residual mismatches paired probes did not close, with the states each one affects	Bound to the simulator build it describes; an entry closes only on paired-trace evidence	Removes the states it affects from the declared ODD; carried into evaluation’s coverage gaps and the fault list
`adaptation_wear_log`	Serialized hardware IDs, contact shock counts, motor current RMS thermal logs, modified layer indices, and regression suite hash	Calibration certificate hashes, hardware lifetime fatigue logs	Re-opening formal qualification cases and preventing silent regression of nominal capabilities
`declared_odd_boundary`	Declared ODD: the supported cells minus the states the omission log excludes (for the latch policy, the nominal strike plate at the guarded approach speed)	Never exceeds the supported cells of `data_support`	Bounds the target ODD that evaluation may sample and the region in which the permission path admits policy proposals

For the original latch campaign and its simulation-residual lineage, a compact manifest links the fields into one decision:

regime_sequence: demonstration checkpoint latch-bc-01 → latch-sim-02, a student initialized from latch-bc-01 and distilled from a simulator teacher trained by reinforcement learning on privileged state (privileged contact wrenches \(\ContactWrench\) distilled into an 8-step observation-history buffer) → gated residual candidate latch-res-03; retain each checkpoint hash, loss or reward specification, and stop rule.
base_checkpoint: no pretrained robot policy underlies the latch checkpoints, which descend from latch-bc-01 as regime_sequence records; any pretrained image encoder they use belongs here by digest. A candidate post-trained from the machine’s intent model would record that model’s weight digest, its pretraining mixture as a known unknown, and its action-space mapping, and would inherit the same declared ODD (section 6.2.8).
policy_inference_contract: specify checkpoint weights hash \(\text{hash}(\ModelParameters)\), the machine’s 80M-parameter ACT architecture with one decoder pass per chunk, chunk prediction horizon \(\ActionChunkHorizon =\) 16 setpoints at 25 ms, and the 40 ms p99 inference target; record the selected execution prefix separately from the 50 ms proposal-renewal deadline and the selected chunk lease. Each renewed prefix requires valid evidence and a feasible continuation or fallback; the prediction horizon is not the active lease.
data_support: cite the latch campaign’s provenance records and its scenario table by digest; their join supports the nominal strike plate at the guarded approach, and lists the displaced plate and the faster approach, whose cells hold no demonstrations, as known absences with no recovery label.
simulation_provenance: record the actual solver build and contact parameters.
omission_log: the example simulator models strike-plate contact with a penalty spring far softer than the plate and omits actuator transport lag, so the contact states those two departures govern are unverified for the checkpoints trained in simulation.
negative_event_registry: cite by digest the authority-record entry, with its pre-trigger ring-buffer trace, for every fixture-shoulder strike, operator veto, or permission-path clamp with its event classification, without treating rejected learner commands as expert targets, and mark episodes an operator truncated so that an update does not score an aborted command as a success.
adaptation_wear_log: associate latch-res-03 with the specific rig, above-threshold samples and separately identified contact events, motor-current history, modified residual weights, and free-space regression result.
declared_odd_boundary: the supported cells minus the states the omission log excludes, here the nominal plate at the guarded approach of 0.03 m/s. The contact states governed by the two omissions stay outside the declared ODD for every simulation-refined checkpoint, latch-sim-02 and latch-res-03 alike, until paired probes close them. The displaced plate and the 0.10 m/s approach lie outside it, and the fast approach would stay outside even with demonstrations, because its contact deadline (section 6.4.2) leaves almost nothing beyond the contact-loop response; there the permission path must impose the guarded approach or refuse policy authority.

A qualification review would withhold operational authority from latch-res-03 until paired contact probes quantify the contact-stiffness error and the omitted lag, and the combined controller passes contact, free-space, stopping, and wear-budget tests. With only the original campaign, the same rule leaves latch-bc-01 as the checkpoint evaluation may sample, provided the support monitor of its chain refuses policy authority off the nominal plate (section 6.7). Trained without a simulator, it carries no omission-log exclusions, so its declared ODD keeps the door’s contact states at the nominal plate and the guarded approach. The manifest records that decision and its evidence rather than converting a training score into permission.

The manifest references the collector, plant, calibration, and action taps through the provenance record of The Dataset Schema. The cited digests preserve which data and scenario table produced the checkpoint; later changes require a new record rather than silently updating its lineage. Simulator provenance similarly fixes the solver and timestep¹² used during optimization. Its separate omission log carries unresolved discrepancies from section 6.5 into evaluation’s coverage gaps and fault-injection tests.

¹² Penalty-Contact Timestep: The contact timescale scales as \(\sqrt{\EffectiveImpactMass/\EffectiveSeriesStiffness}\). For the latch’s 9 kg moving mass it is about 24 ms against the soft penalty spring and 4.7 ms against the real plate, so a timestep sized to resolve the simulated contact is about 5× too coarse once the spring is stiffened to match the plate. The integrator, damping, and solver tolerances set the final step, which is refined until contact onset, peak force, and impulse converge.

Downstream engineering teams consume the manifest throughout the design, evaluation, and release lifecycle. The evaluation record of The Evaluation Record consumes it directly. Its target ODD is sampled from inside the declared ODD, and its catalog of coverage gaps inherits the known absences and the omission log, so evaluation suites probe the specific failure modes each training regime predicts. Targeted exploratory tests outside the candidate domain can also identify gaps, but they do not establish an operational claim there without new support and qualification. The permission path takes the declared ODD as the outer limit of the region in which it admits policy proposals, and uses the recorded support to parameterize out-of-distribution detectors. Verification and the safety case draw on the manifest to establish the provenance of every parameter update and to confirm that each omission is bounded by an independent physical interlock.

↪ The omission log becomes one of the upstream records from which fault records are compiled for release, developed in Fault Records for Release.

Checkpoint 6.3: Policy manifests and declared ODDs

Before releasing a trained policy checkpoint to physical verification, confirm the following relationships between the policy manifest, its omission log, and the declared ODD:

Complete the latch record: A fixture-shoulder strike appears after latch-res-03 was trained with a soft penalty contact and no actuator lag. Which omitted mechanism belongs in omission_log, and which paired probe and admission decision must the manifest record?
Confining the declared ODD: Suppose demonstrations were collected at the fast approach and every omission-log entry were closed by paired probes. Why would the latch policy’s declared ODD still exclude the fast approach, which timing comparison decides it, and what must the permission path impose on a proposal that arrives at that speed?
Negative event logging: Why does the manifest keep the pre-trigger trace of an operator veto or fixture strike as labeled failure evidence rather than as an imitation target?
Declared operational boundaries: latch-bc-01 proposes to open the door while the strike plate sits displaced from its nominal mount. How does the permission path use the data_support field to classify that state as ungrounded extrapolation rather than demonstrated competence, and why does a confident policy output not change the classification?

The manifest turns a training history into a claim with stated limits. The original record sends latch-bc-01 to evaluation and keeps latch-res-03 out of service until paired probes and regression tests close its gaps. The stipulated recovery record sends latch-bc-recovery-02 with a wider, explicitly testable candidate claim. The common ways that decision goes wrong all let a training score stand in for that evidence.

6.9 Fallacies and Pitfalls

Minimizing loss on an offline demonstration split or maximizing cumulative return in simulation measures compliance with training assumptions rather than physical competence on hardware. When an objective rewards behavior that exploits a curated demonstration distribution or a simulator approximation, the ranking of candidates established in training can invert on deployment: the candidate that scored highest in simulation finishes lowest on hardware, through unsupported state drift, contact-force spikes past the structural limit, or torque oscillations that grow into resonant chatter (figure 9). Virtual environments forgive maneuvers that physical hardware penalizes with tripwires, resonance, or thermal overload, so the causal assumptions connecting a training metric to physical execution have to be tested before a policy receives actuator authority.

Two panels. Panel (a) is a slope graph between a simulated-score axis and a hardware-result axis: a red line runs from the exploiting policy high on the simulated score down to low on hardware, and a dark line runs from the robust policy low on the simulated score up to high on hardware, so the lines cross. Panel (b) stacks three small traces: a state trace that drifts out of a shaded band of demonstrated states, a contact force whose red spike rises above a dashed force limit, and an oscillation that grows into large chatter where the actuation frequency matches the resonance frequency. — Figure 9: **Illustrative policy ranking inversion**: (a) A constructed slope graph shows how a candidate favored by a simulated score can lose its advantage under physical evaluation. (b) Possible causes include unsupported recovery states, contact mismatch, and actuator chatter. The rankings are a teaching example, not measured results for the latch checkpoints.

Pitfall: Sanitizing recovery episodes to achieve lower offline validation loss.

Suppose an archive of latch manipulation demonstrations includes human recovery maneuvers, and two separate policies train on competing subsets of this corpus. One policy trains on the full archive, including episodes where a human teleoperator backed off the gripper and re-aligned the jaws after the latch cam caught on the strike plate. The second policy trains on a sanitized subset stripped of all hesitation and retries, achieving a markedly lower loss on its held-out validation split. When deployed on the physical cell, both policies eventually encounter a misaligned cam. The first policy recognizes the contact anomaly, backs off, and seats the latch successfully. The second policy relentlessly drives along the nominal path, building contact force until the safety supervisor’s force tripwire halts the cell. Because the sanitized dataset excluded off-nominal recoveries, the lower validation loss proves only that predicting smooth nominal trajectories is easier to fit than handling physical disturbances. Pruning corrective demonstrations simplifies the loss score at the exact price of removing the feedback behaviors the physical controller needs off the nominal path. Filtering out recovery trajectories to drive down offline prediction loss strips the policy of the corrective demonstrations required to escape compounding physical errors.

Architect’s countermeasure: Retain all authority-record entries during dataset ingestion and admit, as training targets, only the stabilized, human-reviewed segments of attributable expert recoveries. Never prune demonstration corpora simply to minimize offline validation loss, and mandate closed-loop physical perturbation benchmarks to verify recovery competence before signing off on deployment candidate checkpoints.

Fallacy: Transitioning from human demonstrations to simulation reinforcement learning removes the failures training left in the policy.

An assembly team switches from behavioral cloning to simulation reinforcement learning to eliminate lateral drift during automated gasket seating. The resulting policy cures the drift in the virtual environment but commands joint torque oscillations at 65 Hz. The physics simulator permits this aggressive chatter because its rigid-body solver assumes massless motor rotors and instantaneous torque reversal. On the physical machine, this high-frequency command excites drivetrain mechanical resonance and rapidly heats the motor windings toward their thermal trip point via \(\TorqueProducingCurrent^2 \ElectricalResistance\) Joule losses. The drive electronics throttle current to prevent permanent motor damage, causing the assembly stroke to stall before seating completes. Shifting training regimes solved an imitation drift problem by introducing a high-frequency chatter exploit that the simulator did not penalize. Qualification must therefore evaluate the full physical task under sustained cyclic operation, verifying simulator dynamics assumptions against physical hardware telemetry. Switching to simulation reinforcement learning substitutes numerical solver shortcuts and unmodeled mechanical dynamics for human demonstration deficits.

Architect’s countermeasure: Require paired-trace validation across all four reality gap channels before deploying simulation-trained policies. Enforce low-level hardware rate limiters and digital low-pass action filters (\(\ActionFilterCutoffFrequency \le\) 20 Hz) within the embedded motor drives to suppress high-frequency chatter originating from unmodeled rotor inertia.

Fallacy: Assuming online gradient updates preserve predeployment safety envelopes.

When a simulation-trained policy shows steady-state positioning error on a physical assembly fixture, an engineer might enable online policy-gradient fine-tuning directly on the robot. After 5,000 gradient steps, the arm successfully eliminates contact positioning error at the fixture, prompting the team to declare the updated model production-ready. Applying parameter updates \(\FiniteChange\ModelParameters\) directly to the policy network, however, produces an entirely new checkpoint with changed weights. The behavioral trials accumulated during predeployment qualification remain observations of the original checkpoint and do not establish performance of the updated one. Independent permission and fallback properties may remain applicable only while their plant, timing, authority, isolation, and realization premises hold; integration must check those premises after the update. While local contact accuracy improves, unconstrained backpropagation into shared representations triggers catastrophic forgetting in rarely visited state-space regions. As a result, the fine-tuned policy may lose stability during high-speed free-space slews or corrupt its emergency deceleration profiles. Because physical robot time and mechanical wear preclude repeating full-scale testing suites on hardware, regression suites must isolate and verify every operational envelope that gradient updates might perturb. Online gradient updates require fresh behavioral qualification and reopen every assurance claim that depends on the changed checkpoint or execution conditions.

Architect’s countermeasure: When a bounded correction suffices, use gated residual policy architectures (\(\NominalControlCommand = \pi_{\text{base}} + \ResidualGate(\DecisionProcessState)\,\FiniteChange\Policy\)) with an externally enforced binary gate (\(\ResidualGate(\DecisionProcessState) \equiv 0\) in free space) to localize real-world adaptation. Mandate automated, frozen non-regression suites that verify trajectory tracking and stopping profiles before granting execution authority to updated parameters.

Fallacy: Believing that widening parameter distributions can substitute for missing physical mechanisms.

When an assembly policy repeatedly fails on physical hardware, an engineering team might attempt to brute-force sim-to-real transfer by radically widening the simulator’s domain randomization bounds. They randomize link masses by ±50 percent and vary surface friction coefficients across the extreme interval from 0.05 to 1.5. On the physical test bench, however, the trained policy behaves poorly: it moves sluggishly through free space and hesitates indefinitely upon initial contact. Randomization perturbs coefficients exclusively within the governing differential equations already hardcoded in the physics engine. It cannot synthesize missing physical phenomena such as gearbox backlash, joint cable compliance, Stribeck friction, or sensor transport jitter. When dynamically incompatible virtual plants produce indistinguishable observations, a memoryless policy cannot choose separately for each plant; average-return optimization may favor a slow compromise or may sacrifice low-probability cases. While recurrent or history-conditioned architectures can adapt to wide parametric variations by estimating system properties online, even an adaptive policy cannot identify or compensate for physical mechanisms that never existed in its training simulator. Expanding parameter randomization bounds merely shifts the probability distribution across existing simulator equations; it cannot synthesize unmodeled physical dynamics.

Architect’s countermeasure: Calibrate baseline physics engine parameters against real hardware test fixtures before introducing domain randomization. Explicitly document whether parameter distributions derive from empirical instrument measurements or heuristic curricula such as automatic domain randomization. Enumerate known physical omissions, including transmission elasticity and actuator deadbands, in the policy manifest’s omission log, ensuring that hardware-level safety watchdogs bound operations within verified modeling assumptions.

Fallacy: Assuming multi-step generative decoders can execute synchronously within inner actuator control loops.

Aiming to capture multimodal trajectory distributions observed in offline demonstration logs, an engineer replaces the single-pass ACT feedforward decoder in latch-bc-01 with an iterative 16-step diffusion denoising head. The team deploys the new architecture synchronously into the robot’s real-time inner control loop, scheduling a complete iterative sampling cycle for every 25 ms actuator setpoint. Under the hardware inference budget evaluated in section 6.2.6, executing all denoising steps consumes 48 ms, exceeding the allowable control step deadline. Because the synchronous dispatch pipeline stalls awaiting model convergence, the system routinely misses setpoint deadlines. The low-level controller must either freeze on stale command setpoints or trip safety watchdogs on every iteration overrun. The weight-streaming floor of The memory wall bounds each evaluation from below, and Action chunking shows that re-reading the action head on every evaluation can dominate the decoder’s cost; 16 dependent evaluations each pay that floor, so the sum misses the setpoint deadline before scheduling jitter is even counted. Deploying multi-step generative policies into high-rate actuator loops requires decoupling iterative trajectory generation from real-time setpoint tracking through asynchronous execution contracts.

Architect’s countermeasure: Decouple iterative policy inference from high-frequency actuator control loops whenever measured inference latency approaches the control period. Formally partition the pipeline into an asynchronous planner generating trajectory chunks and an inner controller tracking them under qualified execution leases. Explicitly track prediction horizons, executed setpoint prefixes, and proposal renewal deadlines. Buffering trajectory chunks never grants permanent operational authority: the execution engine continuously validates physical safety invariants and triggers safe stopping profiles whenever a lease expires without a fresh proposal.

6.10 Summary

A policy’s weights keep what its training experience rewarded and forget which experience that was, so each regime is best remembered by the dependency it leaves behind. Demonstrations leave a dependency on coverage: a point-regression head averages valid modes into an action no one chose, and even a multimodal head drifts into states the demonstrations never recovered from. A reward-labeled log leaves a dependency on the support of the logged actions, and a pretrained base on someone else’s embodiment. Trial on the machine pays for its experience in calendar time, winding heat, and drivetrain life; simulation pays almost nothing per trial but inherits every approximation in its contact solver and every mechanism it omits; fine-tuning on the machine creates a new checkpoint that needs its own behavioral evidence and review of affected assurance premises. None of these dependencies shows in a validation loss or a simulated return.

Orthogonal to the source of experience, every policy must deliver its proposal inside the machine’s deadline \(\CompletionDeadline\). The decoder that represents several modes buys that representation with dependent evaluations, and what decides whether it can run is renewal slack against the jitter allowance, not mean latency. When slack runs out, the chunk lease, not the decoder, bounds how long an admitted prefix keeps moving the arm.

The policy manifest gathers these dependencies into one claim with stated limits. Its declared ODD is the supported cells of the data join minus every state the omission log excludes, never a region read off a training curve. For the latch, that rule sends latch-bc-01 to evaluation at the nominal plate and the guarded approach, holds the simulation-refined checkpoints until paired probes bound the contact and timing omissions, and lets latch-bc-recovery-02 carry a wider claim that new recovery experience has to earn. An admitted action crossing the causal boundary carries no guarantee beyond the evidence that qualifies it, and the load-bearing relations behind that evidence are collected in table 5.

Table 5: Key equations of policy training: Each load-bearing equation of the chapter, the limit it bounds or the decision it settles, and the equation where it is derived.

Equation	What it bounds or decides	Derived in
\(\pi^*(\RobotObservation) = \mathbb{E}[\RobotAction \mid \RobotObservation]\)	The squared-error optimum of a point-regression policy, which lies between demonstrated modes	equation 3
\(\PolicyHistoryActionReturn(\DecisionProcessState, \RobotAction) = \Reward(\DecisionProcessState, \RobotAction) + \DiscountFactor\,\mathbb{E}[\PolicyHistoryActionReturn(\NextDecisionProcessState, \NextRobotAction)]\)	The value recursion that offline and policy-gradient learning estimate from samples	equation 2
\(\ProposerInferenceTime = \ObservationEncodingTime + \sum_{k=1}^{\DecoderEvaluationCount} \DecoderPassTime \le \CompletionDeadline\)	Whether a decoder fits the setpoint step synchronously or the renewal period asynchronously	equation 4
\(\TrialCampaignWallTime = \TrialCount\,(\TrialRolloutTime + \TrialResetTime)/\ParallelRigCount\)	The calendar of a physical training campaign, which rigs divide but wear does not follow	equation 5
\(\UnbrakedImpactForce = \SpeedAtDetection\sqrt{\ContactStiffness\,\Mass}\)	The peak force of a mass coasting into a contact, which a soft penalty spring understates	equation 7
\(\EffectiveSeriesStiffness = (1/\LatchPlateStiffness + 1/\RobotStructuralStiffness)^{-1}\)	The robot-side stiffness above which a coasting approach exceeds the latch limit	equation 8
\(\ContactYieldDeadline = \LatchForceLimit/(\ContactStiffness \SpeedAtDetection)\)	The time from touch to the force limit under constant drive, which confines the declared ODD to the guarded approach	equation 9
\(\DomainRandomizationObjective(\ModelParameters) = \mathbb{E}_{\SimulationParameterVector \sim \ProbabilityFunction(\SimulationParameterVector)}[\ExpectedReturnObjective(\ModelParameters; \SimulationParameterVector)]\)	Return averaged over the parameters a simulator models, never over mechanisms it omits	equation 10
\(\NominalControlCommand = \pi_{\text{base}} + \ResidualGate(\DecisionProcessState)\,\FiniteChange\Policy\)	Where a learned residual may act, and so what the regression evidence must cover	equation 11

Key Takeaways: Policy synthesis and sim-to-real transfer

Every regime leaves a dependency no training score shows: Offline loss and simulated return reward compliance with training data and numerical solvers rather than real-world physics. Candidate model rankings frequently invert when transferred to hardware. When an operating condition violates an unmodeled dependency, it produces a predictable failure signature that identifies exactly which runtime monitor the permission path must enforce.
Mode choice and recovery are separate problems: Point regression can average valid, mutually exclusive demonstrations into an unviable intermediate action, whereas compounding execution drift pushes the system into states that the demonstrations never sampled. Multimodal decoders resolve action ambiguity; targeted recovery experience and closed-loop data resolve distributional drift. Independent runtime limits remain mandatory for both.
Decoder latency must fit the proposal contract: Iterative denoising and deep autoregressive evaluations substantially increase compute latency and memory bandwidth pressure. When a policy cannot generate actions within the synchronous deadline, asynchronous action-chunk playback requires a validated hardware lease and deterministic fallback behavior. An expressive action distribution earns no extra execution time at the actuator.
Training information must survive the sensor boundary: Privileged simulation states, such as ground-truth contact wrenches or global object poses, accelerate training, but the deployed student policy observes only noisy onboard sensor streams. Observation histories disambiguate physical variations only when those conditions leave distinct temporal signatures. Model distillation never removes the necessity for hardware calibration and empirical transfer testing.
Randomization covers parameters, not missing mechanisms: Domain randomization widens policy robustness across modeled simulator parameters, but it cannot bridge physical effects that the numerical solver neglects. Backlash, unmodeled compliance, and sensor transport delays remain absent regardless of parameter range, requiring explicit entries in the omission log and channel-by-channel probe characterization on hardware.
Fine-tuning spends wear and reopens qualification: On-robot trials consume component life, introduce thermal stress, and generate wear that cannot be parallelized across separate test rigs. Behavioral evidence gathered for an earlier checkpoint does not qualify an updated model, and independent permission properties require an integration check of their unchanged premises. A gated residual architecture strictly confines where learned adaptations take effect, but the integrated controller still demands thorough regression testing and hardware-level execution limits.
A wider declared ODD needs new support: The policy manifest records dataset digests, training hyperparameters, and unmodeled simulator phenomena, explicitly excluding operational domains affected by unverified gaps. Scaling simulator throughput or pretraining a foundation model adds no supported cell to the declared ODD. Broadening the deployment boundary requires attributable physical recovery data and rigorous closed-loop hardware qualification.

The declared ODD is where the second law, endogenous data (principle 2), becomes a field in a record. A policy chooses its next observations by acting, so the states it meets in service are not the states its training visited, and no regime in this chapter escapes that. Cloning meets it as drift out of the demonstrated tube, learning by trial on the machine meets it as calendar time and wear spent visiting the policy’s own states, and simulation meets it as states whose dynamics the solver invented. The manifest does not remove the dependency. It writes down where the policy’s own evidence ends, so that everything beyond that line is treated as a claim still to be earned rather than as competence the weights happen to display.

When a candidate looks better than its evidence, the useful question is which of its states were visited on this machine and which were only scored. A wider declared ODD is bought with experience that joins to new cells, and choosing a training regime is choosing what that experience costs: an operator’s hours, a drivetrain’s life, or a probe campaign that closes a simulator’s omissions.

6.11 Historical Perspective and Further Reading

The mathematical foundations of modern policy optimization originate in classical dynamic programming. Bellman (1957) introduced the principle of optimality and the recursive decomposition that defines value functions, while Howard (1960) formulated policy iteration to solve discrete Markov decision processes. Parallel to dynamic programming, learning from demonstration developed along an empirical path in robotics. Pomerleau’s ALVINN demonstrated early autonomous road following by training a shallow artificial neural network to map camera images directly to steering angles (Pomerleau 1989). ALVINN quickly exposed the vulnerability of pure behavioral cloning to covariate shift and compounding error: because the network trained exclusively on demonstrations from an attentive human driver, it never encountered the off-center heading angles produced by its own steering mistakes, leading to catastrophic departures from the roadway (Pomerleau 1991). Ross et al. (2011) formalized this failure mode through the Dataset Aggregation (DAgger) algorithm, proving no-regret reduction bounds by querying an expert supervisor on the actual state distribution induced by the learner’s policy. When demonstrations encompass multiple valid strategies, naive regression averages distinct modes into invalid intermediate actions. Mixture density networks (Bishop 1994) resolved this multimodality for low-dimensional outputs, and diffusion policies (Chi et al. 2025) generalized the approach to high-dimensional action trajectories, executing action chunks with receding horizons on physical manipulators.

Pomerleau, Dean A. 1989. “ALVINN: An Autonomous Land Vehicle in a Neural Network.” Advances in Neural Information Processing Systems (NeurIPS) 1: 305–13.

Pomerleau, Dean A. 1991. “Efficient Training of Artificial Neural Networks for Autonomous Navigation.” Neural Computation 3 (1): 88–97. https://doi.org/10.1162/neco.1991.3.1.88.

Watkins, Christopher J. C. H., and Peter Dayan. 1992. “Q-Learning.” Machine Learning 8 (3–4): 279–92. https://doi.org/10.1007/BF00992698.

Levine, Sergey, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. 2018. “Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection.” The International Journal of Robotics Research 37 (4-5): 421–36.

Kalashnikov, Dmitry, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, et al. 2018. “Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation.” Proceedings of the 2nd Conference on Robot Learning, Proceedings of machine learning research, vol. 87: 651–73.

Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. “Proximal Policy Optimization Algorithms.” arXiv Preprint arXiv:1707.06347.

Makoviychuk, Viktor, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, et al. 2021. “Isaac Gym: High Performance GPU-Based Physics Simulation for Robot Learning.” arXiv Preprint arXiv:2108.10470.

Rudin, Nikita, David Hoeller, Philipp Reist, and Marco Hutter. 2022. “Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning.” Proceedings of the 5th Conference on Robot Learning, Proceedings of machine learning research, vol. 164: 91–100.

Algorithmic reinforcement learning split into two primary branches: policy-gradient methods originating with REINFORCE (Williams 1992) and value-based temporal-difference learning formalizing around Q-learning (Watkins and Dayan 1992). Both paradigms measure sample complexity in abstract environmental transitions, ignoring the physical costs of real-time execution. On a real robot, every trial consumes calendar time, strains mechanical linkages, overheats actuators, and demands human intervention for environment resets. Large-scale physical grasping campaigns (Levine et al. 2018; Kalashnikov et al. 2018) demonstrated that collecting hundreds of thousands of real-world interactions requires dedicated robot farms running continuously for months under heavy maintenance. To bypass this physical bottleneck, robotic learning shifted sample generation into simulation. Proximal Policy Optimization (Schulman et al. 2017) established itself as the standard policy learner by clipping the update objective to discourage excessive policy drift. Simultaneously, physics engines co-located directly on hardware accelerators eliminated host-to-device communication bottlenecks, enabling thousands of parallel rollouts on a single workstation and delivering billions of simulated timesteps per day (Makoviychuk et al. 2021; Rudin et al. 2022).

Simulated rollouts solved the sample-throughput bottleneck, but they elevated the reality gap into the central engineering challenge. Policies trained inside idealized physics often fail catastrophically when deployed against the unmodeled friction, latency, and compliance of physical hardware. Domain randomization addressed this discrepancy by treating the real world as an unobserved draw from a randomized distribution of simulated environments, perturbing visual features (Tobin et al. 2017) and inertial dynamics (Peng et al. 2018). The OpenAI Dactyl project (Andrychowicz et al. 2020) showed that broad dynamics randomization paired with a recurrent policy could reorient a block in a dexterous hand trained entirely in simulation, while reporting the unsimulated failures of the physical hand, chiefly hardware breakage under load; its successor extended the approach to solving a Rubik’s cube with automatic domain randomization (OpenAI et al. 2019). Classical system identification (Ljung 1999) found new life inside neural pipelines through learned actuator networks that capture non-ideal actuator dynamics and drive compliance (Hwangbo et al. 2019). More recently, generalist foundation policies trained across diverse robotic platforms (Kim et al. 2024; Black et al. 2024) use pooled fleet demonstrations to cross-train visual and motor representations. However, borrowing cross-embodiment data introduces unverified physical assumptions and subtle distributional shifts. The policy manifest introduced in this chapter provides the rigorous systems contract required to track these lineage dependencies, documenting simulation distributions, sensor bounds, and training assumptions so that hardware deployment can be audited and verified.

OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, et al. 2019. “Solving Rubik’s Cube with a Robot Hand.” arXiv Preprint arXiv:1910.07113.

Hwangbo, Jemin, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. 2019. “Learning Agile and Dynamic Motor Skills for Legged Robots.” Science Robotics 4 (26): eaau5872. https://doi.org/10.1126/scirobotics.aau5872.

Kim, Moo Jin, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, et al. 2024. “OpenVLA: An Open-Source Vision-Language-Action Model.” arXiv Preprint arXiv:2406.09246.

Black, Kevin, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. 2024. “\(\pi_0\): A Vision-Language-Action Flow Model for General Robot Control.” arXiv Preprint arXiv:2410.24164.

Six foundational works trace the trajectory from classical dynamic programming to modern sim-to-real policy deployment:

Sutton and Barto (2018) provides the definitive derivation of Markov decision processes, value functions, and policy-gradient theorems from first principles, serving as the core reference for the reinforcement learning formulations in section 6.2.1.
Ross et al. (2011) establishes the theoretical bound on compounding error in sequential prediction, introduces the DAgger reduction algorithm, and outlines the recoverability assumptions that section 6.2.4 evaluates on physical robot hardware.
Chi et al. (2025) introduces score-based diffusion models for action sequence generation on physical manipulators, establishing the receding-horizon chunk execution and iterative denoising computational budgets analyzed in section 6.2.6.
Peng et al. (2018) focuses domain randomization on unmodeled physical dynamics rather than visual textures, demonstrating that recurrent policies implicitly identify system dynamics across randomized friction, mass, and damping distributions to achieve robust sim-to-real transfer.
Andrychowicz et al. (2020) offers one of the most comprehensive engineering case studies of an end-to-end sim-to-real deployment, detailing the distributed training pipeline, dynamics randomization ablations, and physical maintenance challenges encountered while reorienting a block with an anthropomorphic hand.
Ljung (1999) provides the definitive treatise on classical system identification, detailing the input-output parameter estimation techniques, excitation design, and validation metrics that ground modern actuator modeling and paired physical probing.

Sutton, Richard S., and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. 2nd ed. MIT Press.

Ross, Stéphane, Geoffrey Gordon, and J. Andrew Bagnell. 2011. “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning.” Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS) 15: 627–35.

Chi, Cheng, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. 2025. “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion.” The International Journal of Robotics Research 44 (10-11): 1684–704. https://doi.org/10.1177/02783649241273668.

Andrychowicz, Marcin, Bowen Baker, Maciek Chociej, Rafał J’ozefowicz, et al. 2020. “Learning Dexterous in-Hand Manipulation.” The International Journal of Robotics Research 39 (1): 3–20.

What’s Next: From policy manifests to evidence

Twenty successful consecutive trials provide limited statistical confidence for a demonstration checkpoint whose manifest declares a single nominal cell and inherits none of the simulator’s omissions. Closed-Loop Evaluation takes the formal policy manifest as its contractual input. It opens with an illustrative campaign of twenty nominal door trials of latch-bc-01, the single cell its manifest declares, and evaluates the bound so small a physical sample can establish. It then evaluates latch-bc-recovery-02 through nominal regression and displaced-plate trials to decide whether the new recovery experience earns the wider claim its manifest requests within the guarded approach.