Responsible Physical AI

17 Responsible Physical AI

Purpose

Physical AI stack focus for this chapter.

Who answers for a machine that moves on its own when its motion harms someone?

A machine’s logs can show, tick by tick, what it sensed, proposed, and admitted, and still not show who decided that it could travel at that speed, among those people, with that person as its only backup. When harm arrives, blame settles on whoever stood closest to the event, often an operator handed a duty no person could have carried out, while the decisions that set the machine’s limits stay spread across engineering teams, suppliers, site managers, and regulators. The machine’s effects also reach people who never chose it: pedestrians and coworkers whose detection rate was never measured for people like them, workers whose jobs shift to rescuing and watching it, and bystanders whose faces fill the records kept to explain its mistakes. A machine is fit to be given authority only when each decision behind its actions names the person who answers for it, each human role it relies on is one a person can actually perform, and each group that bears its risk or its recording has been counted. Assigning responsibility after a collision cannot take back the energy the machine has already delivered across the causal boundary, so it has to be assigned before the machine moves.

Learning Objectives

• Explain why reconstructing what a machine did does not establish who answers for it, and trace an admitted action back to the decisions that allowed it
• Evaluate whether a human role gives its holder the time, information, and authority to act, using the takeover budget and supervisor utilization
• Specify the stages of operating permission, what each stage allows and must report, and what withdraws it
• Calculate how a gap in per-frame detection between groups of people becomes a gap in stopping distance and impact energy
• Size a per-group evaluation whose confidence interval can resolve a detection gap of a stated size
• Distinguish risk borne by people who chose the machine from risk borne by people who did not, and specify exposure reporting for each group
• Design an approval record that names an owner for every decision behind an admitted action, so that a missing owner blocks release

17.1 Who Answered for the Cruise Collision

Release left the running machine with a signed manifest: a verdict, the records it binds by digest, and the standing conditions that withdraw it. The manifest names the lead systems adjudicator who signed it. It does not name who decided the aisle speed the warehouse would accept, who decided that one remote assistant would serve every machine on the floor, or who would report an incident, to whom, and how fast, if the machine hurt someone. Those decisions shape what the machine does once a command crosses the causal boundary as surely as the enforcer’s thresholds do, and none of them appears in a per-tick log.

The Cruise collision in San Francisco on the night of October 2, 2023 shows what happens when they are left unassigned. A human-driven car struck a pedestrian in a crosswalk and threw her into the path of a driverless Cruise robotaxi, which braked hard, ran over her, and stopped. With her still beneath it, the vehicle then began a pullover toward the curb and moved about 20 ft (6.1 m) at up to 7.7 mph before it stopped (Quinn Emanuel Urquhart and Sullivan, LLP 2024). The engineering failure belongs to the permission path. A post-collision classifier placed the pedestrian at the side of the vehicle rather than in front of it, and the fallback it selected moved the vehicle without any evidence that the space beneath it was clear.

Quinn Emanuel Urquhart and Sullivan, LLP. 2024. Report to the Boards of Directors of Cruise LLC, GM Cruise Holdings LLC, and General Motors Holdings LLC Regarding the October 2, 2023 Accident in San Francisco. Quinn Emanuel Urquhart; Sullivan, LLP.

↩︎ The fallback rule that forbids renewed motion while the space beneath a vehicle is unverified is established in The four rungs of the fallback ladder.

The machine’s own records were enough to reconstruct that motion. The engineers who analyzed the event for the company’s boards rebuilt it from the vehicle’s logs and video, including the dropped pedestrian track before impact and the pullover after it (Quinn Emanuel Urquhart and Sullivan, LLP 2024). What failed after the collision was the account given to the people who had licensed the vehicle. On October 3, a Cruise representative told an analyst at the California Public Utilities Commission (CPUC) by telephone that the vehicle had stopped on impact and contacted remote assistance; the description left out the pullover and the dragging. The commission later counted 15 days, October 3 through 18, in which Cruise withheld a full account, and it received the full video on October 19 (California Public Utilities Commission 2024). Cruise’s crash reports to the National Highway Traffic Safety Administration (NHTSA) also omitted the dragging (U.S. Attorney’s Office, Northern District of California 2024). On October 24, the California Department of Motor Vehicles (DMV) suspended Cruise’s permits for driverless testing and driverless deployment, citing among its grounds that the company had misrepresented information related to the safety of its autonomous technology (California Department of Motor Vehicles 2023). A review commissioned by Cruise’s boards found no evidence that anyone intended to mislead regulators, but it described a “passive, non-transparent approach” to disclosure and a leadership focused on correcting media coverage of the crash (Quinn Emanuel Urquhart and Sullivan, LLP 2024).

Each group involved held one piece of the outcome: the engineers who designed the classifier and the fallback, the remote-assistance staff, the people who drafted the reports, and the executives who managed the response. Each could describe its own piece truthfully, and none of those descriptions said who had decided the whole. The CPUC had granted Cruise commercial driverless service in San Francisco at any hour seven weeks earlier (California Public Utilities Commission 2023). The logs answered what the vehicle did. They did not answer who had decided that a driverless vehicle should pull over after a side impact without checking beneath itself, who decided what regulators would be told, or who answered for either decision. Five years earlier, after the Uber test vehicle killed a pedestrian in Tempe, the person who answered in court was the vehicle’s backup operator, who pleaded guilty to endangerment in 2023 (Maricopa County Attorney’s Office 2023).

Maricopa County Attorney’s Office. 2023. Driver of Uber Vehicle Involved in Death of Woman in Tempe Pleads Guilty.

Responsibility for a physical machine therefore has to be engineered like any other premise of its authority: written down before the machine moves, attached to the decisions that set its limits, and checked when it is released. The same questions arise on the warehouse machine without a public road in sight. A remote assistant answers for takeovers, facility management for the crossing rule that keeps people out of the aisle, a data team for the camera traces the black box keeps, and someone, so far unnamed, for the decision that one person can watch every machine on the floor. The sections that follow take these decisions in turn: the links behind one admitted action, the human roles the design relies on, the stages of permission to operate, the detection of different groups of people, the people who bear the machine’s risk and appear in its records, and the labor it moves rather than removes. The approval record collects these decisions in one place that the release gate reads, so that a decision with no named owner blocks operation the way an unknown premise does.

17.2 Decisions Behind an Admitted Action

Consider one command the enforcer admitted in the aisle: a base velocity of 1.3 m/s toward a rack end. The enforcement record shows that it passed the stopping check at that tick, under the thresholds in force, yet behind that single log line stand at least six decisions made by different people at different times. Four of them leave traces in the engineering records built in earlier chapters: the learning team trained and approved the policy weights that proposed the command, the safety engineering team built and signed the enforcer and its thresholds, the intervention design and the remote assistant on shift set the authority state at that tick, and the lead systems adjudicator signed the operating envelope—including the aisle speed and the floor-friction premise beneath it—into the release manifest. The other two decisions govern the site rather than the machine: facility management wrote the crossing rule that lets people enter the aisle only at marked points, while operations decided how many machines one remote assistant would serve. Figure 1 draws the six links and the record, if any, that holds each one.

↩︎ The lead systems adjudicator, whose signature binds the release verdict to a named person, is established in The Three Verdicts.

Figure 1: Six decisions behind one admitted command: An aisle command admitted at one tick traces back to decisions made weeks earlier by six different parties. Four of them land in a record the book has built (the policy manifest, the enforcement record, the authority record, and the release manifest); the crossing rule and the staffing ratio land in none, so no log can name who made them. The approval record lists each decision with its owner and refuses operation while any link has no owner.

When each of these decisions belongs to a different party, every party can truthfully say that the harm came from someone else’s part. Thompson called this the problem of many hands in his study of public officials, where responsibility for a policy that many people shaped ends up resting on no one (Thompson 1980). Nissenbaum showed that computer systems make it worse: they are built by many people, they fail in ways no single person caused, and their failures are easily excused as bugs that no one could have prevented (Nissenbaum 1996). A learned proposer adds a further excuse. Its behavior was learned rather than written, so no engineer wrote the rule it followed, and Matthias argued that machines which learn create a real gap in who can be held responsible for what they do (Matthias 2004). The architecture of this book closes that gap at the point where it matters. The model proposes and never admits (principle 3), so every admitted action passed a check that someone designed, under an envelope that someone signed, and “the model did it” is never a complete answer.

17.2.1 Decisions that leave no trace in a log

When an incident investigator opens a machine’s runtime logs, the decisions that made the incident possible are missing because most of them happened weeks or months before a command crossed the causal boundary. The book’s records capture runtime events: an observation and its age, a proposal, an admission, a change of authority, a fault trial. By contrast, a safety engineer approving an operating envelope, an operations manager setting a staffing ratio, a team disabling a protective function, a company applying for a permit, and a data team declaring what its camera traces may be used for all happen in meetings and documents long before the machine moves. In the Cruise case, the rule that a side impact should trigger a pullover was a design decision made long before October 2, and the choices about what to tell regulators were made in the days after. Neither could appear in the vehicle’s log, because the vehicle did not make them.

Such decisions have to be recorded on purpose, and a chain of them is less complete than any one of its links. Suppose each of the 6 links behind the aisle command can be retrieved when an investigator asks with probability 0.99, independently of the others. The whole chain then resolves with probability \(0.99^{6} = 0.94\). At 0.95 per link it resolves with probability 0.74, and at 0.90, roughly what a decision held only in memory and email achieves once its author has changed jobs, with probability 0.53. An investigator would then find the full chain in about 53 percent of incidents. The same arithmetic makes a conjunctive release argument weaker than any one of its premises, and the remedy is also the same: each link has to be written down as part of the design, not left to accumulate.

↩︎ The conjunctive arithmetic that makes an argument no stronger than its weakest premise is established in Contextual assumptions and the weakest link.

These links are held in the decision block of the approval record (section 17.9). For each decision it names the type of decision, the role that owns it, the signed approval, the evidence it relied on, and the date after which it must be made again. A link with no owner is not a gap in paperwork; it is a premise of the machine’s authority that no one has accepted.

17.3 Oversight a Person Can Perform

Placing a human operator in the loop—a safety driver behind the wheel, a remote assistant at a console, or a spotter walking beside a robot during commissioning—often shifts blame onto that person without adding any real protection. A person in the loop provides genuine oversight only if three conditions hold: the person must have the time to act before the hazard arrives, the information to understand what the machine is doing, and the authority to change what it does, not only to report it. Santoni de Sio and van den Hoven state the same requirement in two parts: the system must respond to the reasons of the people meant to control it, and every outcome must trace back to someone who understood both the system and their own role in it (Santoni de Sio and Hoven 2018).

Definition 17.1: Human oversight role

A human oversight role is a duty that a design assigns to a person, such as stopping the machine, taking over a channel, or approving a motion, and that the safety argument relies on. The role counts as a countermeasure only if its holder has the time, the information, and the authority to carry it out within the time the hazard leaves.

1. Significance: A role that fails any of the three conditions adds no protection, yet it moves blame onto the person who holds it, so every role the safety case cites must be tested like any other premise.
2. Distinction: The takeover budget measures how long a person needs; the oversight role asks whether the design leaves that much time, gives that person what they need to see, and lets them act on it.
3. Common pitfall: A frequent misconception is that adding a person to the loop makes a design responsible. A person who cannot act in time is a place to put blame, not a safety layer.

When a role fails any of these conditions, the person in it absorbs responsibility for control they never had. Elish calls the person in that position a moral crumple zone: like the crumple zone of a car, the person absorbs the impact, except that here the impact is blame, and what the crumpling protects is the reputation of the system rather than the person (Elish 2019). The three conditions give a design team a test it can apply before the machine moves.

17.3.1 Time to act

Time is the condition most often violated, and it is the easiest to check. At the 1.3 m/s aisle speed, a person who steps out 1.10 m ahead of the machine leaves 0.85 s before the rack end. A remote assistant who has been monitoring rather than driving needs about 4.5 s to take over, the design out-of-loop takeover time, during which the machine travels 5.8 m. The person needs about 5× the time the hazard leaves. No training, attention, or diligence closes a gap of that size, so a design that names the remote assistant as the countermeasure for a person stepping out at a rack end has not added a layer of protection. It has recorded in advance who will be blamed.

↩︎ The out-of-loop takeover time and its price in the stopping budget are established in Human reaction time and travel before braking.

The engineering consequence is already in the book’s design. The machine stops itself first and asks for help from standstill, where waiting costs time but not clearance. What the time test adds is the duty to write that choice down. For every human role the safety case relies on, the approval record lists the response time the role requires, the time the hazard leaves, and a verdict of FEASIBLE, INFEASIBLE, or UNKNOWN. A role marked INFEASIBLE may stay in the design as a productivity resource; it may not appear in the safety argument as a countermeasure.

17.3.2 One person, many machines

A remote assistant who serves many machines is a server in a queue, and queueing theory says how busy that person is. If help requests arrive from the whole fleet at rate \(\ArrivalRate\) and each holds the assistant for a mean time \(\IntervalMean\), the share of time the assistant is busy, the utilization, is \[\ServingUtilization = \ArrivalRate\,\IntervalMean. \tag{1}\] A request that arrives while the assistant is busy waits. Under the simplest queueing model, with random arrivals and random service times, the mean wait before service begins is \[\QueueWaitTime = \frac{\ServingUtilization\,\IntervalMean}{1-\ServingUtilization}. \tag{2}\] The wait grows without bound as \(\ServingUtilization\) approaches one. Moreover, a request arriving while the assistant is busy, which happens with probability \(\ServingUtilization\) in this model, gets no human attention at all until the request ahead of it is finished.

Napkin Math 17.1: One assistant, twenty machines

Problem: A site plans to run 20 mobile manipulators with one remote assistant. Each machine asks for help 0.5 times per operating hour, and each request holds the assistant for 3 min on average. How busy is the assistant, and how long does a request wait?

Variables:

• Requests per machine: 0.5 per operating hour (Illustrative); mean service time \(\IntervalMean =\) 180 s (Illustrative).
• Machines served: 10, 20, or 32 (Illustrative).

Math: By equation 1, 20 machines give \(\ServingUtilization =\) 0.50, and equation 2 gives a mean wait \(\QueueWaitTime =\) 180 s. One request in a hundred waits longer than 23 min. With 10 machines the figures are 0.25 and 60 s; with 32 they are 0.80 and 720 s, about 12 min.

Result: The assistant is busy 50 percent of the time, and a typical request waits minutes. Adding machines past that point makes the wait grow much faster than the fleet.

Systems insight: Every one of these waits is longer than the 0.85 s a person stepping out at a rack end leaves. The ratio of machines to assistants is therefore a productivity budget and never a safety layer. A staffing plan that caps utilization at 0.5 limits this assistant to 20 machines, and the approval record lists that cap as the decision of whoever set it.

↩︎ Admitting a remote-assistant session only when the machine is in a state where waiting costs no clearance is established in Admitting the remote-assistant session.

The queue shows why the staffing ratio is a decision someone must own. A manager who adds machines to one assistant’s load is not changing a number in a spreadsheet; that manager is deciding how long a stranded machine, and the people working around it, will wait for a person. If that decision is written down with the utilization it implies and an owner, it can be reviewed when the fleet grows. If it is not, it changes silently every time a machine is added.

17.3.3 Information and authority

Time is necessary but not enough. A person with time to act must also know what the machine is doing and be able to change it. A console that shows the camera feed but not which channel the machine still owns leaves the operator guessing about the most important fact of a handover, and an operator who believes they hold a channel they do not hold can make a takeover worse. A role whose only output is a report, a flag raised to a supervisor or a note in a log, gives its holder no authority over the motion at all. The test for authority is mechanical. The person must be able to stop the machine, through a channel the permission path honors within the time the hazard leaves, without asking anyone else.

↩︎ Mode confusion, and the per-channel annunciation that keeps an operator’s picture of who owns each channel current, are established in Authority Handover Failure Modes.

For each human role, the approval record therefore holds three fields beside the timing verdict: the information channel the person reads, the authority channel the person can act through, and whether that channel reaches the permission path. A remote assistant on the running machine passes the information and authority tests for a stranded machine at standstill and fails the time test for a person stepping out at a rack end. Both facts belong in the record, because they assign that one person two different duties.

Checkpoint 17.1: Oversight a person can perform

A design names three people in its safety argument: the remote assistant, a teach-pendant operator during commissioning, and a floor supervisor who watches the fleet dashboard.

• Time: Can you state, for each role, the time the role requires and the time the relevant hazard leaves, and mark the role FEASIBLE, INFEASIBLE, or UNKNOWN?
• Load: A site runs 24 machines with one assistant; requests arrive at 0.8 per machine-hour and take 75 s. Can you compute the assistant’s utilization and mean wait, and say whether the assistant may be listed as a countermeasure for any hazard faster than a minute?
• Authority: Can you say which of the three roles can stop the machine through a channel the permission path honors, and which can only report?

17.4 Permission in Stages

Granting a machine full operating permission on day one exposes the public or the workforce before the configuration has accumulated the field evidence needed to support that exposure. Regulators of driverless vehicles therefore grant permission in stages. California’s rules separate testing with a safety driver, testing without one, and deployment to the public, each under its own permit and conditions (California Department of Motor Vehicles 2018a, 2018b). The CPUC adds a second ladder for carrying passengers. Before August 2023, Cruise could carry paying passengers without a safety driver only in limited areas of San Francisco between 10 p.m. and 6 a.m., could carry paying passengers anywhere in the city at any hour only with a safety driver present, and could give free driverless rides anywhere at any hour. On August 10, 2023, the commission removed the time limit on driverless fared service (California Public Utilities Commission 2023). Each rung named a configuration, a place, a time, and who could be aboard.

California Department of Motor Vehicles. 2018b. California Code of Regulations, Title 13, Article 3.8: Deployment of Autonomous Vehicles.

California Public Utilities Commission. 2023. CPUC Approves Permits for Cruise and Waymo to Charge Fares for Passenger Service in San Francisco.

Definition 17.2: Permission stage

A permission stage is a grant, made by the party entitled to grant it, that allows a named configuration of a machine to operate in a named domain, among named groups of people, on named evidence, and under named reporting duties. A machine moves to the next stage only on evidence gathered at the current one, and a stage can be withdrawn for a configuration change, an incident, or a withheld record.

1. Significance: Exposure grows one stage at a time, so the evidence for the next stage is gathered under the protection of the current one, and a failure at one stage costs only what that stage exposes.
2. Distinction: A release verdict is the engineering decision about whether a configuration’s claims are supported; a permission stage is the grant that decides who may be exposed to that configuration and what must be reported about it.
3. Common pitfall: A frequent misconception is that a stage, once reached, belongs to the company. It belongs to a configuration, and it carries duties that, if broken, withdraw it.

Each stage carries reporting duties, and the duties grow with the stage. California requires a manufacturer testing on public roads to report, within ten days and on a set form, any collision that causes damage, injury, or death, and to file an annual report of every disengagement, each transfer of control from the automation to the safety driver (California Department of Motor Vehicles 2018a). Since 2021, a federal order has required companies operating automated driving systems to report crashes to NHTSA; under the version in force in 2023, a crash that involved a death, an injury treated at a hospital, or a pedestrian or other vulnerable road user had to be reported within one day, with an update at ten days (National Highway Traffic Safety Administration 2021). The disengagement reports are the data behind A decade of reported takeovers, whose two series come from two different stages, testing with a driver and testing without one.

A warehouse has no public permit office, but the same ladder exists. The machine is commissioned with a spotter walking beside it, then allowed to run unattended in a zone people cannot enter while it operates, and only then released onto an open floor among workers. The facility owner, the integrator, and the workers’ representatives grant these stages, and the approval record names who granted each one, for which configuration, on what evidence, and with which reporting duties. The stage and the verdict are recorded separately. A COND verdict for restricted inspected-route delivery says which claims the evidence supports; the stage says whether that restricted service may run on the open floor yet.

17.4.1 Leaving a stage on evidence

A stage is left on evidence, not on a calendar date, and the evidence a lower stage produces is partly hidden by the protection that stage provides. A safety driver or a spotter masks failures: when the person intervenes, the machine’s own error never reaches its consequence, and the stage records an intervention rather than a collision. Counting failures at the supervised stage therefore means reconstructing, for each intervention, whether the machine would have failed without it, which is why the disengagement reports matter and why they must say why control was transferred.

↩︎ Interventions that end a trial before its outcome is known, and how they are counted, are established in Confidence from a Finite Campaign.

The exposure arithmetic sets how long a stage must run. To claim, at 95 percent confidence, fewer than one safety-relevant intervention per 1,000 operating hours after observing none, the machines must accumulate about 2,996 failure-free operating hours. A fleet of 10 machines running 16 hours a day reaches that in 19 days. Those hours support the claim only for the configuration that ran them. A new policy, a new enforcer build, or a new site layout starts its own count, because the evidence belongs to a configuration, and so does the stage it supports.

17.4.2 Withdrawal and disclosure

Permission is withdrawn the way it is granted, one stage at a time. The DMV’s October 2023 order suspended Cruise’s driverless testing and deployment permits but left in place its permit to test with a safety driver (California Department of Motor Vehicles 2023), stepping the company back down the ladder rather than off it. The grounds the DMV cited show what a permit protects. One was that the vehicles were not safe for public operation, a finding about the motion. Another was that the company had misrepresented information related to the safety of its technology, a finding about the record.

California Department of Motor Vehicles. 2023. DMV Statement on Cruise LLC Suspension.

California Public Utilities Commission. 2024. Decision 24-07-004: Presiding Officer’s Decision Approving Offered Settlement Terms and Granting Deferral of the Order to Show Cause Proceedings.

National Highway Traffic Safety Administration. 2024. NHTSA Announces Consent Order with Cruise After Company Failed to Fully Report Crash Involving Pedestrian.

U.S. Attorney’s Office, Northern District of California. 2024. Cruise Admits to Submitting a False Report to Influence a Federal Investigation and Agrees to Pay $500,000.

The regulatory fines that followed were for the record. The CPUC counted the 15 days in which it lacked a full account, applied its daily maximum of $7,500 to each, and fined Cruise $112,500 (California Public Utilities Commission 2024). In September 2024, NHTSA imposed a consent order with a $1.5 million penalty for crash reports that omitted the dragging (National Highway Traffic Safety Administration 2024). In November 2024, Cruise entered a deferred prosecution agreement and paid a $500,000 criminal fine for submitting a false record to influence the federal investigation (U.S. Attorney’s Office, Northern District of California 2024). Regulators withdrew authority from the company because of what its vehicle did and because of what the company did not tell them about it.

The incident disclosure duty is the obligation, attached to an operating permit, to report an incident completely and on time to the parties the permit names. It is a decision with an owner like any other: someone decides what an incident report contains, someone signs it, and someone decides when a record has been withheld long enough to break the permit. The permit block of the approval record therefore lists, for each stage, the regulator or grantor, the configuration the stage covers, the reporting duties with their deadlines, and a disclosure log of every report sent, with its author and the evidence attached. A release manifest that binds raw traces to the configuration that produced them, and that an assessor can open, is the engineering half of that duty. The disclosure log is the other half, because it shows what was actually sent, to whom, and when.

↩︎ The release manifest that binds raw traces to the configuration that produced them is established in The Release Manifest.

17.5 When Detection Depends on Who Is Detected

Learned detectors do not see all people equally well. Buolamwini and Gebru found that commercial face-analysis systems misclassified darker-skinned women far more often than lighter-skinned men, and that the benchmarks used to evaluate them contained few of the people they failed on (Buolamwini and Gebru 2018). Pedestrian detection shows the same pattern. Wilson, Hoffman, and Morgenstern annotated pedestrians in a driving dataset by skin type and found that standard detectors reached an average precision of 67.1 percent on lighter-skinned pedestrians and 55.9 percent on darker-skinned ones under a strict overlap criterion, a gap that time of day and occlusion did not explain (Wilson et al. 2019). Testing widely used pedestrian detectors, Li and colleagues found the proportion of children left undetected 20.14 percent higher than that of adults, and significant gender bias at night (Li et al. 2025).

A warehouse has its own groups. A detector trained mostly on people walking upright may see a kneeling worker, a person carrying a carton that hides the torso, a wheelchair user, or a worker without a high-visibility vest later or less often. Where the machine relies only on its certified safety scanner, which stops for any object above its resolution without deciding what it is, these differences do not matter. They matter wherever the design relies on a learned detector to recognize a person: to slow down when a person is near, to keep a wider separation from people than from pallets, or to predict where a person will walk. In those places, a difference in detection becomes a difference in how close the machine comes to each group before it begins to brake.

17.5.1 From a detection gap to a stopping gap

When a detector sees one group of people less reliably than another, the machine travels farther before braking begins and can strike a person in the less-detected group even when the stop was sized to clear the best-detected group. The conversion from a detection rate to a stopping shortfall runs through the camera’s frames. Suppose a design requires that the chance of missing a person in every frame of a window be no more than \(\DetectorMissProbability\), and that a detector reports a person in view in each frame with probability \(\PerFrameDetectionProbability\), independently from frame to frame. The number of consecutive frames the person must be in view before the design may count on detection is \[\ReliableDetectionFrameCount = \left\lceil \frac{\ln \DetectorMissProbability}{\ln\left(1-\PerFrameDetectionProbability\right)} \right\rceil. \tag{3}\] A group the detector sees less often needs more frames, and each extra frame is travel before the machine can count on knowing the person is there. If the stop was sized for the best-detected group, a group that needs \(\FiniteChange\ReliableDetectionFrameCount\) more frames, at camera frame period \(\CameraFramePeriod\) and speed \(\SpeedAtDetection\), loses a distance \[\DetectionRangeShortfall = \SpeedAtDetection\,\CameraFramePeriod\,\FiniteChange\ReliableDetectionFrameCount \tag{4}\] before braking begins. A stop that would have ended just short of a person in the reference group now ends at the person, and the machine is still moving when it gets there. While the shortfall is smaller than the braking distance, braking at \(\CredibleBrakingDeceleration\) over the remaining distance leaves an impact speed and an impact energy of \[\ImpactSpeed = \sqrt{2\,\CredibleBrakingDeceleration\,\DetectionRangeShortfall}, \qquad \KineticEnergy = \tfrac{1}{2}\Mass\ImpactSpeed^2 = \Mass\,\CredibleBrakingDeceleration\,\DetectionRangeShortfall. \tag{5}\] The last form says that the energy delivered to the person grows in direct proportion to the distance lost to late detection. Fairness for a moving machine is therefore measured in meters and joules, not only in percentage points of recall.

Napkin Math 17.2: Two frames late

Problem: The running machine slows near people only after its learned detector reports a person. Its stop is sized for a group the detector sees in 95 percent of frames. A second group, kneeling workers, is seen in 75 percent of frames. At the aisle speed, what impact speed and energy does the second group receive from a stop that would have just cleared the first?

Variables:

• \(\DetectorMissProbability =\) \(10^{-3}\) (Chosen); \(\PerFrameDetectionProbability =\) 0.95 and 0.75 (Illustrative).
• \(\CameraFramePeriod =\) 16.7 ms (Datasheet, navigation camera); \(\SpeedAtDetection =\) 1.3 m/s (Chosen); \(\CredibleBrakingDeceleration =\) 2 m/s² (Illustrative); \(\Mass =\) 368 kg (Derived, loaded).

Math: By equation 3, the first group needs 3 frames and the second 5, so \(\FiniteChange\ReliableDetectionFrameCount =\) 2, or 33.3 ms. By equation 4, \(\DetectionRangeShortfall =\) 43 mm, well inside the 0.42 m braking distance. By equation 5, \(\ImpactSpeed =\) 0.42 m/s and \(\KineticEnergy =\) 32 J.

Result: A stop designed to end short of a standing worker ends in contact with a kneeling one at 0.42 m/s.

Systems insight: A few centimeters of late detection become a collision because the stop was sized with no margin for the slower group. The result also assumes the detector’s misses are independent from frame to frame; a detector that systematically fails on a pose misses it in many frames in a row, and the shortfall grows well beyond this best case.

Table 1 repeats the calculation for three lower-detection groups. The relationship is steep at the low end: a group seen in 60 percent of frames needs 5 extra frames and receives a 0.66 m/s impact carrying 80 J. Figure 2 plots the speed of the machine against distance for the reference group and for that group.

Table 1: A detection gap measured in meters and joules: Each row is a group of people the detector reports with the given probability per frame. The stop is sized for the reference group at the running machine’s aisle speed, loaded mass, and credible deceleration, with an allowed all-frames miss probability of one in a thousand and independent frames. Groups seen less often need more frames, lose the distance shown before braking can begin, and are struck at the speed and energy shown.

Per-frame detection	Frames needed	Extra frames	Distance lost	Impact speed	Impact energy
0.95 (reference)	3	0	0 mm	0 m/s	0 J
0.85	4	1	22 mm	0.29 m/s	16 J
0.75	5	2	43 mm	0.42 m/s	32 J
0.60	8	5	108 mm	0.66 m/s	80 J

Figure 2: One stop, two groups of people: Speed of the running machine against distance traveled from the moment a person enters the detector’s view, for a stop sized so that it ends just short of a person in the best-detected group. That group is detected after 3 frames; a group seen in 60 percent of frames is detected after 8, so braking begins later and the same deceleration leaves the machine still moving when it reaches the person. The impact speed and energy follow from the distance lost, not from the recall gap alone.

17.5.2 Sample sizes for each group

A detection gap can be managed only if it can be measured, and measuring it per group takes more data than measuring it overall. Suppose an evaluation set holds 10,000 people, of whom 2 percent belong to a smaller group, 200 people, and the detector finds 0.95 of both groups. The exact 95 percent interval for the larger group runs from 0.945 to 0.954. For the smaller group it runs from 0.910 to 0.976, about 8× wider, too wide to tell a detection rate of 0.95 from one of 0.92. A pooled detection rate reported for “pedestrians” or “workers” hides exactly the group the machine serves worst.

↩︎ The exact binomial bound, and why a few observed failures drop it sharply, are established in Bounds when failures occur.

Telling a detection rate of 0.95 from one of 0.92 with 80 percent power takes about 1,059 people in each group, about 11 percent of the whole evaluation set for every group the evaluation must resolve. A group that makes up a few percent of the people a site employs will not reach that count by chance; the evaluation has to recruit it deliberately. The same arithmetic sets how long field monitoring must run before a per-group detection rate can be trusted, and it is why the sample size for each group belongs in the plan before data collection begins, not in the report after it.

17.5.3 Where the measurement lives

If per-group detection rates live only in a research benchmark report, the release gate cannot check them before admitting the machine to an open floor. Tracking them does not need a new pipeline; it needs a new dimension in records the book already keeps. The observation contract states its detection qualification for each group the machine must slow down for, the fault list adds a coverage dimension that seeds kneeling, carrying, and seated people as deliberately as it seeds stale frames, and the evaluation record reports detection per group with its confidence interval rather than only in aggregate. From those qualified measurements, the exposure block of the approval record stores, for each group the machine relies on detecting, the sample size, the rate, its interval, and the distance shortfall it implies at the operating speed. A group with no entry is an unknown premise, and the release gate treats it as one.

↩︎ The observation contract, which states what each measurement may be relied on for, is established in The Observation Contract.

Checkpoint 17.2: Detection gaps and sample sizes

• Conversion: A detector reaches a person kneeling between racks two frames later than a standing person. Can you compute, at the running machine’s aisle speed, loaded mass, and credible deceleration, the impact speed and energy if the stop was sized for standing workers?
• Remedy: Can you compare three remedies (a lower speed near racks, a sensor that does not depend on the person’s pose, and a zone rule that keeps the machine out while people kneel there) by what each costs and what evidence each needs?
• Sample: Can you size an evaluation that distinguishes 0.95 from 0.92 detection at 80 percent power, and say what share of a 10,000-person test set each group must be?

17.6 Who Bears the Risk

The people a machine puts at risk are not only the people who chose it. The facility operator chose to buy the machine and the integrator chose how to configure it; a rider chooses to take a robotaxi. The pedestrian in the crosswalk, the coworker at an aisle crossing, and the visitor walking through a loading area made no such choice. An aggregate comparison, such as a fleet’s injury rate against a human benchmark, can improve while the remaining risk concentrates on people who never agreed to carry it. The risk such people carry without having chosen it is unconsented risk, and it has to be counted separately because no aggregate figure reveals it.

↩︎ Fleet crash rates reported outcome by outcome, each with its own interval, are established in Fleet evidence and accountable release.

Choices that look purely operational move that risk around. The site crossing rule, which lets people enter the aisle only at marked points, buys the machine its speed by asking pedestrians to wait and to comply. Suppose 80 percent of the encounters between machines and people happen at three crossings used by 10 percent of the workforce. Those workers then carry 8× the site’s average exposure per person, while everyone else carries 22 percent of it, and the site-wide encounter rate looks the same either way. Routing, the boundaries of the operating domain, and the hours a machine runs all shift exposure among groups in the same way.

↩︎ The site crossing rule, and the access restriction it places on pedestrians, are established in Restricted operation and access boundaries.

The requirement that follows is a reporting one. The exposure block of the approval record lists each group exposed to the machine, whether its members chose that exposure, how their exposure is measured (encounters per person-hour, near misses, close passes), and the routing and domain decisions that changed it. A group that bears risk it did not choose is owed two things a fleet-wide statistic cannot give it: a measurement of its own exposure, and a way to see the machine coming.

17.6.1 Legible intent

People protect themselves from a moving machine by predicting where it will go and getting out of the way, but a machine that follows an efficient path until the last moment leaves nearby coworkers unable to tell whether it will continue straight or turn across their path. Dragan, Lee, and Srinivasa distinguish two properties of robot motion: motion is predictable when it matches what an observer who knows the goal expects, and legible when it lets an observer who does not know the goal infer it quickly (Dragan et al. 2013). A mobile base that drifts toward the side of the aisle it will turn into, or an arm that moves toward the shelf it will reach before it reaches, is legible; a machine that moves along the efficient path until the last moment is predictable only to someone who already knows its plan.

Dragan, Anca D., Kenton C. T. Lee, and Siddhartha S. Srinivasa. 2013. “Legibility and Predictability of Robot Motion.” 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 301–8. https://doi.org/10.1109/HRI.2013.6483603.

Lee, John D., and Katrina A. See. 2004. “Trust in Automation: Designing for Appropriate Reliance.” Human Factors: The Journal of the Human Factors and Ergonomics Society 46 (1): 50–80. https://doi.org/10.1518/hfes.46.1.50_30392.

Explicit signals add to legible motion. A light that projects the planned path on the floor, a sound before a reversing move, or a display that shows which way the machine will turn at a crossing lets a coworker begin to yield earlier, and a person who yields earlier approaches the machine more slowly, which shrinks the approach-speed term of the separation the enforcer must keep. Signals help only if people’s trust follows what the machine can actually do. Lee and See showed that people rely on automation in proportion to their trust in it, and that trust miscalibrated in either direction causes harm: too much leads to people stepping in front of a machine that will not stop for them, too little to people ignoring a machine that would (Lee and See 2004). A signal that the machine slows for people should therefore be shown only where the machine’s detection of people has been measured, group by group, to support it.

↩︎ The human separation distance, whose approach-speed term a person who yields earlier reduces, is established in Stopping Envelopes.

17.7 Data About People

When the warehouse machine logs a near miss at a rack end, its black-box camera frames capture the face of the coworker in the aisle, its authority log records the identifier of the remote assistant on shift, and its training archive holds hours of motion linked to the demonstrators who taught the policy. The records that make a machine answerable are also records about people. The same evidence that settles who answers for an incident can, if it is kept forever and read by anyone, become a tool for watching the people it was meant to protect. How those records are declared, kept, and read is therefore part of the design, not an afterthought to it.

17.7.1 Recorded people as a standing condition

Every record a release relies on also records people: the authority record of The Authority Record names the operator behind each takeover tick by tick, the event-triggered traces of the black box hold camera frames of whoever stood in the aisle (including workers who never touched the console), and the demonstration archive of The Dataset Schema binds each episode to the operator who produced it. If those traces are retained indefinitely or opened to unrestricted browsing, the safety evidence store turns into continuous workplace surveillance. A release must retain the evidence needed for incident reconstruction while keeping identifying data to a minimum and within justified retention limits. The release therefore carries one more premise, and it is administrative rather than physical: each record that identifies a person declares its purpose, its retention period, and who may read it, and the site operates within those declarations.

Where the machine runs in the European Union, the General Data Protection Regulation makes that premise law (European Parliament and Council of the European Union 2016). Article 5(1) requires personal data to be collected for specified, explicit, and legitimate purposes and not further processed in a manner incompatible with them (point b), to be limited to what those purposes need (point c), and to be kept in identifiable form no longer than they require (point e). Article 25(2) makes the minimum the default, in the amount collected, the extent of processing, the storage period, and accessibility. Article 35 requires a data protection impact assessment before processing that is likely to result in a high risk to the people concerned, and Article 88 lets member states and collective agreements set more specific rules for employee data, naming monitoring systems at the workplace among the safeguards those rules must address. Other jurisdictions differ in detail; the engineering premise does not.

Each declaration becomes a standing condition of the kind The Verdict's Standing Conditions defines, with a detector and a response. Incident reconstruction sets the retention of the black-box and authority logs: the traces around an intervention are kept for a declared, justified review period, with any extension documented under the applicable retention rules, and routine ticks with no event roll off the ring buffer as they already do. Access follows the purpose, so an incident reviewer may read a trace and an operator’s supervisor may not score from it. A trace promoted into the experience pipeline carries its purpose and retention with it, which is what the consent and retention field of Demonstration dataset provenance schema records. The detector is an access and retention audit on the evidence store; its response follows from the rule that a verdict stands only while its premises are known-true, so a record used outside its declared purpose, or held past its period, suspends that use and returns the premise to the adjudicator. Stated this way, the premise keeps the traces an investigator needs to reconstruct a stop and bars their use for watching workers.

17.7.2 Keeping less

Continuous video recording keeps every frame of every worker’s shift, creating a permanent surveillance archive when an incident investigation needs only the seconds around a fault. The first way to protect recorded people is therefore to record less of them. Event-triggered retention keeps only the window around an intervention or a boundary excursion: the 5 s before the trigger that the authority record needs to reconstruct a takeover, and the response after it, 30 s in all. At 20 such events per machine over a 16-hour operating day, the machine keeps about 1 percent of its footage, a 96× reduction, and everything else rolls off the ring buffer within seconds.

↩︎ The pre-trigger window the black box must hold to reconstruct a takeover is established in Trustworthy Operational Logs.

Two further steps reduce what the kept footage reveals. Faces and badge numbers can be blurred on the machine before a frame is stored, provided the investigation does not need them; the declaration then records which fields stay identifiable and why, for example that the operator’s identity is kept in the authority record but not in the camera trace. People whose demonstrations train the policy are a separate case: they are recorded on purpose and at length, so their consent, what their episodes may be used for, and how long those episodes are kept travel with the data from the moment of collection. The data block of the approval record lists each record class, the people it records, its purpose, its retention, what is removed before storage, who may read it, and the result of the last access and retention audit.

17.7.3 When safety records become performance scores

The records most likely to be misused are the ones that measure workers. The machine’s authority logs and black-box recorders capture the identity of every operator who initiates a manual takeover, how long each takeover took, and how often each operator intervened. If an employer uses those records to score employee performance, the records have crossed from safety into monitoring. Conformity assessment under the European Union Machinery Regulation addresses the machinery safety requirements for the declared use; it does not govern every later use of the machine’s records. One such use falls under a different law, the European Union Artificial Intelligence Act. Its Annex III lists AI systems intended to monitor and evaluate the performance and behavior of people at work among the employment uses subject to the high-risk classification rules of Article 6, and Article 26(7) requires an employer deploying a high-risk system in the workplace to inform workers’ representatives and the affected workers before putting it into use (European Parliament and Council of the European Union 2024).

The declarations of section 17.7.1 are where the line is drawn. A record declared for incident reconstruction may not be read to rank operators, and an access audit that finds such a read suspends that use and returns the premise to the adjudicator. If a site does want to evaluate workers, that is a separate system with its own declaration, its own legal classification, and its own owner, and the approval record lists it as a separate decision rather than letting it grow out of the safety records.

17.8 The Labor the Machine Moves

Automating a task does not remove the labor the task needs. It moves that labor to the people who take over a machine when its planner stalls, the technicians who sweep and inspect the corridors the machine’s restrictions depend on, the crews who keep the floor in the state the release assumed, and the operators who record demonstrations for the next policy. The intervention frequency logged during release testing defines the human work the deployment creates: how often a technician is dispatched, how fast that person must arrive, and where on the floor that person must be stationed.

↩︎ The authority record that logs each takeover, its actor, and its timing is established in The Authority Record.

Napkin Math 17.3: Where the tote's labor went

Problem: The site of notebook 17.1 runs 20 machines for 16 hours a day with one remote assistant, and sweeps the restricted corridors 4 times a day. How many person-hours of work does the fleet create each day?

Variables:

• Assistant utilization: 50 percent (Derived, notebook 17.1); operating day: 16 hours (Illustrative).
• Corridor sweeps: 4 per day of 20 min each (Illustrative).

Math: The assistant must be staffed for all 16 h hours of operation, of which 8 h go to requests and 8 h to watching and waiting. Sweeps add 1.3 h. The fleet creates 17.3 person-hours of work a day, about 2.2 full-time shifts.

Result: The fleet needs more than two people’s working days of human labor every day it runs, before counting maintenance, charging, or the collection of new demonstrations.

Systems insight: The waiting hours are labor too. A person paid to watch a console for half of every shift, ready to act within seconds, does a job that monitoring research has long shown people perform poorly (Bainbridge 1983), and the design that creates the job owns its conditions.

Acemoglu, Daron, and Pascual Restrepo. 2020. “Robots and Jobs: Evidence from US Labor Markets.” Journal of Political Economy 128 (6): 2188–244. https://doi.org/10.1086/705716.

Burtch, Gordon, Brad Greenwood, and Kiron Ravindran. 2025. “Lucy and the Chocolate Factory: Warehouse Robotics and Worker Safety.” ILR Review 78 (4): 587–613. https://doi.org/10.1177/00197939251333754.

Gihleb, Rania, Osea Giuntella, Luca Stella, and Tianyi Wang. 2022. “Industrial Robots, Workers’ Safety, and Health.” Labour Economics 78: 102205. https://doi.org/10.1016/j.labeco.2022.102205.

The burden is not only hours. Across United States local labor markets, greater exposure to industrial robots between 1990 and 2007 lowered both employment and wages (Acemoglu and Restrepo 2020). Inside warehouses, the effect on safety cuts both ways. Burtch, Greenwood, and Ravindran found warehouse robotics associated with a 40 percent decrease in severe injuries and a 77 percent increase in non-severe ones, an increase they attribute at least partly to the faster pace of work at robotic facilities (Burtch et al. 2025). Across United States commuting zones, Gihleb and colleagues found that greater exposure to industrial robots lowered work-related injury rates while raising other harms, including drug- and alcohol-related deaths (Gihleb et al. 2022). A deployment can make the most dangerous injuries rarer and the work around the machine harder at the same time. Whether mobile manipulators in logistics follow either pattern has not yet been measured at scale. The task, takeover, and technician-hour records built in the book can inform that measurement; estimating causal effects on workers also requires comparison sites and labor-market data that no single deployment holds.

The labor block of the approval record therefore lists the work a deployment moves: takeover hours, technician dispatches, sweep hours, staffed monitoring hours, and the pace targets set for workers who share the floor. It also states the uses of worker data that the deployment bars, so that the labor the machine creates is not measured with records collected for safety. A site that cannot fill these fields does not know what its machine costs the people who work beside it.

↪ The cost of an operating restriction in throughput, against the cost of the test campaign that would lift it, is developed in What a restriction costs.

17.9 The Approval Record

Each section so far has uncovered decisions that can fail or shift risk onto people without leaving a trace in the book’s runtime logs: who approved each link behind an admitted action, which human roles the design relies on and whether a person can perform them, which permission stage the machine holds and what it must report, which groups bear its risk and how well it detects them, what it records about people, and what labor it moves. Left scattered across emails and meeting notes, any one of those decisions can lapse without blocking release. The approval record collects them in a single gate-checked artifact: it binds by digest the authority record, the enforcement record, the fault records, and the release manifest, and it adds the decisions behind them, each with a named owner.

Definition 17.3: Approval record

The approval record is the signed record that names, for each decision behind a machine’s admitted actions, the person who approved it and answers for it, and that declares the human roles the design relies on, the permission stage and its reporting duties, the groups exposed to the machine, the records kept about people, and the labor the deployment moves.

1. Significance: The release gate reads the approval record as a standing condition, so a decision with no named owner refuses operation exactly as an unknown physical premise does.
2. Distinction: The authority record shows who held each channel at each tick; the approval record shows who decided that the machine could be in that situation at all.
3. Common pitfall: A frequent misconception is that a complete runtime log makes someone answerable. A log reconstructs events; only a record of decisions and their owners says who answers for them.

The approval record has seven blocks, a header that binds it to the release and one block for each kind of decision this chapter has set out (table 2). Each block names the decision owner alongside the specific evidence, bounds, or reporting duties that keep the release valid.

Table 2: The approval record: Each block holds a kind of decision the runtime records cannot hold, and each entry names an owner. The header binds the record to the release it governs by digest, so a new release requires a new approval record.

Block	What it holds	Set in
Header	Record identifier; digests of the release manifest, authority record, enforcement record, and fault records; the permission stage; the validity window	section 17.9
Decisions	For each decision behind an admitted action: its type (envelope sign-off, configuration approval, update approval, staffing, protective-function change, permit application, data use), owner, signed approval, evidence, date, and expiry	section 17.2
Human roles	For each role the safety argument cites: the duty, the response time it requires, the time the hazard leaves, the utilization bound, the information and authority channels, and a verdict of FEASIBLE, INFEASIBLE, or UNKNOWN	section 17.3
Permit and reporting	For each stage: the grantor, the configuration digest, the operating domain, the reporting duties and deadlines, and the disclosure log	section 17.4
Exposure	For each group exposed: whether it chose the exposure, how its exposure is measured, its per-group detection (sample size, rate, interval, distance shortfall), and the routing and domain choices that change it	section 17.5, section 17.6
Records about people	For each record class: the people recorded, the purpose, the retention, what is removed before storage, the readers, and the last audit result	section 17.7
Labor	Takeover hours, technician dispatches, sweep hours, staffed monitoring hours, pace targets, and the uses of worker data the deployment bars	section 17.8

↪ Byte layout: The approval record’s fields, their encodings, and the check the release gate applies to each are given in Approval record.

The approval record gates release. The release gate reads it as a standing condition of the manifest, and four findings refuse operation the way an unknown premise does: a decision behind an admitted action class with no named owner, a human role cited as a countermeasure whose verdict is INFEASIBLE or UNKNOWN, an exposed group the machine relies on detecting with no per-group entry, and a declaration about recorded people that has expired or failed its audit. Each returns the premise to the adjudicator with its cause logged. A decision whose owner leaves the organization expires with that person’s role, so the approval record is kept current the same way the standing conditions are: by checking, on a declared schedule, that every premise still holds.

↩︎ Standing conditions, each with a detector, a cadence, and a response that withdraws the grant, are established in The Verdict's Standing Conditions.

Not every open entry can be closed before release. A decision may wait on a regulator, a per-group sample may still be filling, a staffing plan may depend on a hire. Those entries do not vanish into a footnote. Each one leaves the approval record with an operating restriction that holds until it closes, such as a lower speed near the group not yet measured or a zone the machine may not enter, and enters the residual-claims register with an owner and a closure test, alongside the physical premises no sensor can check.

↪ The residual-claims register, which carries each open premise with its restriction, owner, and closure test, is developed in A residual-claims register.

Checkpoint 17.3: An approval record for the Cruise configuration

• Decisions: Using section 17.1, can you list the decisions behind the October 2023 pullover (the post-collision response rule, the operating domain, the commercial stage, the reporting duties) and name the role that should own each?
• Unknown fields: Which of those fields could have been marked UNKNOWN before the collision, and which would have refused operation under the gate rule of this section?
• Disclosure: Can you write the disclosure-log entry that the permit block requires for the first report after the collision: who sends it, to whom, by when, and with what evidence attached?

17.10 Fallacies and Pitfalls

Responsibility for a machine fails in predictable ways, and most of them come from treating it as something settled after an incident rather than designed before one. Each misconception below assumes that a record, a person, or an average carries a responsibility that no one has actually accepted.

Fallacy: A complete runtime log makes someone answerable for what the machine did.

The logs of the Cruise vehicle were complete enough for engineers to rebuild the pullover in detail, and still the question of who had decided the vehicle should pull over, and who decided what regulators would hear, took months and an outside investigation to answer. A log records events. The decisions that made an event possible (the operating envelope, the response rule, the staffing ratio, the permit application) were made earlier, by people who appear in no tick. Six such links, each retrievable with probability 0.90, form a chain an investigator can follow to the end only 53 percent of the time.

Reconstruction is not attribution: only a record of decisions and their owners says who answers for an admitted action.

Pitfall: Naming a person as the countermeasure for a hazard faster than a person can act.

A design that lists the remote assistant as the response to a person stepping out at a rack end asks for a takeover that needs about 4.5 s when the hazard leaves 0.85 s. The person in that role adds no protection and absorbs the blame for an outcome the design controlled. Utilization makes it worse: at 20 machines per assistant, a request waits 180 s on average before anyone looks at it.

Architect’s countermeasure: Test every human role the safety argument cites for time, information, and authority, mark it FEASIBLE, INFEASIBLE, or UNKNOWN, and let only feasible roles count as countermeasures.

Fallacy: One detection rate for “people” describes how safe the machine is for everyone.

A pooled rate is dominated by the largest group. At 200 people, a smaller group’s exact interval is about 8× wider than the rest of the set’s, too wide to resolve a gap of three points in detection. A group detected two frames later at the aisle speed loses 43 mm of clearance and is struck at 0.42 m/s.

Detection must be reported per group, with a sample large enough for its interval to resolve the gap that matters, and converted into meters and joules.

Pitfall: Justifying a deployment for everyone exposed with a favorable fleet-wide comparison.

An aggregate injury rate can fall while risk concentrates on the people who never chose the machine. When 80 percent of encounters happen at crossings used by 10 percent of workers, those workers carry 8× the average exposure, and the site-wide rate does not show it.

Architect’s countermeasure: Report exposure per group and per location, record whether each group chose it, and treat routing and operating-domain choices as decisions about who bears the risk.

Fallacy: Keeping every recording is the safe choice, because an investigation might need it.

Continuous recording keeps every frame of every worker for as long as the storage lasts. Event-triggered retention keeps the windows an investigation actually uses, about 1 percent of the day, and lets the rest roll off within seconds. Records kept without a declared purpose and period drift into uses no one approved, such as ranking operators by their takeovers.

Each record about people is kept only for a declared purpose and period, and read only by the readers that purpose names.

Pitfall: Carrying a permission stage across a configuration change.

A stage authorizes a configuration in a domain under reporting duties. A new policy, a new enforcer build, or an expanded operating domain is a different configuration, and the failure-free hours that supported the old stage, about 2,996 for the claim in section 17.4.1, do not transfer to it. A withheld or incomplete incident report breaks the stage as surely as the motion does.

Architect’s countermeasure: Bind each stage to a configuration digest in the permit block of the approval record, so an update reopens the stage along with the release claims it touches.

Fallacy: Automation removes the work it automates.

A fleet of 20 machines with one remote assistant creates about 17.3 person-hours of rescue, sweeping, and monitoring every operating day, and warehouse robotics has been associated with more minor injuries even as severe ones fell. The work moves to fewer, different people, often under a faster pace.

A deployment accounts for the labor it moves and bars the use of its safety records to measure the people who do it.

17.11 Summary

A machine’s records can reconstruct every admitted action and still leave open who decided that the machine could be in that situation at all. The decisions that set a machine’s limits (the operating envelope, the response rules, the staffing ratio, the permission stage, the uses of its data) are made by different people, before any command crosses the causal boundary, and they appear in no runtime log. Each party can truthfully point to another. The Cruise collision showed the consequence: a record good enough to rebuild the motion, an account to regulators that left out the part that mattered, and penalties imposed for the record rather than the motion. Responsibility therefore has to be designed like any other premise of the machine’s authority, written into a record before the machine moves and checked when it is released.

The same discipline applies to the people around the machine. A human role counts as a countermeasure only if its holder has the time, information, and authority to act; a remote assistant whose takeover needs several times the time the hazard leaves, or whose queue makes requests wait minutes, adds blame rather than protection. Permission to operate is granted in stages, each for a configuration, a domain, and reporting duties, and is withdrawn for a withheld record as well as for unsafe motion. A gap in detection between groups of people becomes a gap in stopping distance and impact energy, and it can be managed only with per-group samples large enough to resolve it. Risk falls on people who did not choose the machine, records about people must be declared and kept no longer than their purpose needs, and the labor a machine moves must be counted. The approval record collects all of these with named owners, and the release gate refuses operation while any of them is missing.

Table 3 collects the chapter’s equations.

Table 3: Key equations of responsibility for physical AI: Each equation, what it bounds or decides about a person’s role or a group’s exposure, and where it is derived.

Equation	What it bounds or decides	Derived in
\(\ServingUtilization = \ArrivalRate\,\IntervalMean\)	The share of time one remote assistant is busy, which caps how many machines that person can serve	equation 1
\(\QueueWaitTime = \ServingUtilization\,\IntervalMean/(1-\ServingUtilization)\)	The mean wait before a help request receives a person’s attention, which no hazard faster than minutes can rely on	equation 2
\(\ReliableDetectionFrameCount = \left\lceil \ln \DetectorMissProbability / \ln(1-\PerFrameDetectionProbability) \right\rceil\)	The frames a person must stay in view before the design may count on detecting them, for one group	equation 3
\(\DetectionRangeShortfall = \SpeedAtDetection\,\CameraFramePeriod\,\FiniteChange\ReliableDetectionFrameCount\)	The clearance a less-detected group loses before braking begins	equation 4
\(\ImpactSpeed = \sqrt{2\,\CredibleBrakingDeceleration\,\DetectionRangeShortfall},\ \KineticEnergy = \Mass\,\CredibleBrakingDeceleration\,\DetectionRangeShortfall\)	The impact speed and energy a stop sized for the best-detected group delivers to a less-detected one	equation 5

Key Takeaways: Responsibility for a machine that acts

• Reconstruction is not attribution: A log shows what was admitted. Only a record of decisions, each with a named owner, shows who made the admission possible, and those decisions must be recorded on purpose because the machine does not make them.
• A role no person can perform assigns blame, not protection: Test every human role the safety argument cites for time, information, and authority, including the queue one person serves, and let only feasible roles count as countermeasures.
• Permission is staged and can be withdrawn: Each stage covers a configuration, a domain, and reporting duties. It is left on evidence gathered at that stage and withdrawn for a configuration change, an incident, or a withheld record.
• Measure fairness in meters and joules: Convert per-group detection into the distance lost before braking and the impact it causes, and size each group’s sample so its interval can resolve the gap.
• Count the people who did not choose the machine: Report exposure per group and per location, and give the people who bear the risk legible motion and signals backed by measured detection.
• Records and labor about people carry their own premises: Declare each record’s purpose, retention, and readers, and account for the work a deployment moves onto people and the uses of their data it bars.
• A missing owner refuses like an unknown premise: The approval record is a standing condition of release; entries that cannot close go to the residual-claims register with a restriction and an owner.

The fourth law, permission must be earned (principle 4), lets a machine operate only as far as its evidence supports, and the earlier chapters of this part applied it to channels, faults, and claims. Responsibility applies it to the decisions behind them: an envelope is supported only if someone accepted it, a human safeguard only if a person can perform it, and a detection claim only if it holds for each group the machine meets. The approval record makes those conditions checkable by the same gate that checks friction and timing, so the release that admits the running machine to an open floor also names who answers for each of its decisions. No assignment of responsibility after a collision returns the energy the machine has already delivered across the causal boundary, which is why every one of these decisions has an owner before the first command is admitted.

17.12 Historical Perspective and Further Reading

The question of who answers for an outcome that many people produced is older than computing. Thompson (1980) named the problem of many hands in public administration, where officials who each contribute a small part to a harmful policy can each disclaim it. Nissenbaum (1996) carried the problem into computing and identified the barriers specific to software: the many hands that build it, the treatment of failures as unavoidable bugs, the use of the computer as a scapegoat, and the ownership of software without liability for it. Matthias (2004) argued that machines which learn make the gap wider, because their behavior is not fully determined by any rule their makers wrote. The field’s founding case remains Therac-25, the radiation therapy machine whose software removed the hardware interlocks that had protected patients (Leveson and Turner 1993).

Matthias, Andreas. 2004. “The Responsibility Gap: Ascribing Responsibility for the Actions of Learning Automata.” Ethics and Information Technology 6 (3): 175–83. https://doi.org/10.1007/s10676-004-3422-1.

Leveson, Nancy G., and Clark S. Turner. 1993. “An Investigation of the Therac-25 Accidents.” Computer 26 (7): 18–41.

Sheridan, Thomas B. 1992. Telerobotics, Automation, and Human Supervisory Control. MIT Press.

Bainbridge, Lisanne. 1983. “Ironies of Automation.” Automatica 19 (6): 775–79. https://doi.org/10.1016/0005-1098(83)90046-8.

A second line of work asked when a person placed in the loop actually controls the outcome. Sheridan (1992) laid out supervisory control, in which a person sets goals and monitors while the machine acts, and Bainbridge (1983) showed why the person left to handle only the rare emergency is the one least prepared to handle it. Santoni de Sio and Hoven (2018) turned the debate over autonomous weapons into general conditions for meaningful human control, and Elish (2019) named the moral crumple zone after tracing how blame was assigned in accidents from aviation to nuclear power.

A third line measured who the machine serves well. Buolamwini and Gebru (2018) showed that the benchmarks used to evaluate face analysis left out the people the systems failed on, Wilson et al. (2019) and Li et al. (2025) found the same pattern in pedestrian detection, and Nissenbaum (2009) recast privacy as the question of whether information flows in the way its context expects, which is the question a declaration about recorded people answers. Regulation followed the same arc, from the General Data Protection Regulation (European Parliament and Council of the European Union 2016) to the European Union Artificial Intelligence Act (European Parliament and Council of the European Union 2024) and the state and federal reporting duties for driverless vehicles (California Department of Motor Vehicles 2018a; National Highway Traffic Safety Administration 2021).

Buolamwini, Joy, and Timnit Gebru. 2018. “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification.” Conference on Fairness, Accountability and Transparency, 77–91.

Li, Xinyue, Zhenpeng Chen, Jie M. Zhang, Federica Sarro, Ying Zhang, and Xuanzhe Liu. 2025. “Bias Behind the Wheel: Fairness Testing of Autonomous Driving Systems.” ACM Transactions on Software Engineering and Methodology 34 (3): 1–24. https://doi.org/10.1145/3702989.

European Parliament, and Council of the European Union. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016. Official Journal of the European Union.

European Parliament and Council of the European Union. 2024. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 Laying down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act). Official Journal of the European Union, L series, 2024/1689, 12.7.2024.

California Department of Motor Vehicles. 2018a. California Code of Regulations, Title 13, Article 3.7: Testing of Autonomous Vehicles.

National Highway Traffic Safety Administration. 2021. Standing General Order 2021-01: Incident Reporting Requirements for Automated Driving Systems and Level 2 Advanced Driver Assistance Systems.

Six works reward careful study:

• Thompson (1980) defines the problem of many hands and the forms of responsibility that survive it, the starting point for recording decisions with their owners.
• Nissenbaum (1996) names the four barriers to answering for computer systems, each of which reappears in a physical machine’s chain of decisions.
• Elish (2019) introduces the moral crumple zone and shows, case by case, how blame settles on the person nearest the event.
• Santoni de Sio and Hoven (2018) states the conditions under which human control of an autonomous system is real rather than nominal.
• Wilson et al. (2019) measures how pedestrian detection differs across skin types, the evidence behind measuring detection per group.
• Nissenbaum (2009) explains privacy as appropriate flow, the frame for declaring what each record about people may be used for.

Thompson, Dennis F. 1980. “Moral Responsibility of Public Officials: The Problem of Many Hands.” American Political Science Review 74 (4): 905–16. https://doi.org/10.2307/1954312.

Nissenbaum, Helen. 1996. “Accountability in a Computerized Society.” Science and Engineering Ethics 2 (1): 25–42. https://doi.org/10.1007/BF02639315.

Elish, Madeleine Clare. 2019. “Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction.” Engaging Science, Technology, and Society 5: 40–60. https://doi.org/10.17351/ests2019.260.

Santoni de Sio, Filippo, and Jeroen van den Hoven. 2018. “Meaningful Human Control over Autonomous Systems: A Philosophical Account.” Frontiers in Robotics and AI 5. https://doi.org/10.3389/frobt.2018.00015.

Wilson, Benjamin, Judy Hoffman, and Jamie Morgenstern. 2019. “Predictive Inequity in Object Detection.” arXiv Preprint arXiv:1902.11097.

Nissenbaum, Helen. 2009. Privacy in Context. Stanford University Press. https://doi.org/10.1515/9780804772891.

What’s Next: From responsibility to engineering judgment

What do the release verdict and its approval record say about the question this book opened with? On the book’s own records the warehouse mobile manipulator is refused, the stipulated branch earns it only a restricted mission, and the approval record adds entries that cannot close before release: a per-group detection sample still filling, a staffing plan that depends on a hire, a crossing rule no sensor checks. Engineering Physical AI reads the verdict as the answer to why a better prediction can still produce a worse physical action, transfers the same four-law method to an unfamiliar machine, and gives each premise left open, physical or human, an owner, a restriction, and a closure test in the residual-claims register.