ClosedLoopCampaignScenario

Statistics scenarios · Implementation: mlsysim/engine/scenario_definitions/statistics.py.

Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.

clean_runs failure-free closed-loop trials of a frozen policy in one cell of table on site, each run long with an automated reset, bounded at one-sided confidence.

What a success target 1 - target_rate costs in runs, cell time, and operator time; the exact bound when recorded_failures runs fail; and the contrasts the bound is taught with: a policy that fails one run in weak_policy_one_in, fallacy_runs clean runs against the target, looks independent looks at the significance level, and a Bonferroni split of the significance level over joint_cells cells. A run that jams costs jam_reset instead of reset.

Notes

Equation: Binomial confidence helpers convert failure-free counts to bounds and invert them; cycle accounting adds run and reset time.

Assumptions: Independent identically distributed trials of a frozen policy in a fixed evaluation cell; planned confidence is one-sided.

Excludes: Distribution shift, adaptive test reuse, dependence between trials, and safety certification.

References

Derivation: Binomial confidence helpers convert failure-free counts to bounds and invert them; cycle accounting adds run and reset time. Individual property contracts identify the exact accounting expression or reusable calculation. Registry entries supply the scenario inputs; this class does not turn them into measured outcomes.

Examples

Evaluate a named scenario using its declared inputs.

>>> from mlsysim import Scenarios
>>> scenario = Scenarios.LatchEvaluationCampaign
>>> round(float(scenario.alpha), 4)
0.05

Attributes

Name	Description
alpha	One-sided confidence tail probability, equal to one minus confidence.
any_false_positive	Probability of at least one false positive across independent looks at the configured
clean_runs
confidence
cycle	Seconds per trial including the normal reset time.
estimate_rule_of_three_runs	Approximate trial requirement for the estimation target from the rule of three.
estimate_runs	Failure-free trial count for the separate estimation target under the configured
estimate_target	Success probability complementary to the separate estimation target failure rate.
estimate_target_rate
failure_upper	Complement of the success lower bound, expressed as a failure probability.
fallacy_lower_bound	Success lower bound from the smaller illustrative `fallacy_runs` campaign.
fallacy_runs
jam_cycle	Seconds per trial when the longer jam-reset time replaces the normal reset.
jam_reset
jam_reset_share	Fraction of a jammed trial cycle spent resetting rather than running.
joint_cells
joint_runs	Failure-free trials per cell with Bonferroni allocation of the confidence tail across
ln_inv_alpha	Dimensionless negative logarithm of the confidence tail probability.
looks
lower_bound	Exact success-probability lower bound after the configured failure-free run count.
lower_bound_with_failures	Exact success lower bound after subtracting recorded failures from the configured total
nine_targets	Success probabilities with successive decimal nines up to the configured target rate’s
observed_rate_with_failures	Observed success fraction after recorded failures, without a confidence adjustment.
omissions
operators
recorded_failures
reset
rule_of_three	Approximate failure-probability ceiling after failure-free runs, using
run
shift
site
standoff
table
target	Success-probability target complementary to the specified target failure rate.
target_rate
target_runs	Failure-free trials required to demonstrate the configured success target.
weak_policy_clean_probability	Probability the weaker policy passes the full campaign without a failure.
weak_policy_one_in
weak_policy_success	Per-run success probability for a policy failing once per `weak_policy_one_in` trials

Methods

Name	Description
campaign_time	Total seconds for `trials` independent serial cycles, defaulting to the configured
clean_probability	Probability that all configured runs succeed at per-run success probability `success`,
lower_bound_at	Exact zero-failure success-probability lower bound for `trials` at the specified or
operator_time	Operator-hours for the serial campaign with the declared concurrent staffing count.
runs_for	Independent failure-free trial count required for target success probability `target`
shifts	Dimensionless staffed shift equivalents required for the campaign’s operator-hours.

campaign_time

engine.scenario_types.ClosedLoopCampaignScenario.campaign_time(trials=None)

Total seconds for trials independent serial cycles, defaulting to the configured campaign count.

Equation: ((clean_runs if trials is None else trials) * cycle).to('s').

clean_probability

engine.scenario_types.ClosedLoopCampaignScenario.clean_probability(success)

Probability that all configured runs succeed at per-run success probability success, assuming independence.

Chance that a policy of per-run success success logs the campaign clean.

Equation: success ** clean_runs.

lower_bound_at

engine.scenario_types.ClosedLoopCampaignScenario.lower_bound_at(
    trials,
    confidence=None,
)

Exact zero-failure success-probability lower bound for trials at the specified or configured confidence.

Equation: calc_clopper_pearson_zero_failure_bound(trials, confidence if confidence is None else confidence).

operator_time

engine.scenario_types.ClosedLoopCampaignScenario.operator_time(trials)

Operator-hours for the serial campaign with the declared concurrent staffing count.

Equation: (operators * campaign_time(trials)).to('hour').

runs_for

engine.scenario_types.ClosedLoopCampaignScenario.runs_for(
    target,
    confidence=None,
)

Independent failure-free trial count required for target success probability target at the selected confidence.

Equation: calc_zero_failure_sample_size(target, confidence if confidence is None else confidence).

shifts

engine.scenario_types.ClosedLoopCampaignScenario.shifts(trials)

ln_inv_alpha

engine.scenario_types.ClosedLoopCampaignScenario.ln_inv_alpha

Dimensionless negative logarithm of the confidence tail probability.

The numerator of every zero-failure count, about 3 at 95 percent.

Equation: -_math.log(alpha).

lower_bound

engine.scenario_types.ClosedLoopCampaignScenario.lower_bound

Exact success-probability lower bound after the configured failure-free run count.

Equation: lower_bound_at(clean_runs).

failure_upper

engine.scenario_types.ClosedLoopCampaignScenario.failure_upper

Complement of the success lower bound, expressed as a failure probability.

Equation: 1.0 - lower_bound.

rule_of_three

engine.scenario_types.ClosedLoopCampaignScenario.rule_of_three

Approximate failure-probability ceiling after failure-free runs, using calc_rule_of_three_bound.

Equation: calc_rule_of_three_bound(clean_runs).

target

engine.scenario_types.ClosedLoopCampaignScenario.target

Success-probability target complementary to the specified target failure rate.

Equation: 1.0 - target_rate.

nine_targets

engine.scenario_types.ClosedLoopCampaignScenario.nine_targets

Success probabilities with successive decimal nines up to the configured target rate’s order.

Success targets with one, two, … nines, up to the campaign target (0.9, 0.99, 0.999).

Equation: nines = round(-_math.log10(target_rate)); result = [1.0 - 10.0 ** (-k) for k in range(1, nines + 1)].

target_runs

engine.scenario_types.ClosedLoopCampaignScenario.target_runs

Failure-free trials required to demonstrate the configured success target.

Equation: runs_for(target).

joint_runs

engine.scenario_types.ClosedLoopCampaignScenario.joint_runs

Failure-free trials per cell with Bonferroni allocation of the confidence tail across joint_cells.

Runs per cell when the significance level is split evenly over joint_cells.

Equation: runs_for(target, 1.0 - alpha / joint_cells).

estimate_target

engine.scenario_types.ClosedLoopCampaignScenario.estimate_target

Success probability complementary to the separate estimation target failure rate.

Equation: 1.0 - estimate_target_rate.

estimate_runs

engine.scenario_types.ClosedLoopCampaignScenario.estimate_runs

Failure-free trial count for the separate estimation target under the configured confidence.

Equation: runs_for(estimate_target).

estimate_rule_of_three_runs

engine.scenario_types.ClosedLoopCampaignScenario.estimate_rule_of_three_runs

Approximate trial requirement for the estimation target from the rule of three.

Equation: calc_rule_of_three_trials(estimate_target).

weak_policy_success

engine.scenario_types.ClosedLoopCampaignScenario.weak_policy_success

Per-run success probability for a policy failing once per weak_policy_one_in trials on average.

Equation: 1.0 - 1.0 / weak_policy_one_in.

weak_policy_clean_probability

engine.scenario_types.ClosedLoopCampaignScenario.weak_policy_clean_probability

Probability the weaker policy passes the full campaign without a failure.

Equation: clean_probability(weak_policy_success).

lower_bound_with_failures

engine.scenario_types.ClosedLoopCampaignScenario.lower_bound_with_failures

Exact success lower bound after subtracting recorded failures from the configured total runs.

Equation: calc_clopper_pearson_lower_bound(clean_runs - recorded_failures, clean_runs, confidence).

observed_rate_with_failures

engine.scenario_types.ClosedLoopCampaignScenario.observed_rate_with_failures

Observed success fraction after recorded failures, without a confidence adjustment.

Equation: (clean_runs - recorded_failures) / clean_runs.

fallacy_lower_bound

engine.scenario_types.ClosedLoopCampaignScenario.fallacy_lower_bound

Success lower bound from the smaller illustrative fallacy_runs campaign.

Equation: lower_bound_at(fallacy_runs).

any_false_positive

engine.scenario_types.ClosedLoopCampaignScenario.any_false_positive

Probability of at least one false positive across independent looks at the configured tail probability.

At least one false positive across looks independent looks at alpha.

Equation: calc_independent_success(alpha, looks).

cycle

engine.scenario_types.ClosedLoopCampaignScenario.cycle

Seconds per trial including the normal reset time.

Equation: (run + reset).to('s').

jam_reset_share

engine.scenario_types.ClosedLoopCampaignScenario.jam_reset_share

Fraction of a jammed trial cycle spent resetting rather than running.

Equation: (jam_reset / (run + jam_reset)).to('').magnitude.

jam_cycle

engine.scenario_types.ClosedLoopCampaignScenario.jam_cycle

Seconds per trial when the longer jam-reset time replaces the normal reset.

Equation: (run + jam_reset).to('s').