ArmComparisonScenario

Statistics scenarios · Implementation: mlsysim/engine/scenario_definitions/statistics.py.

Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.

Two policies run arm_runs times each, with successes_a and successes_b successes; Fisher’s exact test, and the runs per arm that separate p_high from p_low two-sided at alpha with power.

Notes

Equation: Observed tables use Fisher’s exact test; planning uses the arcsine effect size and normal-quantile approximation.

Assumptions: Independent equal-sized arms and fixed planned rates for sample-size calculations.

Excludes: Paired-outcome analysis, sequential peeking, and a causal conclusion from a p-value alone.

References

Derivation: Observed tables use Fisher’s exact test; planning uses the arcsine effect size and normal-quantile approximation. Individual property contracts identify the exact accounting expression or reusable calculation. Registry entries supply the scenario inputs; this class does not turn them into measured outcomes.

Examples

Evaluate a named scenario using its declared inputs.

>>> from mlsysim import Scenarios
>>> scenario = Scenarios.LatchCheckpointComparison
>>> round(float(scenario.fisher_p), 4)
0.106

Attributes

Name	Description
all_in_one_arm	Conditional probability that all observed failures fall in one designated arm under
alpha
arm_runs
cohens_h	Absolute arcsine-transformed effect between the planned high and low success rates.
failures	Combined observed failure count across both arms.
fisher_p	Two-sided Fisher exact p-value for the two equal-size policy arms’ success/failure
p_high
p_low
power
runs_per_arm	Approximate sample size per arm for the configured planned effect.
successes_a
successes_b
total_runs	Total trial count across the two equal-size arms.
z_alpha	Standard-normal quantile for the two-sided significance level.
z_power	Standard-normal quantile for the target detection power.

Methods

Name	Description
runs_per_arm_for	Approximate equal-arm sample size for `effect_h` under the stated significance and

runs_per_arm_for

engine.scenario_types.ArmComparisonScenario.runs_per_arm_for(effect_h)

Approximate equal-arm sample size for effect_h under the stated significance and power.

Equation: calc_arcsine_two_arm_sample_size(effect_h, alpha, power).