TwoProportionTestScenario

Statistics scenarios · Implementation: mlsysim/engine/scenario_definitions/statistics.py.

Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.

A two-arm test sized to detect an absolute lift on a baseline_rate at the normal quantiles z_alpha and z_beta, on daily_requests split between arms.

family_size uncorrected tests at significance give the familywise error.

Notes

Equation: The normal planning approximation uses 2*p*(1-p)*(z_alpha+z_beta)**2/lift**2 per arm.

Assumptions: Independent Bernoulli samples, equal arms, a nonzero lift, and declared normal quantiles.

Excludes: Sequential peeking, correlated samples, and multiplicity correction; familywise_error assumes independent tests.

References

Derivation: Delegates sample planning to mlsysim.physics.statistics.calc_two_proportion_sample_size and detectable lift to calc_two_proportion_detectable_lift. Their equal-arm normal approximation combines two Bernoulli variances with significance/power quantiles. The familywise error uses the complement of all independent tests avoiding a false rejection.

Examples

>>> from mlsysim import Scenarios
>>> s = Scenarios.CtrLiftExperiment
>>> s.total == 2*s.per_variant
True

Attributes

Name	Description
available_per_variant
baseline_rate
collection_time	Hours to collect the combined sample target at the supplied daily request rate.
confidence
daily_requests
family_size
familywise_error	Probability of at least one false rejection under independent tests with the supplied significance.
lift
lift_squared	Squared absolute probability difference used for sample planning.
per_variant	Continuous sample count per arm from the two-proportion normal approximation.
power
relative_lift
significance
target_rate	Alternative response probability formed by adding absolute lift to baseline.
total	Combined continuous sample count across the two equally sized arms.
two_variance	Sum of two Bernoulli variances at the common planning baseline.
z_alpha
z_beta
z_sum_squared	Squared sum of the supplied normal quantiles for significance and power.

Methods

Name	Description
detectable_lift	Smallest absolute lift `n_per_variant` users per arm can detect (default:

detectable_lift

engine.scenario_types.TwoProportionTestScenario.detectable_lift(
    n_per_variant=None,
)

engine.scenario_types.TwoProportionTestScenario.z_sum_squared

Squared sum of the supplied normal quantiles for significance and power.

Equation: (z_alpha + z_beta) ** 2.

two_variance

engine.scenario_types.TwoProportionTestScenario.two_variance

Sum of two Bernoulli variances at the common planning baseline.

Equation: 2 * baseline_rate * (1 - baseline_rate).

lift_squared

engine.scenario_types.TwoProportionTestScenario.lift_squared

Squared absolute probability difference used for sample planning.

Equation: lift ** 2.

per_variant

engine.scenario_types.TwoProportionTestScenario.per_variant

Continuous sample count per arm from the two-proportion normal approximation.

Equation: calc_two_proportion_sample_size(baseline_rate, lift, z_alpha, z_beta).

total

engine.scenario_types.TwoProportionTestScenario.total

Combined continuous sample count across the two equally sized arms.

Equation: 2 * per_variant.

target_rate

engine.scenario_types.TwoProportionTestScenario.target_rate

Alternative response probability formed by adding absolute lift to baseline.

Equation: baseline_rate + lift.

collection_time

engine.scenario_types.TwoProportionTestScenario.collection_time

Hours to collect the combined sample target at the supplied daily request rate.

Equation: (total / daily_requests).to('hour').

familywise_error

engine.scenario_types.TwoProportionTestScenario.familywise_error

Probability of at least one false rejection under independent tests with the supplied significance.

Equation: calc_independent_success(significance, family_size).