DriftConfirmationScenario

Statistics scenarios · Implementation: mlsysim/engine/scenario_definitions/statistics.py.

Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.

Labeled outcomes that confirm a drop of drop from baseline_accuracy.

evidence_budget is the policy’s count, set above the one-sample binomial floor at confidence (two-sided) and power; label_rates maps a label to the rate at which labeled outcomes arrive. compared_drops are further drop sizes to size against the same floor.

Notes

Equation: One-proportion sample planning uses normal confidence/power quantiles; collection time is the chosen evidence count divided by label rate.

Assumptions: Labels arrive at fixed named rates and outcome observations satisfy the independent binomial planning approximation.

Excludes: Sequential testing corrections, labeling delay distributions, and proof the policy evidence budget meets the computed floor.

References

Derivation: the scenario composes mlsysim.physics.statistics.calc_one_proportion_sample_size. Their contracts supply the underlying equations; property equations show the remaining unit conversions and accounting.

Examples

>>> from mlsysim import Scenarios
>>> scenario = Scenarios.LabeledOutcomeDriftConfirmation
>>> value = scenario.z_alpha
>>> round(float(value), 3)
1.96

Attributes

Name	Description
baseline_accuracy
compared_drops
confidence
drop
evidence_budget
label_rates
power
z_alpha	Two-sided normal quantile for the declared confidence, dimensionless.
z_beta	Normal quantile for the declared detection power, dimensionless.

Methods

Name	Description
collection_time	SI duration to collect the policy evidence budget at a named label arrival rate.
collection_time_for	SI duration to collect `samples` at a named positive label arrival rate.
outcomes_needed	One-proportion planning count for the supplied or default accuracy drop.

collection_time

engine.scenario_types.DriftConfirmationScenario.collection_time(rate)

SI duration to collect the policy evidence budget at a named label arrival rate.

Equation: (evidence_budget / label_rates[rate]).to_base_units().

collection_time_for

engine.scenario_types.DriftConfirmationScenario.collection_time_for(
    samples,
    rate,
)

SI duration to collect samples at a named positive label arrival rate.

Equation: (samples / label_rates[rate]).to_base_units().

outcomes_needed

engine.scenario_types.DriftConfirmationScenario.outcomes_needed(drop=None)

One-proportion planning count for the supplied or default accuracy drop.

Equation: calc_one_proportion_sample_size(baseline_accuracy, baseline_accuracy - d, z_alpha, z_beta).