SameTestSetAccuracyComparisonScenario
Statistics scenarios · Implementation: mlsysim/engine/scenario_definitions/statistics.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Two models scored on the same n_samples test items: independent-set interval and McNemar’s limit.
The scores fix the net gap b - c (items only the baseline gets right minus items only the candidate gets right) but not the disagreements b + c; McNemar’s statistic (no continuity correction) exceeds the chi-square critical value z^2 only when b + c <= floor(gap^2 / z^2).
Notes
Equation: The calculation members below define z, baseline_errors, candidate_errors, baseline_error_sd, accuracy_difference, independent_interval, net_gap, chi2_critical, max_discordant, max_discordant_share.
Assumptions: Scores use the same item count; the displayed independent interval is a counterfactual comparison, while the McNemar threshold constrains discordant pairs.
Excludes: Recovering actual discordant counts from aggregate accuracies alone.
References
Derivation: Two models scored on the same n_samples test items: independent-set interval and McNemar’s limit. Each member records the corresponding algebra or named physics function.
Examples
Evaluate the named configuration and display a modeled result.
>>> from mlsysim import Scenarios
>>> case = Scenarios.NinetyFiveAgainstNinetyFourOnSameImages
>>> value = case.max_discordant
>>> print(value)
26Attributes
| Name | Description |
|---|---|
| accuracy_difference | Baseline-minus-candidate accuracy difference as a fraction, not percentage points. |
| baseline_accuracy | |
| baseline_error_sd | Binomial standard deviation of baseline error count, assuming independent items. |
| baseline_errors | Expected baseline error count from sample count times one minus accuracy. |
| candidate_accuracy | |
| candidate_errors | Expected candidate error count from sample count times one minus accuracy. |
| chi2_critical | One-degree-of-freedom chi-square threshold formed by squaring the normal critical value. |
| confidence | |
| independent_interval | Hypothetical independent-test-set interval via calc_proportion_gap_interval; not the paired-data interval. |
| max_discordant | Largest discordant count consistent with the threshold via calc_mcnemar_max_discordant. |
| max_discordant_share | Maximum discordant count divided by test-set count, dimensionless. |
| n_samples | |
| net_gap | Rounded net number of items correct only for baseline minus only for candidate. |
| z | Two-sided standard-normal critical value for the requested confidence. |
n_samples
Type: int.
Declared default: required.
baseline_accuracy
Type: float.
Declared default: required.
candidate_accuracy
Type: float.
Declared default: required.
confidence
Type: float.
Declared default: required.
z
engine.scenario_types.SameTestSetAccuracyComparisonScenario.z
Two-sided standard-normal critical value for the requested confidence.
Equation: _z_two_sided(confidence).
baseline_errors
engine.scenario_types.SameTestSetAccuracyComparisonScenario.baseline_errors
Expected baseline error count from sample count times one minus accuracy.
Equation: n_samples * (1 - baseline_accuracy).
candidate_errors
engine.scenario_types.SameTestSetAccuracyComparisonScenario.candidate_errors
Expected candidate error count from sample count times one minus accuracy.
Equation: n_samples * (1 - candidate_accuracy).
baseline_error_sd
engine.scenario_types.SameTestSetAccuracyComparisonScenario.baseline_error_sd
Binomial standard deviation of baseline error count, assuming independent items.
Binomial standard deviation of the baseline’s error count.
Equation: _math.sqrt(n_samples * p * (1 - p)).
accuracy_difference
engine.scenario_types.SameTestSetAccuracyComparisonScenario.accuracy_difference
Baseline-minus-candidate accuracy difference as a fraction, not percentage points.
Equation: baseline_accuracy - candidate_accuracy.
independent_interval
engine.scenario_types.SameTestSetAccuracyComparisonScenario.independent_interval
Hypothetical independent-test-set interval via calc_proportion_gap_interval; not the paired-data interval.
(low, high) of the accuracy difference had the scores come from independent test sets.
Calculation: mlsysim.physics.statistics.calc_proportion_gap_interval.
net_gap
engine.scenario_types.SameTestSetAccuracyComparisonScenario.net_gap
Rounded net number of items correct only for baseline minus only for candidate.
b - c: items the baseline alone gets right, net.
Equation: round(n_samples * accuracy_difference).
chi2_critical
engine.scenario_types.SameTestSetAccuracyComparisonScenario.chi2_critical
One-degree-of-freedom chi-square threshold formed by squaring the normal critical value.
Equation: z ** 2.
max_discordant
engine.scenario_types.SameTestSetAccuracyComparisonScenario.max_discordant
Largest discordant count consistent with the threshold via calc_mcnemar_max_discordant.
Calculation: mlsysim.physics.statistics.calc_mcnemar_max_discordant.