GroupConfusionScenario

Statistics scenarios · Implementation: mlsysim/engine/scenario_definitions/statistics.py.

Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.

Binary decisions for two or more groups as confusion counts {group: {tp, fp, fn, tn}}.

A positive decision is an approval (or a flag); the positive label is the outcome the decision should select. gap compares the first two groups, first minus second.

Notes

Equation: Each metric is the stated confusion-count numerator divided by its corresponding population denominator.

Assumptions: Supplied confusion counts share the same outcome definition and every requested rate has a nonzero denominator.

Excludes: Sampling uncertainty, causal fairness claims, and metrics for unspecified groups.

References

Derivation: Each metric is the stated confusion-count numerator divided by its corresponding population denominator. Individual property contracts identify the exact accounting expression or reusable calculation. Registry entries supply the scenario inputs; this class does not turn them into measured outcomes.

Examples

Evaluate a named scenario using its declared inputs.

>>> from mlsysim import Scenarios
>>> scenario = Scenarios.CreditServiceTwoGroups
>>> round(float(scenario.pooled_positive_rate), 4)
0.525

Attributes

Name	Description
average_odds_difference	Mean of the absolute group TPR and FPR gaps; this is an unsigned disparity summary.
disparate_impact	Smaller divided by larger group positive rate, a dimensionless selection-rate ratio.
first	First group label in insertion order, used as the signed-gap reference group.
groups
overall_accuracy	Correct-decision fraction pooled by decision count across all groups.
pooled_positive_rate	Predicted-positive fraction pooled by decision count across all groups.
second	Second group label in insertion order, used as the signed-gap comparison group.

Methods

Name	Description
accuracy	Correct-decision fraction in the named group.
base_rate	Actual-positive fraction among all decisions in the group.
fnr	False-negative fraction among actual positives in the group.
fpr	False-positive fraction among actual negatives in the group.
gap	Signed first-minus-second group difference for the named rate `metric`.
positive_rate	Predicted-positive fraction among all decisions in the group.
positives	Predicted-positive count in the group, summing true and false positives.
ppv	True-positive fraction among predicted positives in the group.
share	Named group’s fraction of the combined decision population.
size	Total decisions in the named group, summing all confusion-table cells.
tpr	True-positive fraction among actual positives in the group.

accuracy

engine.scenario_types.GroupConfusionScenario.accuracy(group)

Correct-decision fraction in the named group.

Share of the group decided correctly (added 2026-10-20).

Equation: c = _c(group); result = (c['tp'] + c['tn']) / size(group).

base_rate

engine.scenario_types.GroupConfusionScenario.base_rate(group)

Actual-positive fraction among all decisions in the group.

Equation: c = _c(group); result = (c['tp'] + c['fn']) / size(group).

fnr

engine.scenario_types.GroupConfusionScenario.fnr(group)

False-negative fraction among actual positives in the group.

Share of the positive-label members given a negative decision (added 2026-10-20).

Equation: c = _c(group); result = c['fn'] / (c['tp'] + c['fn']).

fpr

engine.scenario_types.GroupConfusionScenario.fpr(group)

False-positive fraction among actual negatives in the group.

Equation: c = _c(group); result = c['fp'] / (c['fp'] + c['tn']).

gap

engine.scenario_types.GroupConfusionScenario.gap(metric)

Signed first-minus-second group difference for the named rate metric.

metric (positive_rate, tpr, fpr, ppv, base_rate) of the first group minus the second’s.

Equation: metric(first group)-metric(second group); metric names select the documented rate methods.

positive_rate

engine.scenario_types.GroupConfusionScenario.positive_rate(group)

Predicted-positive fraction among all decisions in the group.

Share of the group given a positive decision (the demographic-parity quantity).

Equation: positives(group) / size(group).

positives

engine.scenario_types.GroupConfusionScenario.positives(group)

Predicted-positive count in the group, summing true and false positives.

Positive decisions (approved or flagged).

Equation: c = _c(group); result = c['tp'] + c['fp'].

ppv

engine.scenario_types.GroupConfusionScenario.ppv(group)

True-positive fraction among predicted positives in the group.

Equation: c = _c(group); result = c['tp'] / (c['tp'] + c['fp']).

engine.scenario_types.GroupConfusionScenario.share(group)

Named group’s fraction of the combined decision population.

The group’s share of all members (added 2026-10-20).

Equation: size(group) / sum((size(g) for g in groups)).

size

engine.scenario_types.GroupConfusionScenario.size(group)

Total decisions in the named group, summing all confusion-table cells.

Equation: c = _c(group); result = c['tp'] + c['fp'] + c['fn'] + c['tn'].

tpr

engine.scenario_types.GroupConfusionScenario.tpr(group)

True-positive fraction among actual positives in the group.

Equation: c = _c(group); result = c['tp'] / (c['tp'] + c['fn']).

groups

Type: Dict[str, Dict[str, float]].

Declared default: required.

first

engine.scenario_types.GroupConfusionScenario.first

First group label in insertion order, used as the signed-gap reference group.

Equation: list(groups)[0].

second

engine.scenario_types.GroupConfusionScenario.second

Second group label in insertion order, used as the signed-gap comparison group.

Equation: list(groups)[1].

pooled_positive_rate

engine.scenario_types.GroupConfusionScenario.pooled_positive_rate

Predicted-positive fraction pooled by decision count across all groups.

Share of all members, every group pooled, given a positive decision (added 2026-10-20).

Equation: sum((positives(g) for g in groups)) / sum((size(g) for g in groups)).

overall_accuracy

engine.scenario_types.GroupConfusionScenario.overall_accuracy

Correct-decision fraction pooled by decision count across all groups.

Share of all members of all groups decided correctly (added 2026-10-20).

Equation: right = sum((c['tp'] + c['tn'] for c in groups.values())); result = right / sum((size(g) for g in groups)).

disparate_impact

engine.scenario_types.GroupConfusionScenario.disparate_impact

Smaller divided by larger group positive rate, a dimensionless selection-rate ratio.

Lower positive rate over higher positive rate across the first two groups.

Equation: a, b = (positive_rate(first), positive_rate(second)); result = min(a, b) / max(a, b).

average_odds_difference

engine.scenario_types.GroupConfusionScenario.average_odds_difference

Mean of the absolute group TPR and FPR gaps; this is an unsigned disparity summary.

Mean of the absolute TPR and FPR gaps.

Equation: 0.5 * (abs(gap('tpr')) + abs(gap('fpr'))).