AnnotatorAgreementScenario
Statistics scenarios · Implementation: mlsysim/engine/scenario_definitions/statistics.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Two independent annotators with the same label mix class_shares: chance agreement, and Cohen’s kappa at an observed_agreement.
Notes
Equation: chance agreement = sum(class share squared); kappa = (agreement - chance) / (1 - chance).
Assumptions: Both annotators have the same class marginal distribution; observed agreement is supplied unless the independent-raters case is selected.
Excludes: Estimating reliability intervals, annotator-specific marginals, and task correctness.
References
Derivation: chance agreement = sum(class share squared); kappa = (agreement - chance) / (1 - chance). The individual properties state units and intermediate accounting; this class introduces no fitted coefficient.
Examples
The named scenario supplies illustrative inputs; the output is calculated.
>>> from mlsysim import Scenarios
>>> case = Scenarios.BinaryLabelChanceAgreement
>>> round(float(case.chance_agreement), 3)
0.82Attributes
| Name | Description |
|---|---|
| agreement | Supplied observed agreement, or chance agreement when the independent-raters flag |
| chance_agreement | Chance-agreement fraction from physics.statistics.calc_chance_agreement and |
| class_shares | |
| independent_raters | |
| kappa | Chance-corrected agreement from physics.statistics.calc_cohens_kappa; |
| observed_agreement |
observed_agreement
Type: Optional[float].
Declared default: None.
independent_raters
Type: bool.
Declared default: False.
chance_agreement
engine.scenario_types.AnnotatorAgreementScenario.chance_agreement
Chance-agreement fraction from physics.statistics.calc_chance_agreement and equal class marginals.
Equation: return calc_chance_agreement(class_shares).
agreement
engine.scenario_types.AnnotatorAgreementScenario.agreement
Supplied observed agreement, or chance agreement when the independent-raters flag is set.
Equation: return chance_agreement if independent_raters else observed_agreement.
kappa
engine.scenario_types.AnnotatorAgreementScenario.kappa
Chance-corrected agreement from physics.statistics.calc_cohens_kappa; dimensionless.
Equation: return calc_cohens_kappa(agreement, chance_agreement).