AgreementDriftScenario
Statistics scenarios · Implementation: mlsysim/engine/scenario_definitions/statistics.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
A labeling campaign’s kappa at a baseline and again after elapsed.
Notes
Equation: kappa drop = baseline kappa - later kappa.
Assumptions: The two kappa values concern comparable tasks and label definitions.
Excludes: A fitted temporal decay rate or attribution of the change to a particular cause.
References
Derivation: kappa drop = baseline kappa - later kappa. The individual properties state units and intermediate accounting; this class introduces no fitted coefficient.
Examples
The named scenario supplies illustrative inputs; the output is calculated.
>>> from mlsysim import Scenarios
>>> case = Scenarios.WakeWordLabelQualityDrift
>>> round(float(case.drop), 3)
0.13Attributes
| Name | Description |
|---|---|
| baseline_kappa | |
| drop | Baseline minus later Cohen’s kappa, a signed dimensionless change; elapsed time |
| elapsed | |
| later_kappa |
baseline_kappa
Type: float.
Declared default: required.
later_kappa
Type: float.
Declared default: required.
elapsed
Type: Quantity.
Declared default: required.
drop
engine.scenario_types.AgreementDriftScenario.drop
Baseline minus later Cohen’s kappa, a signed dimensionless change; elapsed time does not scale it.
Equation: return baseline_kappa - later_kappa.