SampledDecisionMonitorScenario
Statistics scenarios · Implementation: mlsysim/engine/scenario_definitions/statistics.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Per-group metric monitoring of a decision service at request_rate.
sample_share of decisions are logged at record_size each; groups subgroups x metrics_per_group metrics are each tested at false_alarm_rate per evaluation interval on a stable process.
Notes
Equation: Daily rate integration determines records and bytes; summed per-metric tail budgets give expected false alerts.
Assumptions: Constant request rate, fixed sampling share, and fixed record size.
Excludes: Bursty logging, storage replication, and a proof that all metric tests are calibrated.
References
Derivation: Daily rate integration determines records and bytes; summed per-metric tail budgets give expected false alerts. Individual property contracts identify the exact accounting expression or reusable calculation. Registry entries supply the scenario inputs; this class does not turn them into measured outcomes.
Examples
Evaluate a named scenario using its declared inputs.
>>> from mlsysim import Scenarios
>>> scenario = Scenarios.FairnessMonitor10K
>>> round(float(scenario.decisions_per_day), 4)
864000000.0Attributes
| Name | Description |
|---|---|
| bonferroni_level | Per-metric false-alarm probability after dividing the familywise budget by metric count. |
| decisions_per_day | Decision count in one day at the constant request rate. |
| false_alarm_rate | |
| false_alerts_bonferroni | Expected false-alert count summed over metrics at the corrected per-metric level. |
| false_alerts_per_interval | Expected false-alert count per interval without multiplicity correction. |
| groups | |
| log_per_day | Decimal GB of daily logs at the declared record size. |
| metrics | Total monitored metric count across groups. |
| metrics_per_group | |
| record_size | |
| records_per_day | Daily logged record count after applying the supplied sample fraction. |
| request_rate | |
| sample_share |
request_rate
Type: Quantity.
Declared default: required.
record_size
Type: Quantity.
Declared default: required.
groups
Type: int.
Declared default: required.
metrics_per_group
Type: int.
Declared default: required.
false_alarm_rate
Type: float.
Declared default: required.
decisions_per_day
engine.scenario_types.SampledDecisionMonitorScenario.decisions_per_day
Decision count in one day at the constant request rate.
Equation: (request_rate * Q_(1, 'day')).to('').magnitude.
records_per_day
engine.scenario_types.SampledDecisionMonitorScenario.records_per_day
Daily logged record count after applying the supplied sample fraction.
Equation: decisions_per_day * sample_share.
log_per_day
engine.scenario_types.SampledDecisionMonitorScenario.log_per_day
Decimal GB of daily logs at the declared record size.
Equation: (records_per_day * record_size).to('GB').
metrics
engine.scenario_types.SampledDecisionMonitorScenario.metrics
Total monitored metric count across groups.
Equation: groups * metrics_per_group.
false_alerts_per_interval
engine.scenario_types.SampledDecisionMonitorScenario.false_alerts_per_interval
Expected false-alert count per interval without multiplicity correction.
Expected false alerts per interval if nothing is wrong (no independence needed).
Equation: metrics * false_alarm_rate.
bonferroni_level
engine.scenario_types.SampledDecisionMonitorScenario.bonferroni_level
Per-metric false-alarm probability after dividing the familywise budget by metric count.
Equation: false_alarm_rate / metrics.
false_alerts_bonferroni
engine.scenario_types.SampledDecisionMonitorScenario.false_alerts_bonferroni
Expected false-alert count summed over metrics at the corrected per-metric level.
Equation: metrics * bonferroni_level.