MonitorFleetAlertScenario
Statistics scenarios · Implementation: mlsysim/engine/scenario_definitions/statistics.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
models x metrics_per_model monitors checked every check_interval at a per-check false_positive_rate on a stable process; dedup_share of crossings are merged away.
Notes
Equation: Expected crossings are models*metrics*checks*false_positive_rate; independent-event complement gives any-alert probability.
Assumptions: Per-check error probability is stable. Set dedup_share explicitly before requesting remaining_after_dedup; it is an assumed reduction fraction.
Excludes: Temporal dependence for the any-alert calculation, incident precision, and learned deduplication behavior.
References
Derivation: Linearity of expectation gives monitor_count*checks_per_monitor*per_check_error for expected crossings. A separately supplied deduplication fraction reduces that count. The any-alert probability additionally assumes independent checks and uses mlsysim.physics.agents.calc_independent_success for the complement of no crossings.
Examples
>>> from mlsysim import Scenarios
>>> s = Scenarios.ThreeSigmaModelFleetAlerts
>>> s = s.variant(dedup_share=0.5)
>>> s.remaining_after_dedup <= s.daily_false_alerts
TrueAttributes
| Name | Description |
|---|---|
| check_interval | |
| checks_per_day | Dimensionless number of checks per monitor in one day. |
| daily_false_alerts | Expected daily false crossings before deduplication. |
| dedup_share | |
| false_positive_rate | |
| metrics_per_model | |
| models | |
| monitors | Count of monitored model-metric pairs. |
| remaining_after_dedup | Expected daily false alerts after applying the assumed merged-away fraction. |
Methods
| Name | Description |
|---|---|
| any_false_alert | Chance that at least one of checks independent checks crosses falsely. |
any_false_alert
engine.scenario_types.MonitorFleetAlertScenario.any_false_alert(checks)Chance that at least one of checks independent checks crosses falsely.
Equation: calc_independent_success(false_positive_rate, checks).
models
Type: int.
Declared default: required.
metrics_per_model
Type: int.
Declared default: required.
check_interval
Type: Quantity.
Declared default: required.
false_positive_rate
Type: float.
Declared default: required.
monitors
engine.scenario_types.MonitorFleetAlertScenario.monitors
Count of monitored model-metric pairs.
Equation: models * metrics_per_model.
checks_per_day
engine.scenario_types.MonitorFleetAlertScenario.checks_per_day
Dimensionless number of checks per monitor in one day.
Equation: (Q_(1, 'day') / check_interval).to('').magnitude.
daily_false_alerts
engine.scenario_types.MonitorFleetAlertScenario.daily_false_alerts
Expected daily false crossings before deduplication.
Equation: monitors * checks_per_day * false_positive_rate.
remaining_after_dedup
engine.scenario_types.MonitorFleetAlertScenario.remaining_after_dedup
Expected daily false alerts after applying the assumed merged-away fraction.
Equation: daily_false_alerts * (1 - dedup_share).