Skip to main content

Statistics Scenarios

Sampling, uncertainty, evidence, and evaluation decisions.

All domains · Classes A–Z

Source locations

  • Inputs and calculations: mlsysim/engine/scenario_definitions/statistics.py.
  • Named configurations: mlsysim/engine/scenario_catalog/statistics/.
  • Result schemas: mlsysim/engine/result_types/statistics.py.

Public imports and existing class reference URLs remain stable.

Scenario contracts

  • AccuracyFrontierScenario: A hypothetical accuracy-against-disparity frontier, floor + span (1 - exp(-rate x)), with named candidate disparities on it.
  • AgreementDriftScenario: A labeling campaign’s kappa at a baseline and again after elapsed.
  • AnnotatorAgreementScenario: Two independent annotators with the same label mix class_shares: chance agreement, and Cohen’s kappa at an observed_agreement.
  • ArgumentConfidenceScenario: A stipulated safety argument: premises required premises at premise_confidence each, and a claim carried by legs independent legs at leg_confidence each.
  • ArmComparisonScenario: Two policies run arm_runs times each, with successes_a and successes_b successes; Fisher’s exact test, and the runs per arm that separate p_high from p_low two-sided at alpha with power.
  • AutomationBiasScenario: A human reviews an AI’s output; the human accepts miss_acceptance of the AI’s misses.
  • BiasFeedbackScenario: A gap passed at launch, refit from the next audit, iterated over retraining cycles: gap(k) = launch_gap * factor^k against a release gate (all in percentage points).
  • BinShareShiftScenario: A feature binned at the training quantiles: training shares against recent shares, scored by PSI.
  • BinomialIntervalsScenario: Several success counts out of one number of independent tasks.
  • CappedSelectionCostScenario: Candidates for one fixed task, an exact checker, and a cap on candidates.
  • CertifiedRadiusScenario: Binary randomized smoothing at noise noise_sd, against an l-infinity budget on one input.
  • ClosedLoopCampaignScenario: clean_runs failure-free closed-loop trials of a frozen policy in one cell of table on site, each run long with an automated reset, bounded at one-sided confidence.
  • ConfidenceRegimesScenario: Stipulated proposer confidence against task success in several operating regimes.
  • ConstraintAccuracyCostScenario: Accuracy before and after a fairness constraint (percent), with the policy’s stated rates.
  • CrashRateDenominatorScenario: A reported before-and-after crash rate and the same comparison on the cohort whose exposure is complete (rates per million miles).
  • DetectionDelayLossScenario: A fault that cuts quality_drop of annual_value until it is detected, found after slow_detection or fast_detection, incidents_per_year times a year.
  • DriftConfirmationScenario: Labeled outcomes that confirm a drop of drop from baseline_accuracy.
  • EfficiencyClaimGapScenario: A datasheet operating point against a sustained one: operations per watt at each, and their ratio.
  • EqualOpportunityPriceScenario: Utility per applicant of raising group’s TPR to reference_group’s by moving its threshold, when that move raises its FPR by fpr_increase: a repaid loan earns repay_margin, a default loses default_loss (both USD).
  • ExplanationLogScenario: Logging an explanation with every prediction: predictions_per_day for days_per_month, each a map_size_low to map_size_high attribution map (or tabular attributions of attribution_bytes_low to attribution_bytes_high per feature).
  • ExplorationExposureScenario: A ranking service showing slots_per_user items a day to daily_users from a catalog of catalog_items, with explore_share of slots served by randomized exploration.
  • FalseAlarmBudgetScenario: One detection window every window, listening_per_day of each day, over horizon.
  • FloatBitFlipScenario: One stored binary32 value with bit flipped_bit flipped (bit 30 is the top exponent bit).
  • GaussianDriftScenario: A continuous feature fitted as a normal at reference and now: KL divergence of the reference from the current fit.
  • GroupConfusionScenario: Binary decisions for two or more groups as confusion counts {group: {tp, fp, fn, tn}}.
  • HazardExposureScenario: Failure-free exposure that bounds each of rates (constant hazards per hour) at one-sided confidence, against what rigs rigs or one rig over rig_time accumulate.
  • HostAcceleratorBoardScenario: One board, a host CPU and an attached accelerator: model-only and end-to-end latency, active energy.
  • InlineOverheadScenario: A safeguard added inline to a request path of base_latency, costing overhead_low to overhead_high (fractions of the base) per request.
  • InterventionCampaignScenario: interventions interventions in missions supervised runs of mission: burden per hour and per kilometer, and an exact two-sided 1 - alpha interval on the rate.
  • KsDriftScenario: A two-sample K-S drift test of n_reference training against n_current recent values of one feature.
  • LaplaceMeanReleaseScenario: The mean of a public roster of group_size values in [0, value_range] released under epsilon-DP by one Laplace draw on the sum (replace-one adjacency).
  • MeanShiftTestScenario: A feature mean that moves from reference_mean to observed_mean over n_samples requests.
  • MeanVersusTailScenario: A benchmark’s mean latency against a production tail stated as a range.
  • MembershipAuditScenario: A membership-inference audit on the ROC plane against the ceiling a pure epsilon-DP release allows; the audit curve FPR + a FPR (1 - FPR) is illustrative.
  • MissionExposureScenario: Failure-free missions of mission at machine’s drive limit, with reset between them, that bound hazard_rate at one-sided confidence; and the exposure a relaxed target of one failure per relaxed_mtbf needs.
  • MonitorFleetAlertScenario: models x metrics_per_model monitors checked every check_interval at a per-check false_positive_rate on a stable process; dedup_share of crossings are merged away.
  • PairedComparisonScenario: One candidate and one baseline trial on each sampled task, outcomes kept in pairs.
  • PreferenceLabelBudgetScenario: Human preference labels for a reward model: comparisons_low to comparisons_high at price_low to price_high per comparison.
  • ProportionGapTestScenario: An observed gap between two rates, rate_a and rate_b, on n_per_group decisions each.
  • ProportionPlanningScenario: Planning targets for one acceptance rate and for two independent arms.
  • ProportionSampleSizeScenario: Samples that pin a pass fraction rate to within half_width at confidence: z^2 p (1 - p) / m^2.
  • ProxyMetricLatencyScenario: A search-width change that buys an offline metric gain with latency, against a serving budget.
  • QuantizedAcceptanceGatesScenario: A float model and its quantized copy against acceptance gates set before compression.
  • RetrainingIntervalScenario: Retraining a model whose accuracy decays as drift fits it, serving query_rate queries worth value_per_query per unit accuracy, at run_cost a run. footprint and accelerator_price price the compute share of a run; compared_intervals and scheduled_interval are cadences to compare with the optimum.
  • RoadExposureScenario: Miles-to-demonstrate requirements per event class (zero failures, precision, improvement), in millions of miles at human benchmark rates per 100 million miles, against one dated fleet_miles total (millions) at one-sided confidence.
  • SameTestSetAccuracyComparisonScenario: Two models scored on the same n_samples test items: independent-set interval and McNemar’s limit.
  • SampledDecisionMonitorScenario: Per-group metric monitoring of a decision service at request_rate.
  • SegmentAccuracyScenario: Accuracy measured separately on each traffic segment; traffic_shares and accuracies are keyed by segment name.
  • SequentialTestScenario: Wald’s SPRT between an unacceptable failure rate p_unacceptable and an acceptable one p_acceptable at error rates alpha and beta; the score after trace_clean_runs clean runs and one failure; and the fixed zero-failure test for success 1 - p_unacceptable at the same alpha.
  • ShardedUnlearningScenario: Exact unlearning by sharded retraining (SISA) against a full retrain of workload.
  • StrongScalingPairScenario: The same fixed job on base_devices and on scaled_devices: speedup, ideal time, and efficiency.
  • SubgroupEvaluationBudgetScenario: The labeled data a per-group evaluation needs: one group’s rate within half_width at confidence (worst case rate 0.5), the collection random sampling needs to reach a target_per_group from a group of group_share, and the per-group count that pins a gap between two equal-size groups to gap_half_width.
  • TaskMixConsistencyScenario: Equally weighted task types and a number of runs that must all pass.
  • TwoProportionTestScenario: A two-arm test sized to detect an absolute lift on a baseline_rate at the normal quantiles z_alpha and z_beta, on daily_requests split between arms.
  • ZeroEventExposureScenario: Exposure needed, with zero events observed, to bound a rate of budget per horizon at one-sided confidence, and the size of recording that exposure at sample_rate.
Back to top