Reliability Scenarios

Failure exposure, recovery, and operational safeguards.

All domains · Classes A–Z

Source locations

• Inputs and calculations: mlsysim/engine/scenario_definitions/reliability.py.
• Named configurations: mlsysim/engine/scenario_catalog/reliability/.
• Result schemas: mlsysim/engine/result_types/reliability.py.

Public imports and existing class reference URLs remain stable.

Scenario contracts

• AccuracyDecayScenario: A deployed model read at initial_accuracy and, elapsed later, at later_accuracy; noticed is when someone outside monitoring saw it. decay_rate fits A(t) = A0 exp(-gamma t) through the two readings.
• AutomationBreakevenScenario: A pipeline that costs build_time once, against manual_rate of work by hand; the pipeline itself needs upkeep_rate. Break-even is build / (manual - upkeep).
• AvailabilityScenario: A job that needs all n_components, each up a fraction availability of the time.
• BulkheadAdmissionScenario: One burst of blocking calls to a stalled dependency, uncapped and with a bulkhead.
• CheckpointIntervalScenario: Checkpoints of workload written at write_bandwidth, against failures every mtbf.
• CheckpointTraceScenario: Checkpoint strides priced on turns turns that crash after the turns in crashes.
• DataParallelAttackSurfaceScenario: workload trained on fleet with dp_degree data-parallel replicas: the gradient each step exchanges, the checkpoint it writes, and the traffic of a robust aggregate that gathers every replica’s gradient instead of a ring AllReduce of their sum.
• DualBankUpdateScenario: An over-the-air update that keeps a second flash bank: two copies of model at bytes_per_param in the flash of device.
• EncryptedInferenceLatencyScenario: One inference at plaintext_latency, with symmetric encryption around it (encryption_overhead) and computed under homomorphic encryption (fhe_multiplier).
• FailureIntervalScenario: n_components identical components, each with component_mtbf, over a run.
• FailureIntervalSweepScenario: Interval between failures of component_mtbf components, at each job size in sizes.
• GroupCommitScenario: batch_size intents gathered for up to batch_timer and acknowledged by one flush_time.
• HealthProbeScenario: A readiness probe every interval, failing after timeout, removing after failure_threshold misses.
• HotSpareScenario: spare_devices held idle beside the job in failures, priced at device_hour_rate.
• IrreversibleLossScenario: A workload’s wrong-change risk priced with its pre-pivot check, from its agent profile.
• OutputShapingScenario: An API that returns the top top_k of class_count scores, rounded from from_decimals to to_decimals decimal places.
• ProtectedMemoryFitScenario: workload’s weights at bytes_per_weight against hardware’s protected memory (an enclave page cache), before activations.
• QueryExtractionBudgetScenario: An extractor that needs extraction_queries answers from a priced API, under three billing regimes: pay per query, a paid tier capped per day, and a lifetime per-principal allowance after which every query is billed at the per-query price.
• RecoveryBudgetScenario: Cold restarts of workload after failures of n_devices components of component_mtbf.
• ReleaseFootprintScenario: Resident copies of model that release controls hold on device.
• RepairBreakEvenScenario: One bounded repair attempt against compensating and rebuilding to the same endpoint.
• RetentionCostScenario: One trajectory’s log (turns x envelope and payload) and workspace snapshot kept for period.
• RetryAmplificationScenario: Nested retry layers, each with its own allowance, under one failing logical operation.
• RetryJitterScenario: A cohort of eligible retries released together, synchronized or with full jitter.
• SdcExposureScenario: Silent-corruption exposure of n_devices, each at event_rate per device-hour, per step_time step.
• SeriesFailureScenario: A synchronous job on n_devices that stops when any one component fails.
• SkewCostScenario: Training-serving skew that makes skew_rate of query_rate wrong, each wrong prediction costing cost_per_error.
• SnapshotVolumeScenario: A trajectory of turns turns growing state_growth per turn, checkpointed every stride.
• StragglerCostScenario: One of n_devices in a synchronous job runs at slow_speed of its normal rate.
• ThroughputCostScenario: The same workload and resource budget served at baseline_rate without a set of controls and at controlled_rate with them.
• UndetectedLoopScenario: An append-only loop that resends its history, run to a turn ceiling or stopped early.