Training Scenarios

Training work, state, updates, and parallel execution.

All domains · Classes A–Z

Source locations

• Inputs and calculations: mlsysim/engine/scenario_definitions/training.py.
• Named configurations: mlsysim/engine/scenario_catalog/training/.
• Result schemas: mlsysim/engine/result_types/training.py.

Public imports and existing class reference URLs remain stable.

Scenario contracts

• ActivationLogicCostScenario: Logic cost of two activation datapaths, in transistor equivalents (an illustrative budget).
• ActivationMemoryScenario: Activation memory of workload at a microbatch, under each recomputation strategy.
• AdversarialTrainingPassesScenario: Work per batch of adversarial training, in standard forward-backward passes (equal cost, no reuse).
• AlignmentMemoryScenario: PPO and DPO alignment of policy with critic-sized reward and value models.
• AttentionTilingScenario: One head’s attention at seq_len tokens, whole or in square tiles, and what each kernel stores.
• CenteredRewardScenario: Binary terminal rewards centered on a fixed baseline.
• ContinuationForkCostScenario: A trajectory whose value is sampled by forking continuations at chosen turns.
• ConvLayerScenario: One kernel-by-kernel convolution, stride one, over a height-by-width-by-in_channels input to out_channels feature maps; values of bytes_per_value each.
• CrossEntropyScenario: Cross-entropy losses for named predicted probabilities of the correct class, among classes.
• DataParallelScaleOutScenario: A data-parallel job of workload on nodes nodes, from a one-device profile.
• DenseLayerScenario: One dense layer: in_features inputs to out_features outputs, bytes_per_weight each.
• DenseNetworkScenario: A dense network of layer_widths at batch_size: parameters, FLOPs, traffic, memory.
• DenseScalingSweepScenario: Dense networks between a fixed input and output, swept in hidden width and in depth.
• DpSgdRunScenario: Inputs a DP-SGD privacy accountant needs for one Poisson-sampled run on dataset_size examples; this type does not estimate epsilon.
• DpTrainingPlanScenario: An illustrative differential-privacy planning scenario for one training run of base_cost.
• ElementwiseIntensityScenario: An unfused elementwise operation: flops_per_element FLOPs while operands_moved values of bytes_per_value each are read or written.
• EpochOverheadScenario: One training epoch of network on a host CPU and an accelerator: work against fixed overhead.
• FederatedUplinkScenario: A week of raw captures against compressed model updates sent at named round cadences.
• ForkValueScenario: Binary continuations forked before and after one turn, with their pass counts.
• FrameRateHeadroomScenario: workload at frame_rate on an accelerator of peak_rate that sustains a utilization_low to utilization_high share of peak; alternative_flops is a heavier model per frame.
• GradientChainScenario: The activation-slope factor a gradient keeps after depth layers, at each slope’s peak.
• GroupBarrierScenario: The rollout latencies of one synchronous group.
• GroupCoefficientsScenario: The rewards of one group of rollouts and the coefficient convention applied to them.
• KernelShareScenario: One kernel class’s multiply-adds against a whole model’s, per example.
• LayerStackScenario: Weight-only count of a network built from convolution and dense layers, input first (biases omitted).
• LengthNormalizationScenario: Two rollouts with one group coefficient and different lengths, averaged per rollout.
• LossNormalizationScenario: A long trajectory split into focus_pieces records, batched with other_examples short ones.
• LowRankUpdateScenario: A rank-rank update to one out_dim by in_dim weight matrix.
• ParadigmCostScenario: One input classified three ways: hand-written rules, engineered features, and a learned network.
• ParallelLayoutScenario: workload trained on a TP x PP x DP layout: step time, efficiency, and state per device.
• PerExampleGradientScenario: Naive DP-SGD keeps one gradient per example: bytes_per_param per parameter of each named model, batch_size copies per step.
• PerInvocationSerializationScenario: A turns-turn trajectory stored as one example per model call, each turn adding tokens_per_turn.
• PersonalizationStorageScenario: contexts personalized models stored as full copies or as one backbone plus adapters.
• PipelineScheduleScenario: A synchronous pipeline of stages stages fed microbatches microbatches per step.
• PowerLawScenario: A power-law term A R^-alpha and the resource multiple that halves it.
• PreferencePairScenario: Log-probability sums over a preferred and a dispreferred record’s selected tokens.
• PublishedRunScenario: A published training run: its total compute spread over its accelerators and days.
• RatioClipScenario: One token’s old probability under a symmetric ratio clip.
• RecurrentStepScenario: One step of a recurrent layer: input_dim inputs, a hidden_dim state, and gates affine maps per step (one for a plain RNN, four for an LSTM); steps steps in a sequence.
• ReferenceTrainingStepScenario: One optimizer step of a reference pretraining job, fixed once for every chapter that quotes it.
• RepeatedVerificationScenario: A timing-sensitive check repeated on one candidate, with an all-pass rule.
• ResidualAdapterScenario: Residual bottleneck adapters of width to bottleneck, one at each of sites.
• RewardCompositionScenario: One update’s rollouts scored by a verifier that accepts some failures.
• RolloutSlotCycleScenario: Sandbox slots running one episode at a time, with a cold and a snapshot reset.
• RolloutStepBudgetScenario: One update’s tasks, group size, rollout length, and the policy’s pass rate on the mix.
• ScalingOverheadScenario: A planned speedup when communication and coordination take a share of the scaled step.
• SequencePackingScenario: Stored example lengths against capacity-token buffers.
• SustainedThroughputScenario: A run budgeted from peak throughput, then run at its sustained share of peak.
• TargetMaskScenario: One serialized training example by segment, and the responses kept as context only.
• TensorParallelLayerScenario: One layer of workload split tp_size ways: compute against the exchange on each link.
• TokenizedBatchFeedScenario: A host that tokenizes each batch and copies it to the accelerator, against the device’s step.
• TrainingRunScenario: Training workload on fleet: peak time to delivered wall-clock time.
• TrainingStateFitScenario: Weights, gradients, and optimizer state of workload at one precision, against hardware’s memory.
• TrainingStateScenario: Persistent mixed-precision Adam state of workload and its shards.
• UpdateWorkingSetScenario: A full on-device update with Adam: its working set and the DRAM bytes of one step.
• VerifierPoolScenario: Completed trajectories arriving in batches, each verified by repeated trials.
• ZeroShardingScenario: workload’s training state per device at each ZeRO stage over n_devices.