TrainingStateFitScenario

Training scenarios · Implementation: mlsysim/engine/scenario_definitions/training.py.

Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.

Weights, gradients, and optimizer state of workload at one precision, against hardware’s memory.

bytes_per_value stores each weight and gradient; the optimizer keeps optimizer_bytes_per_parameter (two FP32 values for Adam, one for SGD with momentum). Activations are not included.

Notes

Equation: Parameter state sums weights, gradients, optimizer state, and optional master weights; adapter mode keeps base weights frozen.

Assumptions: Bytes per value/state and the activation budget are supplied; activation storage scales linearly with batch.

Excludes: Runtime workspaces, allocator reserve, optimizer-specific extra temporaries, and capacity safety margins.

References

Derivation: Uses the neural training-state accounting for full-model weights, gradients, and optimizer state, then adds an explicitly configured master copy. In adapter mode the base weights remain resident while trained adapter parameters receive their own state. The activation budget is supplied separately, so parameter-state fit and total training fit are different questions.

Examples

>>> from mlsysim import Scenarios
>>> s = Scenarios.Gpt2Fp32AdamOnH100
>>> s.weights.to("byte").magnitude > 0
True

Attributes

Name	Description
activations
adapted_matrices
adapter_count	Trainable adapter parameters, stated or from the adapter shape.
adapter_fraction	Dimensionless adapter parameter count relative to all base-model parameters.
adapter_gradients	Adapter gradient storage in bytes at the configured value width.
adapter_in_dim
adapter_kept_share	Dimensionless fraction of full-tuning parameter state retained by this setup.
adapter_master	Adapter master-copy storage in bytes, or zero without a master format.
adapter_optimizer_state	Adapter optimizer-state storage in bytes.
adapter_out_dim
adapter_parameters
adapter_rank
adapter_reduction	Full-state bytes over adapter-run bytes.
adapter_reduction_ceiling	The reduction as adapters shrink to nothing: bytes per trained over bytes per frozen parameter.
adapter_saved	Parameter-state bytes avoided by the configured adapter setup relative to full tuning.
adapter_state	Everything the adapters hold: weights, gradients, master copy, optimizer state.
adapter_weights	Adapter weight storage in bytes at the configured value width.
batch_size
bytes_per_trained_parameter	Weights, gradients, master copy, and optimizer state for one trained parameter.
bytes_per_value
capacity_share	State before activations as a share of `hardware`’s memory.
checkpoint	A resumable checkpoint: weights, master copy, and optimizer state; gradients are recomputed.
copies	Total state in units of the weights.
fits	Whether parameter state alone fits the selected device; false if no hardware is supplied.
frozen_share	The frozen base’s share of the adapter run’s state.
full_state	State if every parameter trained: the same recipe with no adapters.
gradients	Full-model gradient storage in bytes from the shared neural training-state accounting.
hardware
master_bytes_per_parameter
master_weights	Optional full-model master-weight storage in bytes; zero when no master copy is configured.
max_resident_batch	Integer floor of the continuous resident-batch limit, without clamping infeasible values.
optimizer_bytes_per_parameter
optimizer_state	Full-model optimizer storage in bytes from the shared neural training-state accounting.
optimizer_values_per_parameter	Optimizer byte budget per parameter divided by the configured weight-value width.
resident_batch_limit	Sequences whose activations fit beside the state in `hardware`’s memory (added 2026-10-20).
total	Parameter-state bytes: frozen base weights plus adapter state, or full training state and master weights.
training_capacity_multiple	State and saved activations in units of `hardware`’s memory.
training_copies	Training total in units of the weights.
training_total	Weights, gradients, optimizer state, and saved activations.
update_bytes_per_parameter	Bytes one optimizer update moves per parameter: read the gradient, the updated
update_time	The optimizer update at `hardware`’s memory bandwidth (elementwise, bandwidth-bound).
update_traffic	Full-model optimizer-update traffic in GB under the declared read/write accounting.
weights	Base-model weight storage in bytes from the shared neural training-state accounting.
workload

Methods

Name	Description
activations_at	Saved activations at another batch size, scaled linearly from `activations`.
checkpoint_write_time	Seconds to write `checkpoint` at `bandwidth`.

activations_at

engine.scenario_types.TrainingStateFitScenario.activations_at(batch_size)

Saved activations at another batch size, scaled linearly from activations.

Equation: (activations * batch_size / batch_size).to('byte').

checkpoint_write_time

engine.scenario_types.TrainingStateFitScenario.checkpoint_write_time(bandwidth)

training_total

engine.scenario_types.TrainingStateFitScenario.training_total

Weights, gradients, optimizer state, and saved activations.

Equation: (total + activations).to('byte').

training_copies

engine.scenario_types.TrainingStateFitScenario.training_copies

Training total in units of the weights.

Equation: (training_total / weights).to('').magnitude.

weights

engine.scenario_types.TrainingStateFitScenario.weights

Base-model weight storage in bytes from the shared neural training-state accounting.

Equation: _state()[0].

gradients

engine.scenario_types.TrainingStateFitScenario.gradients

Full-model gradient storage in bytes from the shared neural training-state accounting.

Equation: _state()[1].

optimizer_state

engine.scenario_types.TrainingStateFitScenario.optimizer_state

Full-model optimizer storage in bytes from the shared neural training-state accounting.

Equation: _state()[2].

optimizer_values_per_parameter

engine.scenario_types.TrainingStateFitScenario.optimizer_values_per_parameter

Optimizer byte budget per parameter divided by the configured weight-value width.

Equation: (optimizer_bytes_per_parameter / bytes_per_value).to('').magnitude.

master_weights

engine.scenario_types.TrainingStateFitScenario.master_weights

Optional full-model master-weight storage in bytes; zero when no master copy is configured.

Equation: (workload.parameters.to('count').magnitude * master_bytes_per_parameter).to('byte').

bytes_per_trained_parameter

engine.scenario_types.TrainingStateFitScenario.bytes_per_trained_parameter

Weights, gradients, master copy, and optimizer state for one trained parameter.

Equation: master = Q_(0, 'byte') if master_bytes_per_parameter is None else master_bytes_per_parameter; (2 * bytes_per_value + master + optimizer_bytes_per_parameter).to('byte').

adapter_state

engine.scenario_types.TrainingStateFitScenario.adapter_state

Everything the adapters hold: weights, gradients, master copy, optimizer state.

Equation: (adapter_count * bytes_per_trained_parameter).to('byte').

adapter_weights

engine.scenario_types.TrainingStateFitScenario.adapter_weights

Adapter weight storage in bytes at the configured value width.

Equation: (adapter_count * bytes_per_value).to('byte').

adapter_gradients

engine.scenario_types.TrainingStateFitScenario.adapter_gradients

Adapter gradient storage in bytes at the configured value width.

Equation: (adapter_count * bytes_per_value).to('byte').

adapter_master

engine.scenario_types.TrainingStateFitScenario.adapter_master

Adapter master-copy storage in bytes, or zero without a master format.

Equation: master = Q_(0, 'byte') if master_bytes_per_parameter is None else master_bytes_per_parameter; (adapter_count * master).to('byte').

adapter_optimizer_state

engine.scenario_types.TrainingStateFitScenario.adapter_optimizer_state

Adapter optimizer-state storage in bytes.

Equation: (adapter_count * optimizer_bytes_per_parameter).to('byte').

adapter_fraction

engine.scenario_types.TrainingStateFitScenario.adapter_fraction

Dimensionless adapter parameter count relative to all base-model parameters.

Equation: adapter_count / workload.parameters.to('count').magnitude.

full_state

engine.scenario_types.TrainingStateFitScenario.full_state

State if every parameter trained: the same recipe with no adapters.

Equation: variant(adapter_parameters=None, adapter_rank=None).total.

adapter_saved

engine.scenario_types.TrainingStateFitScenario.adapter_saved

Parameter-state bytes avoided by the configured adapter setup relative to full tuning.

Equation: (full_state - total).to('byte').

adapter_reduction

engine.scenario_types.TrainingStateFitScenario.adapter_reduction

Full-state bytes over adapter-run bytes.

Equation: (full_state / total).to('').magnitude.

adapter_kept_share

engine.scenario_types.TrainingStateFitScenario.adapter_kept_share

Dimensionless fraction of full-tuning parameter state retained by this setup.

Equation: (total / full_state).to('').magnitude.

frozen_share

engine.scenario_types.TrainingStateFitScenario.frozen_share

The frozen base’s share of the adapter run’s state.

Equation: (weights / total).to('').magnitude.

adapter_reduction_ceiling

engine.scenario_types.TrainingStateFitScenario.adapter_reduction_ceiling

The reduction as adapters shrink to nothing: bytes per trained over bytes per frozen parameter.

Equation: (bytes_per_trained_parameter / bytes_per_value).to('').magnitude.

total

engine.scenario_types.TrainingStateFitScenario.total

Parameter-state bytes: frozen base weights plus adapter state, or full training state and master weights.

Equation: Adapter mode uses base_weight_bytes + adapter_state_bytes; full tuning sums weights, gradients, optimizer state, and master weights.

copies

engine.scenario_types.TrainingStateFitScenario.copies

Total state in units of the weights.

Equation: (total / weights).to('').magnitude.

fits

engine.scenario_types.TrainingStateFitScenario.fits

Whether parameter state alone fits the selected device; false if no hardware is supplied.

Equation: hardware is not None and total <= hardware.memory.capacity.

capacity_share

engine.scenario_types.TrainingStateFitScenario.capacity_share

State before activations as a share of hardware’s memory.

Equation: (total / hardware.memory.capacity).to('').magnitude.

training_capacity_multiple

engine.scenario_types.TrainingStateFitScenario.training_capacity_multiple

State and saved activations in units of hardware’s memory.

Equation: (training_total / hardware.memory.capacity).to('').magnitude.

resident_batch_limit

engine.scenario_types.TrainingStateFitScenario.resident_batch_limit

Sequences whose activations fit beside the state in hardware’s memory (added 2026-10-20).

Equation: per_sequence = activations / batch_size; ((hardware.memory.capacity - total) / per_sequence).to('').magnitude.

max_resident_batch

engine.scenario_types.TrainingStateFitScenario.max_resident_batch

Integer floor of the continuous resident-batch limit, without clamping infeasible values.

Equation: _math.floor(resident_batch_limit).

checkpoint

engine.scenario_types.TrainingStateFitScenario.checkpoint

A resumable checkpoint: weights, master copy, and optimizer state; gradients are recomputed.

Equation: (weights + master_weights + optimizer_state).to('byte').

update_bytes_per_parameter

engine.scenario_types.TrainingStateFitScenario.update_bytes_per_parameter

Bytes one optimizer update moves per parameter: read the gradient, the updated weights, and the optimizer state; write the state and the weights back.

Equation: p = workload.parameters.to('count').magnitude; updated = (master_weights if master_bytes_per_parameter is not None else weights) / p; (gradients / p + 2 * updated + 2 * optimizer_state / p).to('byte').

update_traffic

engine.scenario_types.TrainingStateFitScenario.update_traffic

Full-model optimizer-update traffic in GB under the declared read/write accounting.

Equation: (update_bytes_per_parameter * workload.parameters.to('count').magnitude).to('GB').

update_time

engine.scenario_types.TrainingStateFitScenario.update_time

The optimizer update at hardware’s memory bandwidth (elementwise, bandwidth-bound).

Equation: (update_traffic / hardware.memory.bandwidth).to('ms').