DenseScalingSweepScenario

Training scenarios · Implementation: mlsysim/engine/scenario_definitions/training.py.

Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.

Dense networks between a fixed input and output, swept in hidden width and in depth.

Width sweep: hidden_layers hidden layers, each of every width in widths. Depth sweep: 1 to max_depth hidden layers of fixed_width. Each doubling factor compares the last setting with the one half its size.

Notes

Equation: Width and depth sweeps call calc_dense_macs; saved state calls calc_dense_saved_activations.

Assumptions: The width list contains its final width halved; max_depth is at least two for doubling comparisons.

Excludes: Runtime, hardware saturation, and accuracy changes with model size.

References

Derivation: Calls mlsysim.physics.neural.calc_dense_macs for each constructed width sequence and calc_dense_saved_activations for the corresponding training-state count. Ratios compare already evaluated settings; they do not assume exact quadratic or linear scaling for the fixed input/output layers.

Examples

>>> from mlsysim import Scenarios
>>> s = Scenarios.DenseWidthDepthSweep
>>> len(s.width_sweep_macs) == len(s.widths)
True

Attributes

Name	Description
depth_double_factor	Dimensionless MAC increase between the final depth and its half-depth entry.
depth_doubled_from	Hidden depth used as the denominator of the final depth-doubling comparison.
depth_sweep_macs	Per-example multiply-accumulate counts for each hidden depth at fixed width.
depth_sweep_saved_activations	Values saved per example: the input and every layer output.
depths	Enumerate hidden-layer counts from one through the configured maximum.
fixed_width
hidden_layers
input_dim
max_depth
output_dim
saved_activations_double_factor	Dimensionless saved-activation increase between final and half-depth entries.
width_double_factor	Dimensionless MAC increase between the final width and its half-width entry.
width_doubled_from	Hidden width used as the denominator of the final width-doubling comparison.
width_sweep_macs	Per-example multiply-accumulate counts for each hidden width at fixed depth.
widths

Methods

Name	Description
net	Build input-hidden-output layer widths for the requested hidden depth and width.

net

engine.scenario_types.DenseScalingSweepScenario.net(hidden_layers, width)

Build input-hidden-output layer widths for the requested hidden depth and width.

Equation: [input_dim] + [width] * hidden_layers + [output_dim].