DenseLayerScenario
Training scenarios · Implementation: mlsysim/engine/scenario_definitions/training.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
One dense layer: in_features inputs to out_features outputs, bytes_per_weight each.
Notes
Equation: A matrix with n inputs and m outputs has n*m weights; a batch costs 2*B*n*m FLOPs.
Assumptions: Dense multiplication, one weight read shared by the batch, and one read/write of activations in the traffic variant.
Excludes: Cache hierarchy, bias/activation FLOPs, and kernel scheduling; target-intensity inversions require a positive denominator.
References
Derivation: Every output dot product reads n weights, so m outputs require n*m weights and MACs per example. Counting one multiply and one add per MAC gives 2*B*n*m FLOPs. The traffic variant separately counts input reads, shared weight reads, and output writes.
Examples
>>> from mlsysim import Scenarios
>>> s = Scenarios.MnistMlpFirstLayer
>>> s.variant(batch_size=2*s.batch_size).flops == 2*s.flops
TrueAttributes
| Name | Description |
|---|---|
| activation_bytes_per_value | |
| batch_size | |
| biases | Bias count: one scalar for each output feature. |
| bytes_per_weight | |
| flops | 2 x batch x in x out: one multiply and one add per weight per example. |
| in_features | |
| input_bytes | Batch input traffic in bytes under one read per activation value. |
| intensity | FLOPs per weight byte at batch_size: each weight read serves the whole batch. |
| macs_per_example | Multiply-accumulate count for one input vector, excluding bias and activation work. |
| out_features | |
| output_bytes | Batch output traffic in bytes under one write per activation value. |
| parameters | Weights plus biases (what a framework’s parameter count reports). |
| traffic_bytes | Input read, weights read, and output written once each. |
| traffic_intensity | FLOP per byte of the one-pass traffic. |
| weight_bytes | Stored weight-matrix footprint in bytes; biases are excluded. |
| weights | Matrix weight count: one connection for every input-output feature pair. |
Methods
| Name | Description |
|---|---|
| batch_for_intensity | Batch size at which the weight-only intensity reaches target FLOP/byte. |
| batch_for_traffic_intensity | Batch at which traffic_intensity reaches target FLOP/byte. |
batch_for_intensity
engine.scenario_types.DenseLayerScenario.batch_for_intensity(target)Batch size at which the weight-only intensity reaches target FLOP/byte.
Equation: target / variant(batch_size=1).intensity.
batch_for_traffic_intensity
engine.scenario_types.DenseLayerScenario.batch_for_traffic_intensity(target)Batch at which traffic_intensity reaches target FLOP/byte.
Solves 2 B n m = target (B n a + n m w + B m a) for B, with n, m the widths and a, w the activation and weight bytes per value.
Equation: n, m = (in_features, out_features); a, w = (_act_bytes, bytes_per_weight.to('byte').magnitude); target * n * m * w / (2 * n * m - target * a * (n + m)).
in_features
Type: int.
Declared default: required.
out_features
Type: int.
Declared default: required.
bytes_per_weight
Type: Quantity.
Declared default: required.
batch_size
Type: int.
Declared default: 1.
weights
engine.scenario_types.DenseLayerScenario.weights
Matrix weight count: one connection for every input-output feature pair.
Equation: in_features * out_features.
biases
engine.scenario_types.DenseLayerScenario.biases
Bias count: one scalar for each output feature.
Equation: out_features.
parameters
engine.scenario_types.DenseLayerScenario.parameters
Weights plus biases (what a framework’s parameter count reports).
Equation: weights + biases.
macs_per_example
engine.scenario_types.DenseLayerScenario.macs_per_example
Multiply-accumulate count for one input vector, excluding bias and activation work.
Equation: weights.
weight_bytes
engine.scenario_types.DenseLayerScenario.weight_bytes
Stored weight-matrix footprint in bytes; biases are excluded.
Equation: (weights * bytes_per_weight).to('byte').
intensity
engine.scenario_types.DenseLayerScenario.intensity
FLOPs per weight byte at batch_size: each weight read serves the whole batch.
Equation: flops = 2 * batch_size * weights; flops / weight_bytes.to('byte').magnitude.
activation_bytes_per_value
Type: Optional[Quantity].
Declared default: None.
flops
engine.scenario_types.DenseLayerScenario.flops
2 x batch x in x out: one multiply and one add per weight per example.
Equation: Q_(2 * batch_size * weights, 'flop').
input_bytes
engine.scenario_types.DenseLayerScenario.input_bytes
Batch input traffic in bytes under one read per activation value.
Equation: Q_(batch_size * in_features * _act_bytes, 'byte').
output_bytes
engine.scenario_types.DenseLayerScenario.output_bytes
Batch output traffic in bytes under one write per activation value.
Equation: Q_(batch_size * out_features * _act_bytes, 'byte').
traffic_bytes
engine.scenario_types.DenseLayerScenario.traffic_bytes
Input read, weights read, and output written once each.
Equation: input_bytes + weight_bytes + output_bytes.
traffic_intensity
engine.scenario_types.DenseLayerScenario.traffic_intensity
FLOP per byte of the one-pass traffic.
Equation: (flops / traffic_bytes).to('flop/byte').magnitude.