ConvLayerScenario

Training scenarios · Implementation: mlsysim/engine/scenario_definitions/training.py.

Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.

One kernel-by-kernel convolution, stride one, over a height-by-width-by-in_channels input to out_channels feature maps; values of bytes_per_value each.

same_padding keeps the output the size of the input; without it the window stays inside the input (“valid”), as an unpadded unfold does.

Notes

Equation: Each output position reuses k*k*Cin*Cout weights; batch FLOPs are twice positions*weights*batch.

Assumptions: Stride one, same or valid padding, and one-pass traffic at the declared precisions.

Excludes: Dilation, groups other than the separate depthwise illustration, workspace, and measured kernel traffic.

References

Derivation: A convolution filter contains k*k*Cin weights for each output channel. Applying that filter at each output location repeats its MACs without adding new parameters. The separate depthwise-plus-pointwise construction uses one spatial filter per input channel followed by channel mixing.

Examples

>>> from mlsysim import Scenarios
>>> s = Scenarios.MnistFirstConvLayer
>>> s.variant(batch_size=2*s.batch_size).flops == 2*s.flops
True

Attributes

Name	Description
activation_bytes	Output activation footprint in bytes for one example.
activation_to_weight_ratio	Dimensionless one-example output footprint divided by filter storage.
batch_input_bytes	Input activation bytes read once for the whole batch.
batch_output_bytes	Output activation bytes written once for the whole batch.
batch_size
bytes_per_value
dense_weights	Weights of a dense layer from the whole input to `out_channels` outputs.
depthwise_weights	One `kernel`-by-`kernel` filter per input channel (multiply-adds per position).
flops	2 x batch x MACs: one multiply and one add per weight per position.
height
in_channels
input_values	Input activation count across spatial positions and channels.
kernel
macs	Multiply-accumulate count for one example, reusing each filter at every output position.
max_copies	Most rows of the patch matrix one input value can appear in.
out_channels
output_height	Output rows for stride-one same or valid padding.
output_width	Output columns for stride-one same or valid padding.
outputs	Output activation values for one example.
pointwise_weights	A 1x1 convolution mixing `in_channels` into `out_channels` (multiply-adds per position).
positions	Output positions per feature map, each a reuse of every filter weight.
same_padding
separable_fraction	Depthwise-separable parameter fraction relative to a standard convolution.
separable_limit	The ratio’s limit with many output channels: the filter area.
separable_ratio	Standard over depthwise separable weights (and multiply-adds per position).
separable_weights	Combined depthwise and pointwise weight count.
sharing_ratio	Dense weights over convolution weights for the same input and feature count.
traffic_bytes	Batch input read, weights read, and batch output written once each.
traffic_intensity	FLOP per byte of the one-pass traffic.
unfolded_entries	Entries of the im2col patch matrix: one row of `kernel*2 in_channels` per position.
weight_bytes	Filter storage in bytes, using a separate weight width when supplied.
weight_bytes_per_value
weights	Filter weight count across spatial kernel, input channels, and output channels.
width

Methods

Name	Description
dense_weight_growth	Factor by which the dense layer’s weights grow when the input becomes `height` by `width`.

dense_weight_growth

engine.scenario_types.ConvLayerScenario.dense_weight_growth(height, width)

Factor by which the dense layer’s weights grow when the input becomes height by width.

Equation: height * width / (height * width).