TransformerBlockScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
workload’s decoder blocks at a context of context_tokens: the classic block’s weights (4 d^2 attention projections, 2 x ffn_multiple d^2 MLP), its dense and attention multiply-adds, and, with hardware, the attention arithmetic of one prompt at peak.
Notes
Equation: Shared transformer helpers count dense block parameters and full attention products; two FLOPs are counted per MAC.
Assumptions: Classic dense blocks at the declared FFN multiplier; exact_parameters specifically describes the GPT-2 bias-enabled layout.
Excludes: Gated FFNs, grouped-query geometry, kernel fusion, and vocabulary-output work from prompt_flops.
References
Derivation: Uses mlsysim.physics.transformer.calc_transformer_block_weights, calc_attention_score_macs, and calc_attention_crossover for the classic block geometry. Matrix MAC counts become FLOPs by counting a multiply and add separately. The exact_parameters property adds GPT-2-specific biases, normalization parameters, and position embeddings rather than treating the coarse block rule as exact.
Examples
>>> from mlsysim import Scenarios
>>> s = Scenarios.Gpt2XlBlockFullWindow
>>> s.block_macs == s.dense_macs+s.attention_macs
TrueAttributes
| Name | Description |
|---|---|
| attention_macs | One block’s two attention products over the context. |
| attention_projections | |
| attention_share | Dimensionless attention-product fraction of one block’s prompt MACs. |
| attention_step_flops | The attention products’ share of a training step, all layers. |
| attention_time_at_peak | Ideal attention-only time in minutes at the selected device’s peak compute rate. |
| attention_to_dense | Dimensionless attention-product MAC count divided by projection-and-MLP MACs. |
| attention_weights | Attention projection parameter count across all decoder blocks. |
| attention_weights_per_block | Attention projection parameter count returned by calc_transformer_block_weights. |
| batch_size | |
| block_coefficient | Weights per block in units of d^2 (12 for the classic block). |
| block_macs | Dense projection plus attention multiply-accumulates for one block over the prompt. |
| block_rule_parameters | Block matrix parameters plus one token embedding table; biases and position table omitted. |
| block_weights | All blocks’ weights. |
| context_tokens | |
| crossover | Context length at which the attention products equal the dense products (6 d). |
| crossover_multiple | The crossover in units of the width (six for the classic block). |
| dense_macs | One block’s projection and MLP multiply-adds over the context. |
| dense_to_attention | Dimensionless projection-and-MLP MAC count divided by attention-product MACs. |
| exact_parameters | GPT-2 layout counted in full: projection and MLP biases, two LayerNorms per block, |
| ffn_multiple | |
| flops_per_token | Two FLOPs (a multiply and an add) per block weight per token. |
| hardware | |
| layer_forward_flops | One block’s forward FLOPs for the whole batch: dense and attention products, two FLOPs each. |
| mlp_share | Dimensionless feed-forward matrix share of one block’s counted parameters. |
| mlp_weights | Feed-forward projection parameter count across all decoder blocks. |
| mlp_weights_per_block | Feed-forward projection parameter count returned by calc_transformer_block_weights. |
| output_layer_flops | The output projection onto the vocabulary, per token. |
| prompt_attention_flops | Every block’s attention products over the prompt, two FLOPs per multiply-add. |
| prompt_dense_flops | Dense block work in FLOPs over all prompt tokens and layers. |
| prompt_flops | Prompt FLOPs combining dense block work and attention products, excluding vocabulary output work. |
| six_p_step_flops | The same step by the 6P-per-token rule, which omits the attention products. |
| token_table | Token-embedding weights: one row of width d per vocabulary entry. |
| training_step_flops | One training step counted layer by layer, attention products included. |
| training_to_forward | A training step in forward passes: 6P over 2P per token. |
| weights_per_block | Combined attention and feed-forward matrix parameter count for one decoder block. |
| width | Model hidden width shared by the classic decoder-block formulas. |
| workload |
workload
Type: Workload.
Declared default: required.
context_tokens
Type: int.
Declared default: required.
ffn_multiple
Type: int.
Declared default: 4.
attention_projections
Type: int.
Declared default: 4.
hardware
Type: Optional[HardwareNode].
Declared default: None.
batch_size
Type: int.
Declared default: 1.
width
engine.scenario_types.TransformerBlockScenario.width
Model hidden width shared by the classic decoder-block formulas.
Equation: workload.hidden_dim.
attention_weights_per_block
engine.scenario_types.TransformerBlockScenario.attention_weights_per_block
Attention projection parameter count returned by calc_transformer_block_weights.
Equation: _w()[0].
mlp_weights_per_block
engine.scenario_types.TransformerBlockScenario.mlp_weights_per_block
Feed-forward projection parameter count returned by calc_transformer_block_weights.
Equation: _w()[1].
weights_per_block
engine.scenario_types.TransformerBlockScenario.weights_per_block
Combined attention and feed-forward matrix parameter count for one decoder block.
Equation: _w()[2].
block_coefficient
engine.scenario_types.TransformerBlockScenario.block_coefficient
Weights per block in units of d^2 (12 for the classic block).
Equation: weights_per_block // width ** 2.
block_weights
engine.scenario_types.TransformerBlockScenario.block_weights
All blocks’ weights.
Equation: weights_per_block * workload.layers.
attention_weights
engine.scenario_types.TransformerBlockScenario.attention_weights
Attention projection parameter count across all decoder blocks.
Equation: attention_weights_per_block * workload.layers.
mlp_weights
engine.scenario_types.TransformerBlockScenario.mlp_weights
Feed-forward projection parameter count across all decoder blocks.
Equation: mlp_weights_per_block * workload.layers.
token_table
engine.scenario_types.TransformerBlockScenario.token_table
Token-embedding weights: one row of width d per vocabulary entry.
Equation: workload.vocab_size * width.
block_rule_parameters
engine.scenario_types.TransformerBlockScenario.block_rule_parameters
Block matrix parameters plus one token embedding table; biases and position table omitted.
Equation: block_weights + token_table.
exact_parameters
engine.scenario_types.TransformerBlockScenario.exact_parameters
GPT-2 layout counted in full: projection and MLP biases, two LayerNorms per block, the final LayerNorm, and a position table of context_length rows (nanoGPT, bias=True).
Equation: d, f = (width, ffn_multiple); per_block = d * 3 * d + 3 * d + (d * d + d) + (d * f * d + f * d) + (f * d * d + d) + 2 * (2 * d); workload.layers * per_block + workload.vocab_size * d + workload.context_length * d + 2 * d.
flops_per_token
engine.scenario_types.TransformerBlockScenario.flops_per_token
Two FLOPs (a multiply and an add) per block weight per token.
Equation: Q_(2 * block_weights, 'flop').
output_layer_flops
engine.scenario_types.TransformerBlockScenario.output_layer_flops
The output projection onto the vocabulary, per token.
Equation: Q_(2 * token_table, 'flop').
dense_macs
engine.scenario_types.TransformerBlockScenario.dense_macs
One block’s projection and MLP multiply-adds over the context.
Equation: weights_per_block * context_tokens.
attention_macs
engine.scenario_types.TransformerBlockScenario.attention_macs
One block’s two attention products over the context.
Equation: calc_attention_score_macs(context_tokens, width).
block_macs
engine.scenario_types.TransformerBlockScenario.block_macs
Dense projection plus attention multiply-accumulates for one block over the prompt.
Equation: dense_macs + attention_macs.
dense_to_attention
engine.scenario_types.TransformerBlockScenario.dense_to_attention
Dimensionless projection-and-MLP MAC count divided by attention-product MACs.
Equation: dense_macs / attention_macs.
attention_to_dense
engine.scenario_types.TransformerBlockScenario.attention_to_dense
Dimensionless attention-product MAC count divided by projection-and-MLP MACs.
Equation: attention_macs / dense_macs.
crossover
engine.scenario_types.TransformerBlockScenario.crossover
Context length at which the attention products equal the dense products (6 d).
Equation: calc_attention_crossover(width, ffn_multiple, attention_projections).
crossover_multiple
engine.scenario_types.TransformerBlockScenario.crossover_multiple
The crossover in units of the width (six for the classic block).
Equation: crossover / width.
prompt_dense_flops
engine.scenario_types.TransformerBlockScenario.prompt_dense_flops
Dense block work in FLOPs over all prompt tokens and layers.
Equation: (flops_per_token * context_tokens).to('flop').
prompt_attention_flops
engine.scenario_types.TransformerBlockScenario.prompt_attention_flops
Every block’s attention products over the prompt, two FLOPs per multiply-add.
Equation: Q_(2 * attention_macs * workload.layers, 'flop').
prompt_flops
engine.scenario_types.TransformerBlockScenario.prompt_flops
Prompt FLOPs combining dense block work and attention products, excluding vocabulary output work.
Equation: prompt_dense_flops + prompt_attention_flops.
attention_time_at_peak
engine.scenario_types.TransformerBlockScenario.attention_time_at_peak
Ideal attention-only time in minutes at the selected device’s peak compute rate.
Equation: (prompt_attention_flops / hardware.compute.peak_flops).to('minute').
training_to_forward
engine.scenario_types.TransformerBlockScenario.training_to_forward
A training step in forward passes: 6P over 2P per token.
Equation: p = workload.parameters; (calc_transformer_training_flops(p, 1) / calc_transformer_decode_flops(p, 1)).to('').magnitude.
layer_forward_flops
engine.scenario_types.TransformerBlockScenario.layer_forward_flops
One block’s forward FLOPs for the whole batch: dense and attention products, two FLOPs each.
Equation: Q_(2 * block_macs * batch_size, 'flop').to('GFLOP').
training_step_flops
engine.scenario_types.TransformerBlockScenario.training_step_flops
One training step counted layer by layer, attention products included.
Equation: (layer_forward_flops * workload.layers * training_to_forward).to('TFLOP').
six_p_step_flops
engine.scenario_types.TransformerBlockScenario.six_p_step_flops
The same step by the 6P-per-token rule, which omits the attention products.
Equation: calc_transformer_training_flops(workload.parameters, batch_size * context_tokens).to('TFLOP').
attention_step_flops
engine.scenario_types.TransformerBlockScenario.attention_step_flops
The attention products’ share of a training step, all layers.
Equation: Q_(2 * attention_macs * batch_size * workload.layers * training_to_forward, 'flop').to('TFLOP').