PackedQkvProjectionScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
tokens rows through workload’s packed query-key-value projection, one d x 3d matrix.
Notes
Equation: Packed width is projections*d; projection MACs are tokens*d*packed_width.
Assumptions: Equal-width dense projections, usually query/key/value, with no bias counting.
Excludes: Grouped-query asymmetry, attention products, and output projection.
References
Derivation: Concatenating equally wide projections produces a d-by-(projections*d) matrix. Every token row performs one MAC for every matrix entry, giving tokens*d*packed_width. Grouped-query projections require a different geometry.
Examples
>>> from mlsysim import Scenarios
>>> s = Scenarios.Gpt2SmallSixTokenQkvProjection
>>> s.macs == s.tokens*s.width*s.packed_width
TrueAttributes
| Name | Description |
|---|---|
| macs | Multiply-accumulate count for all token rows through the packed projection. |
| packed_width | Combined output width of the configured query/key/value projections. |
| projections | |
| tokens | |
| width | Model hidden width used as the projection input dimension. |
| workload |
workload
Type: Workload.
Declared default: required.
tokens
Type: int.
Declared default: required.
projections
Type: int.
Declared default: 3.
width
engine.scenario_types.PackedQkvProjectionScenario.width
Model hidden width used as the projection input dimension.
Equation: workload.hidden_dim.
packed_width
engine.scenario_types.PackedQkvProjectionScenario.packed_width
Combined output width of the configured query/key/value projections.
Equation: projections * width.
macs
engine.scenario_types.PackedQkvProjectionScenario.macs
Multiply-accumulate count for all token rows through the packed projection.
Equation: tokens * width * packed_width.