ContextMemoryCrossingScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Three memories of workload as context grows on hardware: fixed weights, a KV cache linear in tokens, and one layer’s stored scores (all heads) quadratic in tokens.
Notes
Equation: Weights are fixed, full-request KV is linear in context, and one layer of scores grows quadratically.
Assumptions: All query heads materialize scores and the workload supplies its KV geometry.
Excludes: Other layers of scores in score_crossing, workspaces, and a combined residency feasibility test.
References
Derivation: Equate the constant weight footprint with linear KV storage to find their crossing. Equate device capacity with the quadratic score footprint for a score-only capacity crossing. Equating linear KV with quadratic scores and canceling one positive context factor gives the KV-score crossing.
Examples
>>> from mlsysim import Scenarios
>>> s = Scenarios.Gpt2XlContextMemoryOnH100
>>> s.kv_crossing > 0
TrueAttributes
| Name | Description |
|---|---|
| bytes_per_value | |
| hardware | |
| kv_crossing | Tokens at which the KV cache equals the weights. |
| kv_per_token | Per-request KV-cache bytes added by one retained token. |
| kv_score_crossing | Tokens at which the KV cache and one layer’s scores are equal. |
| score_bytes_per_pair | One layer’s stored scores for one token pair, all heads. |
| score_crossing | Tokens at which one layer’s stored scores fill the device. |
| weights | Model weight footprint in decimal GB at the chosen storage width. |
| workload |
workload
Type: Workload.
Declared default: required.
hardware
Type: HardwareNode.
Declared default: required.
bytes_per_value
Type: Quantity.
Declared default: required.
weights
engine.scenario_types.ContextMemoryCrossingScenario.weights
Model weight footprint in decimal GB at the chosen storage width.
Equation: workload.size_in_bytes(bytes_per_value).to('GB').
kv_per_token
engine.scenario_types.ContextMemoryCrossingScenario.kv_per_token
Per-request KV-cache bytes added by one retained token.
Equation: workload.get_kv_cache_size(1, 1, bytes_per_value).to('byte').
score_bytes_per_pair
engine.scenario_types.ContextMemoryCrossingScenario.score_bytes_per_pair
One layer’s stored scores for one token pair, all heads.
Equation: (workload.heads * bytes_per_value).to('byte').
kv_crossing
engine.scenario_types.ContextMemoryCrossingScenario.kv_crossing
Tokens at which the KV cache equals the weights.
Equation: (weights / kv_per_token).to('').magnitude.
score_crossing
engine.scenario_types.ContextMemoryCrossingScenario.score_crossing
Tokens at which one layer’s stored scores fill the device.
Equation: _math.sqrt((hardware.memory.capacity / score_bytes_per_pair).to('').magnitude).
kv_score_crossing
engine.scenario_types.ContextMemoryCrossingScenario.kv_score_crossing
Tokens at which the KV cache and one layer’s scores are equal.
Equation: (kv_per_token / score_bytes_per_pair).to('').magnitude.