KvFootprintScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
workload’s KV cache for tokens kept tokens; with hardware and kv_share, how many such requests the cache’s share of the labeled device memory holds. kv_width_factor scales the key-value width (one half: half the key-value heads).
Notes
Equation: The workload cache-size contract gives bytes per token; token count and KV-width factor scale the footprint.
Assumptions: One request per cache estimate, fixed precision, and an explicit fractional reservation when using capacity methods.
Excludes: Runtime reserve, batch scheduler limits, and automatic reconciliation of the reservation with weight residency.
References
Derivation: The workload method get_kv_cache_size supplies the cache geometry and storage convention. The wrapper scales its one-token result by retained tokens and an explicit width multiplier; dividing the reserved device-memory budget by this footprint gives continuous token or request capacity.
Examples
>>> from mlsysim import Scenarios
>>> s = Scenarios.Gpt2XlKvFullWindow
>>> s.total.to("GB").magnitude > 0
TrueAttributes
| Name | Description |
|---|---|
| bytes_per_value | |
| hardware | |
| kv_budget | Decimal GB reserved for KV cache as the configured share of labeled device capacity. |
| kv_share | |
| kv_width_factor | |
| per_token | Per-request cache bytes per kept token, with the explicit KV-width multiplier applied. |
| requests | Continuous number of full-context requests fitting the reserved KV budget. |
| tokens | |
| tokens_in_budget | Continuous cache-token capacity in the reserved budget. |
| total | Per-request KV cache in decimal GB for the configured retained token count. |
| weights | Model weight footprint in decimal GB at the chosen storage width. |
| workload |
workload
Type: Workload.
Declared default: required.
tokens
Type: int.
Declared default: required.
bytes_per_value
Type: Quantity.
Declared default: required.
hardware
Type: Optional[HardwareNode].
Declared default: None.
kv_width_factor
Type: float.
Declared default: 1.0.
per_token
engine.scenario_types.KvFootprintScenario.per_token
Per-request cache bytes per kept token, with the explicit KV-width multiplier applied.
Equation: (workload.get_kv_cache_size(1, 1, bytes_per_value) * kv_width_factor).to('byte').
total
engine.scenario_types.KvFootprintScenario.total
Per-request KV cache in decimal GB for the configured retained token count.
Equation: (per_token * tokens).to('GB').
weights
engine.scenario_types.KvFootprintScenario.weights
Model weight footprint in decimal GB at the chosen storage width.
Equation: workload.size_in_bytes(bytes_per_value).to('GB').
kv_budget
engine.scenario_types.KvFootprintScenario.kv_budget
Decimal GB reserved for KV cache as the configured share of labeled device capacity.
Equation: (_labeled_capacity(hardware) * kv_share).to('GB').
tokens_in_budget
engine.scenario_types.KvFootprintScenario.tokens_in_budget
Continuous cache-token capacity in the reserved budget.
Equation: (kv_budget / per_token).to('').magnitude.
requests
engine.scenario_types.KvFootprintScenario.requests
Continuous number of full-context requests fitting the reserved KV budget.
Equation: (kv_budget / total).to('').magnitude.