PausedKvStateScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
A request paused on a tool: keep, move over the host links, or drop and recompute its KV state.
Notes
Equation: Host-link bandwidth=node.accelerators_per_node*accelerator.interconnect.bandwidth; pause choices delegate to PausedKvStateModel.
Assumptions: Independent host links and the cache geometry inherited from pool/prefill scenarios.
Excludes: PCIe contention beyond the supplied link fraction and measured reload behavior.
References
Implementation reference: mlsysim.solvers.PausedKvStateModel.solve documents the composed equation and its sources.
Examples
The named registry scenario supplies the stated inputs; this is a modeled calculation, not a new measurement.
>>> from mlsysim import Scenarios
>>> scenario = Scenarios.RepairToolWaitPausedState
>>> round((scenario.link).to('GB/s').magnitude, 6)
512.0Attributes
| Name | Description |
|---|---|
| block_tokens | |
| context_tokens | |
| link | Aggregate host-link bandwidth: accelerators per node times each accelerator’s interconnect bandwidth; assumes independent links and excludes contention. |
| link_fraction | |
| pool | |
| prefill | |
| result | Return mlsysim.solvers.PausedKvStateModel.solve for this scenario; output fields retain the units and feasibility semantics specified by that solver. |
| returns | |
| wait |
pool
Type: KvAdmissionScenario.
Declared default: required.
prefill
Type: PrefixRebuildScenario.
Declared default: required.
context_tokens
Type: int.
Declared default: required.
block_tokens
Type: int.
Declared default: required.
wait
Type: Quantity.
Declared default: required.
link_fraction
Type: float.
Declared default: required.
returns
Type: int.
Declared default: required.
link
engine.scenario_types.PausedKvStateScenario.link
Aggregate host-link bandwidth: accelerators per node times each accelerator’s interconnect bandwidth; assumes independent links and excludes contention.
Equation: The calculation follows the stated accounting rule; the method body shows each intermediate term.
result
engine.scenario_types.PausedKvStateScenario.result
Return mlsysim.solvers.PausedKvStateModel.solve for this scenario; output fields retain the units and feasibility semantics specified by that solver.
Equation: mlsysim.solvers.PausedKvStateModel.solve (scenario fields).