WeightReadScenario
Optimization scenarios · Implementation: mlsysim/engine/scenario_definitions/optimization.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Every weight of workload read once from hardware’s memory at bytes_per_weight.
The weight-streaming floor of one batch-one pass: per frame for a vision model, per token for a decoder. Unlike DecodeScenario it assumes no FLOP count, so it serves convolutional networks as well as decoders. energy_per_byte (optional) prices the reads.
Notes
Equation: read_time is the worked quantity; each property below states its equation and units.
Assumptions: Weights and optional full KV state stream once at the registered peak memory bandwidth.
Excludes: Compute, runtime overhead, cache reuse, interdevice communication, and serving-latency validation.
References
Derivation: The equations below follow the stated counts, geometric relations, or accounting definitions; scenario inputs are supplied assumptions or observations, not fitted predictions.
Examples
Execute the registered worked inputs; this is a modeled calculation.
>>> from mlsysim import Scenarios
>>> case = Scenarios.Llama2_7BTokenOnFlagshipPhone
>>> value = case.read_time
>>> print(f"{value:.3f~P}")
273.438 msAttributes
| Name | Description |
|---|---|
| bytes_per_weight | |
| context_tokens | |
| energy_per_byte | |
| hardware | |
| kv_bytes_per_value | |
| kv_cache_bytes | Batch-one KV storage in bytes; requires transformer cache geometry and context settings. |
| read_energy | Energy in joules for one weight read; requires energy_per_byte. |
| read_power | Average read power in watts under the stated energy-per-byte assumption. |
| read_time | Weight-only streaming lower bound in milliseconds. |
| separate_unpack_bytes | Bytes moved by separate unpacking, including packed read, unpacked write/read, and KV read. |
| separate_unpack_time | Streaming lower bound in milliseconds including the separate unpack pass. |
| token_bytes | Bytes read per token, all weights plus the complete resident KV cache. |
| token_rate | Reciprocal streaming time in tokens per second for one request. |
| token_time | Weight-plus-KV streaming lower bound in milliseconds per token. |
| unpacked_bytes_per_weight | |
| weight_bytes | Workload weight storage at the supplied precision, in byte units. |
| workload |
workload
Type: Workload.
Declared default: required.
hardware
Type: HardwareNode.
Declared default: required.
bytes_per_weight
Type: Quantity.
Declared default: required.
energy_per_byte
Type: Optional[Quantity].
Declared default: None.
weight_bytes
engine.scenario_types.WeightReadScenario.weight_bytes
Workload weight storage at the supplied precision, in byte units.
Equation: return self.workload.size_in_bytes(self.bytes_per_weight)
read_time
engine.scenario_types.WeightReadScenario.read_time
Weight-only streaming lower bound in milliseconds.
Equation: return (self.weight_bytes / self.hardware.memory.bandwidth).to(‘millisecond’)
read_energy
engine.scenario_types.WeightReadScenario.read_energy
Energy in joules for one weight read; requires energy_per_byte.
Equation: return (self.weight_bytes * self.energy_per_byte).to(‘joule’)
read_power
engine.scenario_types.WeightReadScenario.read_power
Average read power in watts under the stated energy-per-byte assumption.
Average power of the reads alone, sustained over one pass.
Equation: return (self.read_energy / self.read_time).to(‘watt’)
context_tokens
Type: Optional[int].
Declared default: None.
kv_bytes_per_value
Type: Optional[Quantity].
Declared default: None.
unpacked_bytes_per_weight
Type: Optional[Quantity].
Declared default: None.
kv_cache_bytes
engine.scenario_types.WeightReadScenario.kv_cache_bytes
Batch-one KV storage in bytes; requires transformer cache geometry and context settings.
Keys and values of context_tokens tokens at batch one.
Equation: return self.workload.get_kv_cache_size(seq_len=self.context_tokens, batch_size=1, precision=self.kv_bytes_per_value).to(‘byte’)
token_bytes
engine.scenario_types.WeightReadScenario.token_bytes
Bytes read per token, all weights plus the complete resident KV cache.
Bytes read per generated token: every weight and the whole KV cache.
Equation: return (self.weight_bytes + self.kv_cache_bytes).to(‘byte’)
token_time
engine.scenario_types.WeightReadScenario.token_time
Weight-plus-KV streaming lower bound in milliseconds per token.
Equation: return (self.token_bytes / self.hardware.memory.bandwidth).to(‘millisecond’)
token_rate
engine.scenario_types.WeightReadScenario.token_rate
Reciprocal streaming time in tokens per second for one request.
Equation: return (1 / self.token_time).to(‘1/second’)
separate_unpack_bytes
engine.scenario_types.WeightReadScenario.separate_unpack_bytes
Bytes moved by separate unpacking, including packed read, unpacked write/read, and KV read.
Bytes per token when a separate kernel unpacks the weights first: read the packed weights, write them unpacked, read them back for the matrix multiply, read the cache.
Equation: unpacked = self.workload.size_in_bytes(self.unpacked_bytes_per_weight) return (self.weight_bytes + 2 * unpacked + self.kv_cache_bytes).to(‘byte’)
separate_unpack_time
engine.scenario_types.WeightReadScenario.separate_unpack_time
Streaming lower bound in milliseconds including the separate unpack pass.
Equation: return (self.separate_unpack_bytes / self.hardware.memory.bandwidth).to(‘millisecond’)