DecodeBudgetScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
A constructed decode-step budget, cut interval by interval, against a TPOT target.
Notes
Equation: Baseline intervals=baseline_step*shares; precision and fusion replace declared portions; weight floor=weight_shard/HBM_bandwidth.
Assumptions: Constructed ordered intervals: GEMM, attention, elementwise, AllReduce, launch; supplied cuts are scenario assumptions.
Excludes: Predicting the cuts from kernels, allocator/workspace costs, and calibrated execution time.
References
Derivation: This is a constructed interval budget: supplied fractional cuts remove portions of named baseline intervals, while supplied absolute costs replace other intervals. The independent weight-read floor follows bytes divided by peak bandwidth; it does not validate those cuts.
Examples
The named registry scenario supplies the stated inputs; this is a modeled calculation, not a new measurement.
>>> from mlsysim import Scenarios
>>> scenario = Scenarios.Llama3_70B_ReplicaDecodeBudget
>>> round((scenario.floor).to('ms').magnitude, 6)
5.223881Attributes
| Name | Description |
|---|---|
| accelerator | |
| after_fusion | Per-interval times in ms after the precision change, attention reduction, and replacement elementwise/replay costs; follows the fixed interval ordering. |
| after_precision | Per-interval times in ms after removing gemm_cut of the first baseline interval; other intervals remain unchanged. |
| allreduce_per_collective | Average constructed collective time in microseconds: baseline collective interval divided by calls per step. |
| allreduces_per_layer | |
| allreduces_per_step | Number of AllReduce calls per token step: allreduces_per_layer * workload.layers. |
| attention_cut | |
| baseline | Constructed per-interval times in ms, multiplying baseline_step by each declared share; these are supplied budget allocations, not measurements. |
| baseline_shares | |
| baseline_step | |
| collective_estimate | Launch-plus-transfer estimate in microseconds for two FP16 hidden-width vectors on NVLink; uses peak one-direction bandwidth and excludes topology contention. |
| collective_launch | |
| floor | Weight-streaming lower bound in ms: one weight shard divided by peak HBM bandwidth; excludes KV reads, compute, and collectives. |
| fused_elementwise | |
| gemm_cut | |
| intervals | |
| replay_overhead | |
| totals | Total step times in ms for baseline, precision, and fusion budgets, respectively, by summing their intervals. |
| tp_size | |
| tpot_target | |
| weight_bytes_per_param | |
| weight_shard | Per-accelerator weight storage in decimal GB: checkpoint size divided evenly by tp_size. |
| workload |
workload
Type: Workload.
Declared default: required.
tp_size
Type: int.
Declared default: required.
accelerator
Type: HardwareNode.
Declared default: required.
weight_bytes_per_param
Type: Quantity.
Declared default: required.
tpot_target
Type: Quantity.
Declared default: required.
baseline_step
Type: Quantity.
Declared default: required.
intervals
Type: List[str].
Declared default: required.
gemm_cut
Type: float.
Declared default: required.
attention_cut
Type: float.
Declared default: required.
fused_elementwise
Type: Quantity.
Declared default: required.
replay_overhead
Type: Quantity.
Declared default: required.
collective_launch
Type: Quantity.
Declared default: required.
allreduces_per_layer
Type: int.
Declared default: 2.
baseline
engine.scenario_types.DecodeBudgetScenario.baseline
Constructed per-interval times in ms, multiplying baseline_step by each declared share; these are supplied budget allocations, not measurements.
Equation: The calculation follows the stated accounting rule; the method body shows each intermediate term.
after_precision
engine.scenario_types.DecodeBudgetScenario.after_precision
Per-interval times in ms after removing gemm_cut of the first baseline interval; other intervals remain unchanged.
Equation: The calculation follows the stated accounting rule; the method body shows each intermediate term.
after_fusion
engine.scenario_types.DecodeBudgetScenario.after_fusion
Per-interval times in ms after the precision change, attention reduction, and replacement elementwise/replay costs; follows the fixed interval ordering.
Equation: The calculation follows the stated accounting rule; the method body shows each intermediate term.
totals
engine.scenario_types.DecodeBudgetScenario.totals
Total step times in ms for baseline, precision, and fusion budgets, respectively, by summing their intervals.
Equation: [sum(self.baseline), sum(self.after_precision), sum(self.after_fusion)]
weight_shard
engine.scenario_types.DecodeBudgetScenario.weight_shard
Per-accelerator weight storage in decimal GB: checkpoint size divided evenly by tp_size.
Equation: (self.workload.size_in_bytes(self.weight_bytes_per_param) / self.tp_size).to(‘GB’)
floor
engine.scenario_types.DecodeBudgetScenario.floor
Weight-streaming lower bound in ms: one weight shard divided by peak HBM bandwidth; excludes KV reads, compute, and collectives.
Equation: (self.weight_shard / self.accelerator.memory.bandwidth).to(‘ms’)
allreduces_per_step
engine.scenario_types.DecodeBudgetScenario.allreduces_per_step
Number of AllReduce calls per token step: allreduces_per_layer * workload.layers.
Equation: self.allreduces_per_layer * self.workload.layers
allreduce_per_collective
engine.scenario_types.DecodeBudgetScenario.allreduce_per_collective
Average constructed collective time in microseconds: baseline collective interval divided by calls per step.
Equation: Q_(self.baseline[3] / self.allreduces_per_step, ‘ms’).to(‘microsecond’)
collective_estimate
engine.scenario_types.DecodeBudgetScenario.collective_estimate
Launch-plus-transfer estimate in microseconds for two FP16 hidden-width vectors on NVLink; uses peak one-direction bandwidth and excludes topology contention.
Equation: The calculation follows the stated accounting rule; the method body shows each intermediate term.