TensorParallelSpeedupScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Stated per-layer compute split tp_size ways, plus the AllReduces each layer adds.
Notes
Equation: T_parallel_layer=sum(T_serial_phase)/tp_size+collectives_per_layer*T_allreduce.
Assumptions: Ideal equal compute partition and identical layers; all collective costs serialize with compute.
Excludes: Communication overlap, load imbalance, and capacity checks.
References
Derivation: The ideal partition assumption divides each serial compute phase by the tensor-parallel degree. Collectives add serial delays, so their count times per-collective time is added to the compute sum.
Examples
The named registry scenario supplies the stated inputs; this is a modeled calculation, not a new measurement.
>>> from mlsysim import Scenarios
>>> scenario = Scenarios.Llama2_70BPrefillTensorParallel8
>>> round(scenario.speedup, 6)
6.896552Attributes
| Name | Description |
|---|---|
| allreduce_time | |
| collectives_per_layer | |
| layers | |
| parallel_layer_time | Time in ms per parallel layer: ideally sharded compute plus every declared AllReduce, with no overlap. |
| parallel_total | Total modeled time in ms across layers identical compute-plus-collective layers. |
| serial_layer_time | Total compute time in ms across the declared serial phases of one layer. |
| serial_layer_times | |
| serial_total | Compute time in ms across layers identical serial layers. |
| sharded_compute_time | Ideal layer compute time in ms: serial phase sum divided by tp_size. |
| speedup | Dimensionless layer speedup: serial_layer_time / parallel_layer_time; communication can make it less than one. |
| tp_size |
Methods
| Name | Description |
|---|---|
| sharded | Compute time in ms for a named phase after ideal division by positive tp_size; excludes that phase’s communication. |
sharded
engine.scenario_types.TensorParallelSpeedupScenario.sharded(phase)Compute time in ms for a named phase after ideal division by positive tp_size; excludes that phase’s communication.
Equation: (self.serial_layer_times[phase] / self.tp_size).to(‘ms’)
serial_layer_times
Type: Dict[str, Quantity].
Declared default: required.
tp_size
Type: int.
Declared default: required.
allreduce_time
Type: Quantity.
Declared default: required.
collectives_per_layer
Type: int.
Declared default: 2.
layers
Type: int.
Declared default: required.
serial_layer_time
engine.scenario_types.TensorParallelSpeedupScenario.serial_layer_time
Total compute time in ms across the declared serial phases of one layer.
Equation: sum(self.serial_layer_times.values(), Q_(0, ‘ms’)).to(‘ms’)
sharded_compute_time
engine.scenario_types.TensorParallelSpeedupScenario.sharded_compute_time
Ideal layer compute time in ms: serial phase sum divided by tp_size.
Equation: (self.serial_layer_time / self.tp_size).to(‘ms’)
parallel_layer_time
engine.scenario_types.TensorParallelSpeedupScenario.parallel_layer_time
Time in ms per parallel layer: ideally sharded compute plus every declared AllReduce, with no overlap.
Equation: (sum((self.sharded(p) for p in self.serial_layer_times), Q_(0, ‘ms’)) + self.collectives_per_layer * self.allreduce_time).to(‘ms’)
speedup
engine.scenario_types.TensorParallelSpeedupScenario.speedup
Dimensionless layer speedup: serial_layer_time / parallel_layer_time; communication can make it less than one.
Equation: (self.serial_layer_time / self.parallel_layer_time).to(’’).magnitude
serial_total
engine.scenario_types.TensorParallelSpeedupScenario.serial_total
Compute time in ms across layers identical serial layers.
Equation: (self.serial_layer_time * self.layers).to(‘ms’)
parallel_total
engine.scenario_types.TensorParallelSpeedupScenario.parallel_total
Total modeled time in ms across layers identical compute-plus-collective layers.
Equation: (self.parallel_layer_time * self.layers).to(‘ms’)