HostDispatchScenario
Optimization scenarios · Implementation: mlsysim/engine/scenario_definitions/optimization.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
The host’s work to issue one eager step against device_time of accelerator work.
The step has blocks blocks of ops_per_block Python-level operations, each paying dispatch, and launches_per_block kernels per block (default: one per operation), each paying launch. The model is additive with no host-device overlap: the overhead ratio is host time over device time, and above 1 the host takes more than half of the step.
Notes
Equation: Host time is operation_count*dispatch + launch_count*launch; the additive step includes device_time.
Assumptions: Constant overhead per event and serial host/device timing.
Excludes: Asynchronous overlap, graph capture, and measured interpreter or driver behavior.
References
Derivation: Serial event accounting adds dispatch cost once per Python operation and launch cost once per kernel launch. Adding device work then gives the assumed serial step; removing the entire host term defines a ceiling rather than a predicted optimization gain.
Examples
>>> from mlsysim import Scenarios
>>> s = Scenarios.Gpt2XlEagerDecodeDispatch
>>> s.host_time == s.dispatch_time+s.launch_time
TrueAttributes
| Name | Description |
|---|---|
| blocks | |
| device_time | |
| dispatch | |
| dispatch_time | Total Python-side dispatch time in microseconds. |
| host_share | Host time as a share of the serial step, host plus device. |
| host_time | Serial dispatch-plus-launch time in microseconds. |
| launch | |
| launch_ratio | Kernel launches alone over device time. |
| launch_time | Total kernel-launch overhead in microseconds. |
| launches | Kernel launch count across blocks; defaults to one launch per operation. |
| launches_per_block | |
| operations | Count of Python-level operations across all blocks. |
| ops_per_block | |
| overhead_ratio | Host time over device time (the dispatch overhead ratio). |
| per_op_cost | Dispatch plus launch, for an operation that launches one kernel. |
| speedup_limit | Serial step over device time: the most removing all host work could gain. |
ops_per_block
Type: int.
Declared default: required.
blocks
Type: int.
Declared default: 1.
launches_per_block
Type: Optional[int].
Declared default: None.
dispatch
Type: Quantity.
Declared default: required.
launch
Type: Quantity.
Declared default: required.
device_time
Type: Quantity.
Declared default: required.
operations
engine.scenario_types.HostDispatchScenario.operations
Count of Python-level operations across all blocks.
Equation: blocks * ops_per_block.
launches
engine.scenario_types.HostDispatchScenario.launches
Kernel launch count across blocks; defaults to one launch per operation.
Equation: per = ops_per_block if launches_per_block is None else launches_per_block; blocks * per.
per_op_cost
engine.scenario_types.HostDispatchScenario.per_op_cost
Dispatch plus launch, for an operation that launches one kernel.
Equation: (dispatch + launch).to('microsecond').
dispatch_time
engine.scenario_types.HostDispatchScenario.dispatch_time
Total Python-side dispatch time in microseconds.
Equation: (operations * dispatch).to('microsecond').
launch_time
engine.scenario_types.HostDispatchScenario.launch_time
Total kernel-launch overhead in microseconds.
Equation: (launches * launch).to('microsecond').
host_time
engine.scenario_types.HostDispatchScenario.host_time
Serial dispatch-plus-launch time in microseconds.
Equation: (dispatch_time + launch_time).to('microsecond').
overhead_ratio
engine.scenario_types.HostDispatchScenario.overhead_ratio
Host time over device time (the dispatch overhead ratio).
Equation: (host_time / device_time).to('').magnitude.
launch_ratio
engine.scenario_types.HostDispatchScenario.launch_ratio
Kernel launches alone over device time.
Equation: (launch_time / device_time).to('').magnitude.
speedup_limit
engine.scenario_types.HostDispatchScenario.speedup_limit
Serial step over device time: the most removing all host work could gain.
Equation: ((host_time + device_time) / device_time).to('').magnitude.