SelfContainedReplicaScenario
Optimization scenarios · Implementation: mlsysim/engine/scenario_definitions/optimization.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
The smallest model replica of workload whose training state fits without sharding.
Tensor parallelism spans one node of fleet; the pipeline depth is the smallest power of two at which each accelerator’s share of the bytes_per_param training state fits its memory. An elastic job that keeps this layout fixed grows or shrinks one whole replica at a time.
Notes
Equation: pipeline depth is the first power of two satisfying state / (tensor degree * pipeline depth) <= device memory.
Assumptions: Training-state bytes per parameter are supplied; every shard receives an equal share and tensor parallelism spans one node.
Excludes: Activations, runtime workspace, communication buffers, and layout- performance ranking.
References
Derivation: pipeline depth is the first power of two satisfying state / (tensor degree * pipeline depth) <= device memory. The individual properties state units and intermediate accounting; this class introduces no fitted coefficient.
Examples
The named scenario supplies illustrative inputs; the output is calculated.
>>> from mlsysim import Scenarios
>>> case = Scenarios.Gpt3SelfContainedReplicaDgxH100
>>> round(float(case.tensor_parallel), 3)
8.0Attributes
| Name | Description |
|---|---|
| bytes_per_param | |
| fleet | |
| pipeline_depth | Smallest power-of-two pipeline depth whose equal training-state shards fit device |
| replica_gpus | Devices per complete replica: tensor-parallel degree times pipeline depth. |
| tensor_parallel | Tensor-parallel degree equal to the accelerator count in one node. |
| workload |
workload
Type: Workload.
Declared default: required.
fleet
Type: Fleet.
Declared default: required.
bytes_per_param
Type: Quantity.
Declared default: required.
tensor_parallel
engine.scenario_types.SelfContainedReplicaScenario.tensor_parallel
Tensor-parallel degree equal to the accelerator count in one node.
Equation: return fleet.node.accelerators_per_node.
pipeline_depth
engine.scenario_types.SelfContainedReplicaScenario.pipeline_depth
Smallest power-of-two pipeline depth whose equal training-state shards fit device memory.
Equation: Follow the pipeline_depth calculation below; branch conditions and rounding are part of its definition.
replica_gpus
engine.scenario_types.SelfContainedReplicaScenario.replica_gpus
Devices per complete replica: tensor-parallel degree times pipeline depth.
Equation: return tensor_parallel * pipeline_depth.