SyncStrategyScenario
Communication scenarios · Implementation: mlsysim/engine/scenario_definitions/communication.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Data-parallel training to a fixed target under one synchronization strategy.
Iterations to the target and the communication share are stated inputs: each iteration costs the one-worker iteration time times 1 + overhead, so speedup over one worker is base / (iterations x (1 + overhead)).
Notes
Equation: Use the dimensional products, ratios, or selection rules documented on the calculated properties below.
Assumptions: Iterations to target are supplied for each strategy, not predicted; communication stretches a fixed base iteration duration.
Excludes: The inverse-square-root staleness learning-rate factor is an illustrative policy, not a convergence theorem.
References
Derivation: the calculated properties below state their defining arithmetic and units. They are scenario accounting or input-selection rules, not independently measured relationships.
Examples
>>> from mlsysim import Scenarios
>>> case = Scenarios.TransformerBsp8Workers
>>> round(case.global_batch, 4)
256Attributes
| Name | Description |
|---|---|
| base_iteration_time | |
| base_iterations | |
| communication_overhead | |
| cost | Rental bill for the modeled run. Equation: iterations times iteration duration times |
| device_hour_rate | |
| global_batch | Samples processed in one global iteration. Equation: workers * per_worker_batch. |
| ideal_iterations | Iteration count under perfect sample-efficiency scaling. Equation: base_iterations / | | [iteration_time](#mlsysim.engine.scenario_types.SyncStrategyScenario.iteration_time) | Per-iteration time after the stated communication overhead. Equation: | | [iterations](#mlsysim.engine.scenario_types.SyncStrategyScenario.iterations) | | | [per_worker_batch](#mlsysim.engine.scenario_types.SyncStrategyScenario.per_worker_batch) | | | [sample_efficiency](#mlsysim.engine.scenario_types.SyncStrategyScenario.sample_efficiency) | Ideal iteration count divided by actual target-reaching iteration count. Equation: | | [speedup](#mlsysim.engine.scenario_types.SyncStrategyScenario.speedup) | Time-to-target speedup against the single-worker baseline. Equation:base_iterations / |
| stale_lr_factor | Illustrative learning-rate reduction for stale gradients. Equation: ``1 / sqrt(1 + |
| staleness | |
| workers |
workers
Type: int.
Declared default: required.
per_worker_batch
Type: int.
Declared default: required.
iterations
Type: int.
Declared default: required.
communication_overhead
Type: float.
Declared default: required.
base_iterations
Type: int.
Declared default: required.
base_iteration_time
Type: Quantity.
Declared default: required.
device_hour_rate
Type: Quantity.
Declared default: required.
staleness
Type: int.
Declared default: 0.
global_batch
engine.scenario_types.SyncStrategyScenario.global_batch
Samples processed in one global iteration. Equation: workers * per_worker_batch. Units: samples; every worker contributes a full local batch.
ideal_iterations
engine.scenario_types.SyncStrategyScenario.ideal_iterations
Iteration count under perfect sample-efficiency scaling. Equation: base_iterations / workers. Units: iterations; no quality degradation is assumed for this reference.
iteration_time
engine.scenario_types.SyncStrategyScenario.iteration_time
Per-iteration time after the stated communication overhead. Equation: base_iteration_time * (1 + communication_overhead). Units: time; overhead is relative to base iteration time.
speedup
engine.scenario_types.SyncStrategyScenario.speedup
Time-to-target speedup against the single-worker baseline. Equation: base_iterations / (iterations * (1 + communication_overhead)). Units: dimensionless; both runs use the same base iteration duration.
sample_efficiency
engine.scenario_types.SyncStrategyScenario.sample_efficiency
Ideal iteration count divided by actual target-reaching iteration count. Equation: ideal_iterations / iterations. Units: dimensionless; target-reaching iterations are supplied, not predicted.
cost
engine.scenario_types.SyncStrategyScenario.cost
Rental bill for the modeled run. Equation: iterations times iteration duration times worker count times device-hour price. Units: USD returned as a scalar; workers are billed throughout the run.
stale_lr_factor
engine.scenario_types.SyncStrategyScenario.stale_lr_factor
Illustrative learning-rate reduction for stale gradients. Equation: 1 / sqrt(1 + staleness). Units: dimensionless; this prescribed heuristic is not a convergence guarantee.