FleetMfuScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Program goodput (MFU) of a training step on one node and on a fleet of fleet_devices.
Useful work per step is the model’s training arithmetic for the tokens in the step (6 x parameters x tokens); MFU divides it by every device’s peak over the measured step time. Their ratio is the scaling efficiency measured from the node.
Notes
Equation: MFU = useful training FLOPs / (device count * peak FLOP/s * step time); scaling ratio = fleet MFU / node MFU.
Assumptions: Supplied step times are comparable and training work follows the parameter-dominated helper without full recomputation.
Excludes: Predicting step latency, identifying causes of scaling loss, and correcting measurement uncertainty.
References
Derivation: mFU = useful training FLOPs / (device count * peak FLOP/s * step time); scaling ratio = fleet MFU / node MFU. The individual properties state units and intermediate accounting; this class introduces no fitted coefficient.
Examples
The named scenario supplies illustrative inputs; the output is calculated.
>>> from mlsysim import Scenarios
>>> case = Scenarios.Llama2_70BFleetMfu128
>>> round(float(case.node_useful_work.to('flop').magnitude), 3)
925680000000000.0Attributes
| Name | Description |
|---|---|
| extra_share | Extra step duration divided by fleet step duration, a signed fractional |
| extra_step_time | Fleet step duration minus node step duration, in milliseconds. |
| fleet_devices | |
| fleet_mfu | Useful fleet-step FLOPs divided by the fleet peak-work envelope. |
| fleet_peak_work | Fleet peak FLOPs over the supplied fleet step duration, the MFU denominator. |
| fleet_step_time | |
| fleet_tokens | |
| fleet_useful_work | Useful training FLOPs in one fleet step from the parameter-token training-work |
| hardware | |
| node_devices | |
| node_mfu | Useful node-step FLOPs divided by all node devices’ peak work over the observed |
| node_step_time | |
| node_tokens | |
| node_useful_work | Useful training FLOPs in one node step from the parameter-token training-work |
| scaling_efficiency | Fleet MFU divided by node MFU, a dimensionless per-device scaling ratio. |
| workload |
workload
Type: Workload.
Declared default: required.
hardware
Type: HardwareNode.
Declared default: required.
node_devices
Type: int.
Declared default: required.
fleet_devices
Type: int.
Declared default: required.
node_tokens
Type: int.
Declared default: required.
fleet_tokens
Type: int.
Declared default: required.
node_step_time
Type: Quantity.
Declared default: required.
fleet_step_time
Type: Quantity.
Declared default: required.
node_useful_work
engine.scenario_types.FleetMfuScenario.node_useful_work
Useful training FLOPs in one node step from the parameter-token training-work helper.
Equation: return _useful(node_tokens).
fleet_useful_work
engine.scenario_types.FleetMfuScenario.fleet_useful_work
Useful training FLOPs in one fleet step from the parameter-token training-work helper.
Equation: return _useful(fleet_tokens).
fleet_peak_work
engine.scenario_types.FleetMfuScenario.fleet_peak_work
Fleet peak FLOPs over the supplied fleet step duration, the MFU denominator.
Equation: return fleet_devices * hardware.compute.peak_flops * fleet_step_time.
node_mfu
engine.scenario_types.FleetMfuScenario.node_mfu
Useful node-step FLOPs divided by all node devices’ peak work over the observed step time.
Equation: peak = node_devices * hardware.compute.peak_flops * node_step_time; return (node_useful_work / peak).to('').magnitude.
fleet_mfu
engine.scenario_types.FleetMfuScenario.fleet_mfu
Useful fleet-step FLOPs divided by the fleet peak-work envelope.
Equation: return (fleet_useful_work / fleet_peak_work).to('').magnitude.
scaling_efficiency
engine.scenario_types.FleetMfuScenario.scaling_efficiency
Fleet MFU divided by node MFU, a dimensionless per-device scaling ratio.
Equation: return fleet_mfu / node_mfu.
extra_step_time
engine.scenario_types.FleetMfuScenario.extra_step_time
Fleet step duration minus node step duration, in milliseconds.
Equation: return (fleet_step_time - node_step_time).to('ms').