PerExampleGradientScenario
Training scenarios · Implementation: mlsysim/engine/scenario_definitions/training.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Naive DP-SGD keeps one gradient per example: bytes_per_param per parameter of each named model, batch_size copies per step.
Notes
Equation: Naive gradient storage is parameters*bytes_per_param per example and batch_size times that for a full batch.
Assumptions: Each example retains a separate dense full-parameter gradient at the selected precision.
Excludes: Ghost clipping, microbatching, optimizer state, activations, and a differential-privacy guarantee.
References
Derivation: Naive gradient storage is parameters*bytes_per_param per example and batch_size times that for a full batch. This is declared scenario accounting rather than a fitted empirical law.
Examples
>>> from mlsysim import Scenarios
>>> scenario = Scenarios.DpSgdPerExampleGradients
>>> value = scenario.per_batch(next(iter(scenario.models)))
>>> str(value.units)
'B'Attributes
| Name | Description |
|---|---|
| batch_size | |
| bytes_per_param | |
| models |
Methods
| Name | Description |
|---|---|
| per_batch | Naive per-example gradient bytes across the full batch; no streaming or clipping |
| per_example | Gradient bytes for named model parameters at the selected gradient precision. |
per_batch
engine.scenario_types.PerExampleGradientScenario.per_batch(name)Naive per-example gradient bytes across the full batch; no streaming or clipping optimization.
Equation: (per_example(name) * batch_size).to('byte').
per_example
engine.scenario_types.PerExampleGradientScenario.per_example(name)Gradient bytes for named model parameters at the selected gradient precision.
Equation: models[name].size_in_bytes(bytes_per_param).to('byte').
models
Type: Dict[str, Workload].
Declared default: required.
batch_size
Type: int.
Declared default: required.
bytes_per_param
Type: Quantity.
Declared default: required.