PretrainingAmortizationScenario
Data scenarios · Implementation: mlsysim/engine/scenario_definitions/data.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
tasks tasks trained from scratch at scratch_compute each against one pretraining run of pretrain_compute plus finetune_compute per task, with the labels each path needs.
Notes
Equation: scratch_total is the worked quantity; each property below states its equation and units.
Assumptions: Pretraining is paid once, per-task costs are constant, and scratch compute exceeds fine-tuning compute for crossover.
Excludes: Quality differences, task heterogeneity, and a monetary conversion of accelerator time.
References
Derivation: The equations below follow the stated counts, geometric relations, or accounting definitions; scenario inputs are supplied assumptions or observations, not fitted predictions.
Examples
Execute the registered worked inputs; this is a modeled calculation.
>>> from mlsysim import Scenarios
>>> case = Scenarios.FoundationModelTenTasks
>>> value = case.scratch_total
>>> print(f"{value:.3f~P}")
10000.000 hAttributes
| Name | Description |
|---|---|
| crossover_tasks | Continuous task count at compute break-even; requires scratch cost above fine-tuning cost. |
| finetune_compute | |
| finetune_label_cost | Label cost for fine-tuning across every task. |
| finetune_labels | |
| foundation_total | One pretraining run plus per-task fine-tuning accelerator time. |
| label_cost_reduction | Dimensionless quantity comparing scratch-label cost with fine-tuning-label cost. |
| label_price | |
| marginal_compute_reduction | Dimensionless per-task scratch-to-fine-tuning compute ratio. |
| pretrain_compute | |
| scratch_compute | |
| scratch_label_cost | Label cost for scratch training across every task. |
| scratch_labels | |
| scratch_total | Total accelerator time when every task trains from scratch. |
| tasks |
pretrain_compute
Type: Quantity.
Declared default: required.
scratch_compute
Type: Quantity.
Declared default: required.
finetune_compute
Type: Quantity.
Declared default: required.
tasks
Type: int.
Declared default: required.
scratch_labels
Type: Quantity.
Declared default: required.
finetune_labels
Type: Quantity.
Declared default: required.
label_price
Type: Quantity.
Declared default: required.
scratch_total
engine.scenario_types.PretrainingAmortizationScenario.scratch_total
Total accelerator time when every task trains from scratch.
Equation: return self.scratch_compute * self.tasks
foundation_total
engine.scenario_types.PretrainingAmortizationScenario.foundation_total
One pretraining run plus per-task fine-tuning accelerator time.
Equation: return self.pretrain_compute + self.finetune_compute * self.tasks
scratch_label_cost
engine.scenario_types.PretrainingAmortizationScenario.scratch_label_cost
Label cost for scratch training across every task.
Equation: return self.scratch_labels * self.label_price * self.tasks
finetune_label_cost
engine.scenario_types.PretrainingAmortizationScenario.finetune_label_cost
Label cost for fine-tuning across every task.
Equation: return self.finetune_labels * self.label_price * self.tasks
label_cost_reduction
engine.scenario_types.PretrainingAmortizationScenario.label_cost_reduction
Dimensionless quantity comparing scratch-label cost with fine-tuning-label cost.
Equation: return self.scratch_label_cost / self.finetune_label_cost
marginal_compute_reduction
engine.scenario_types.PretrainingAmortizationScenario.marginal_compute_reduction
Dimensionless per-task scratch-to-fine-tuning compute ratio.
Equation: return (self.scratch_compute / self.finetune_compute).to(’’).magnitude
crossover_tasks
engine.scenario_types.PretrainingAmortizationScenario.crossover_tasks
Continuous task count at compute break-even; requires scratch cost above fine-tuning cost.
Tasks after which pretraining has paid for itself in compute.
Equation: return (self.pretrain_compute / (self.scratch_compute - self.finetune_compute)).to(’’).magnitude