PreferenceLabelBudgetScenario
Statistics scenarios · Implementation: mlsysim/engine/scenario_definitions/statistics.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Human preference labels for a reward model: comparisons_low to comparisons_high at price_low to price_high per comparison.
Notes
Equation: Multiply comparisons by price for each endpoint and compare the resulting budgets.
Assumptions: The stated comparison-count and price endpoints define two planning cases.
Excludes: Annotator-quality variation, overhead, and a learning-performance guarantee.
References
Derivation: Multiply comparisons by price for each endpoint and compare the resulting budgets. Individual property contracts identify the exact accounting expression or reusable calculation. Registry entries supply the scenario inputs; this class does not turn them into measured outcomes.
Examples
Evaluate a named scenario using its declared inputs.
>>> from mlsysim import Scenarios
>>> scenario = Scenarios.RewardModelLabelBudget
>>> round(float(scenario.budget_low.magnitude), 4)
25000.0Attributes
| Name | Description |
|---|---|
| budget_high | USD for the upper comparison-count and per-comparison price assumptions. |
| budget_low | USD for the lower comparison-count and per-comparison price assumptions. |
| comparisons_high | |
| comparisons_low | |
| price_high | |
| price_low | |
| spread | Dimensionless upper-to-lower labeling-budget ratio. |
comparisons_low
Type: float.
Declared default: required.
comparisons_high
Type: float.
Declared default: required.
price_low
Type: Quantity.
Declared default: required.
price_high
Type: Quantity.
Declared default: required.
budget_low
engine.scenario_types.PreferenceLabelBudgetScenario.budget_low
USD for the lower comparison-count and per-comparison price assumptions.
Equation: (comparisons_low * price_low).to('USD').
budget_high
engine.scenario_types.PreferenceLabelBudgetScenario.budget_high
USD for the upper comparison-count and per-comparison price assumptions.
Equation: (comparisons_high * price_high).to('USD').
spread
engine.scenario_types.PreferenceLabelBudgetScenario.spread
Dimensionless upper-to-lower labeling-budget ratio.
Equation: (budget_high / budget_low).to('').magnitude.