SpeculativeRoundScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
A draft proposes draft_length tokens; the target verifies them in one step.
Notes
Equation: Time per token=(draft_cost+verify_time)/expected_tokens; expected_tokens is the geometric acceptance sum including a bonus token.
Assumptions: Constant independent acceptance probability and nonoverlapping draft/verify phases; provide total draft time or per-step draft time.
Excludes: Changing context lengths, model residency, and measuring acceptance quality.
References
Implementation references: mlsysim.physics.serving.calc_speculative_expected_tokens. These helper contracts state the source assumptions; the scenario binds their inputs and derives the displayed ratios and budgets.
Examples
The named registry scenario supplies the stated inputs; this is a modeled calculation, not a new measurement.
>>> from mlsysim import Scenarios
>>> scenario = Scenarios.Llama3_70B_ReplicaSpeculation
>>> round(scenario.expected_tokens, 6)
3.52182Attributes
| Name | Description |
|---|---|
| acceptance | |
| draft_cost | Total draft time in ms; use explicit draft_time when supplied, otherwise draft_length * draft_step. |
| draft_length | |
| draft_step | |
| draft_time | |
| expected_tokens | Expected emitted token count per round, including the bonus token, via calc_speculative_expected_tokens and its constant-acceptance assumption. |
| per_token_time | Amortized ms per emitted token: round_time / expected_tokens. |
| round_time | Serial draft-plus-verification time in ms; assumes the two phases do not overlap. |
| tpot_gain | Dimensionless ratio of the declared verification time to amortized speculative token time; depends on interpreting that verification input as the baseline step cost. |
| verify_time |
acceptance
Type: float.
Declared default: required.
draft_length
Type: int.
Declared default: required.
draft_time
Type: Optional[Quantity].
Declared default: None.
verify_time
Type: Quantity.
Declared default: required.
draft_step
Type: Optional[Quantity].
Declared default: None.
draft_cost
engine.scenario_types.SpeculativeRoundScenario.draft_cost
Total draft time in ms; use explicit draft_time when supplied, otherwise draft_length * draft_step.
Equation: The calculation follows the stated accounting rule; the method body shows each intermediate term.
expected_tokens
engine.scenario_types.SpeculativeRoundScenario.expected_tokens
Expected emitted token count per round, including the bonus token, via calc_speculative_expected_tokens and its constant-acceptance assumption.
Equation: The calculation follows the stated accounting rule; the method body shows each intermediate term.
round_time
engine.scenario_types.SpeculativeRoundScenario.round_time
Serial draft-plus-verification time in ms; assumes the two phases do not overlap.
Equation: (self.draft_cost + self.verify_time).to(‘ms’)
per_token_time
engine.scenario_types.SpeculativeRoundScenario.per_token_time
Amortized ms per emitted token: round_time / expected_tokens.
Equation: (self.round_time / self.expected_tokens).to(‘ms’)
tpot_gain
engine.scenario_types.SpeculativeRoundScenario.tpot_gain
Dimensionless ratio of the declared verification time to amortized speculative token time; depends on interpreting that verification input as the baseline step cost.
Equation: (self.verify_time / self.per_token_time).to(’’).magnitude