FixedBatchScheduleScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Requests in one batch of fixed membership, one token per iteration.
Notes
Equation: Slot work=batch_width*max(output_lengths); continuous scheduling assigns each waiting request to the earliest free slot.
Assumptions: One token per iteration, fixed initial slot count, and arrival-ordered waiting requests; input lists are aligned and nonempty.
Excludes: Prefill costs and variable token times; the request-level comparison starts the entire waiting wave together without a second-wave width check.
References
Implementation references: mlsysim.physics.serving.calc_padding_waste_fraction. These helper contracts state the source assumptions; the scenario binds their inputs and derives the displayed ratios and budgets.
Examples
The named registry scenario supplies the stated inputs; this is a modeled calculation, not a new measurement.
>>> from mlsysim import Scenarios
>>> scenario = Scenarios.FourRequestDecodeBatch
>>> round(scenario.idle_share, 6)
0.375Attributes
| Name | Description |
|---|---|
| all_lengths | Initial and waiting output lengths concatenated in arrival order, in token iterations. |
| all_useful_iterations | Useful token iterations across both initial and waiting requests. |
| continuous_iterations | Completion iteration of the greedy continuous-batching schedule, taking its latest endpoint. |
| continuous_utilization | Fraction of available slot-iterations occupied by useful work under greedy slot reuse. |
| idle_share | Fraction of fixed-batch slot-iterations spent after a request completes; delegates to calc_padding_waste_fraction. |
| iteration_time | |
| max_output | Longest initial output in tokens, which sets the fixed batch’s completion iteration. |
| mean_output | Mean initial output length in tokens; requires a nonempty batch. |
| output_lengths | |
| prompt_lengths | |
| request_level_iterations | Completion iteration of the illustrated request-level schedule, taking the latest span endpoint. |
| request_level_utilization | Useful token work divided by initial batch width times the illustrated request-level horizon; the trace’s second-wave assumption can exceed that capacity. |
| requests_advanced | Initial request count plus one replacement opportunity for each slot ending before the longest initial output; not a complete queue simulation. |
| slot_iterations | Available decode slot-iterations until the longest initial request ends: batch width times longest output. |
| total_tokens | Prompt-plus-output token count for each paired request; zip requires aligned input lists to retain every request. |
| useful_iterations | Useful initial token iterations, summing actual output lengths. |
| waiting_lengths |
Methods
| Name | Description |
|---|---|
| continuous_spans | Start/end iteration pairs from greedy slot reuse: each waiting request enters the next free initial-batch slot. |
| request_level_spans | Start/end iteration pairs with all waiting requests starting after the initial batch; this illustrative trace does not enforce a second-wave slot limit. |
continuous_spans
engine.scenario_types.FixedBatchScheduleScenario.continuous_spans()Start/end iteration pairs from greedy slot reuse: each waiting request enters the next free initial-batch slot.
Equation: The calculation follows the stated schedule construction rule; the method body shows each intermediate term.
request_level_spans
engine.scenario_types.FixedBatchScheduleScenario.request_level_spans()Start/end iteration pairs with all waiting requests starting after the initial batch; this illustrative trace does not enforce a second-wave slot limit.
Equation: The calculation follows the stated schedule construction rule; the method body shows each intermediate term.
output_lengths
Type: List[int].
Declared default: required.
prompt_lengths
Type: Optional[List[int]].
Declared default: None.
iteration_time
Type: Optional[Quantity].
Declared default: None.
waiting_lengths
Type: Optional[List[int]].
Declared default: None.
total_tokens
engine.scenario_types.FixedBatchScheduleScenario.total_tokens
Prompt-plus-output token count for each paired request; zip requires aligned input lists to retain every request.
Equation: [p + o for p, o in zip(self.prompt_lengths, self.output_lengths)]
max_output
engine.scenario_types.FixedBatchScheduleScenario.max_output
Longest initial output in tokens, which sets the fixed batch’s completion iteration.
Equation: max(self.output_lengths)
mean_output
engine.scenario_types.FixedBatchScheduleScenario.mean_output
Mean initial output length in tokens; requires a nonempty batch.
Equation: sum(self.output_lengths) / len(self.output_lengths)
slot_iterations
engine.scenario_types.FixedBatchScheduleScenario.slot_iterations
Available decode slot-iterations until the longest initial request ends: batch width times longest output.
Equation: len(self.output_lengths) * self.max_output
useful_iterations
engine.scenario_types.FixedBatchScheduleScenario.useful_iterations
Useful initial token iterations, summing actual output lengths.
Equation: sum(self.output_lengths)
requests_advanced
engine.scenario_types.FixedBatchScheduleScenario.requests_advanced
Initial request count plus one replacement opportunity for each slot ending before the longest initial output; not a complete queue simulation.
Equation: len(self.output_lengths) + sum((1 for x in self.output_lengths if x < self.max_output))
all_lengths
engine.scenario_types.FixedBatchScheduleScenario.all_lengths
Initial and waiting output lengths concatenated in arrival order, in token iterations.
Equation: list(self.output_lengths) + list(self.waiting_lengths or [])
all_useful_iterations
engine.scenario_types.FixedBatchScheduleScenario.all_useful_iterations
Useful token iterations across both initial and waiting requests.
Equation: sum(self.all_lengths)
request_level_iterations
engine.scenario_types.FixedBatchScheduleScenario.request_level_iterations
Completion iteration of the illustrated request-level schedule, taking the latest span endpoint.
Equation: max((end for _, end in self.request_level_spans()))
continuous_iterations
engine.scenario_types.FixedBatchScheduleScenario.continuous_iterations
Completion iteration of the greedy continuous-batching schedule, taking its latest endpoint.
Equation: max((end for _, end in self.continuous_spans()))
request_level_utilization
engine.scenario_types.FixedBatchScheduleScenario.request_level_utilization
Useful token work divided by initial batch width times the illustrated request-level horizon; the trace’s second-wave assumption can exceed that capacity.
Equation: self.all_useful_iterations / (len(self.output_lengths) * self.request_level_iterations)
continuous_utilization
engine.scenario_types.FixedBatchScheduleScenario.continuous_utilization
Fraction of available slot-iterations occupied by useful work under greedy slot reuse.
Equation: self.all_useful_iterations / (len(self.output_lengths) * self.continuous_iterations)