BatchWindowScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Requests arriving as a Poisson stream into a batching window of length window.
The window collects lambda W arrivals on average; a request waits half the window on average; the batch a request finds itself in is size-biased, 1 + lambda W on average, and its tail_quantile is 1 plus that quantile of the Poisson count.
Notes
Equation: The calculation members below define mean_arrivals, arrival_spread, empty_share, count_at_least, mean_wait, budget_share, seen_batch_mean, seen_batch_tail.
Assumptions: Arrivals follow a homogeneous Poisson process and positions in a fixed window are uniform.
Excludes: Worker backlog, service time, and arrival correlations.
References
Derivation: Requests arriving as a Poisson stream into a batching window of length window. Each member records the corresponding algebra or named physics function.
Examples
Evaluate the named configuration and display a modeled result.
>>> from mlsysim import Scenarios
>>> case = Scenarios.BatchWindow10ms
>>> value = case.mean_wait
>>> print(f"{value:.3g~}")
5 msAttributes
| Name | Description |
|---|---|
| arrival_rates | |
| budget_share | Dimensionless share of latency budget consumed by mean formation wait. |
| latency_budget | |
| mean_wait | Mean window-formation wait in ms under uniform arrival position, half the window. |
| tail_quantile | |
| window |
Methods
| Name | Description |
|---|---|
| arrival_spread | Poisson arrival-count standard deviation, square root of the mean count. |
| count_at_least | Poisson probability of at least the supplied count, via calc_poisson_tail. |
| empty_share | Probability of no arrivals, exponential of minus the mean count. |
| mean_arrivals | Expected arrivals in the window, arrival rate times duration, dimensionless count. |
| seen_batch_mean | Expected tagged-request batch size, the request itself plus expected additional arrivals. |
| seen_batch_tail | Tagged-request batch count threshold via calc_poisson_reserve under the stated tail quantile. |
arrival_spread
engine.scenario_types.BatchWindowScenario.arrival_spread(arrival_rate)Poisson arrival-count standard deviation, square root of the mean count.
Standard deviation of the count, sqrt(lambda W).
Equation: _math.sqrt(mean_arrivals(arrival_rate)).
count_at_least
engine.scenario_types.BatchWindowScenario.count_at_least(arrival_rate, at_least)Poisson probability of at least the supplied count, via calc_poisson_tail.
Calculation: mlsysim.physics.statistics.calc_poisson_tail.
mean_arrivals
engine.scenario_types.BatchWindowScenario.mean_arrivals(arrival_rate)Expected arrivals in the window, arrival rate times duration, dimensionless count.
Equation: (arrival_rate * window).to('').magnitude.
seen_batch_mean
engine.scenario_types.BatchWindowScenario.seen_batch_mean(arrival_rate)Expected tagged-request batch size, the request itself plus expected additional arrivals.
Equation: 1 + mean_arrivals(arrival_rate).
seen_batch_tail
engine.scenario_types.BatchWindowScenario.seen_batch_tail(arrival_rate)Tagged-request batch count threshold via calc_poisson_reserve under the stated tail quantile.
1 plus the tail_quantile of the Poisson count: the smallest s with P(N >= s) below 1 - q.
Calculation: mlsysim.physics.statistics.calc_poisson_reserve.
window
Type: Quantity.
Declared default: required.
arrival_rates
Type: List[Quantity].
Declared default: [].
latency_budget
Type: Optional[Quantity].
Declared default: None.
tail_quantile
Type: float.
Declared default: 0.99.
mean_wait
engine.scenario_types.BatchWindowScenario.mean_wait
Mean window-formation wait in ms under uniform arrival position, half the window.
Equation: (window / 2).to('ms').