MM1CapacityPlanScenario

Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.

Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.

Accelerators for peak_rate when each is an M/M/1 server whose tail must meet tail_budget.

The tail is tail_factor times the mean time in system S/(1 - rho), so the highest safe utilization is 1 - k S / T. The plan divides the peak by that utilization, rounds up to whole accelerators, then adds headroom. Optional: batched_rate re-plans with batched service at the same utilization; accelerator_price prices a plan per million requests.

Notes

Equation: The calculation members below define service_rate, time_in_system, tail_time, exact_tail_factor, max_utilization, required_rate, accelerators_exact, accelerators, accelerators_planned, utilization_after_failure, saturated_accelerators_exact, batched_accelerators, batched_accelerators_planned, cost_per_million.

Assumptions: Each replica has exponential service and Poisson arrivals; load balances evenly; the configured tail factor approximates the target quantile.

Excludes: Correlated bursts, heterogeneous replicas, batching-induced distribution changes, and startup delays.

References

Derivation: Accelerators for peak_rate when each is an M/M/1 server whose tail must meet tail_budget. Each member records the corresponding algebra or named physics function.

Examples

Evaluate the named configuration and display a modeled result.

>>> from mlsysim import Scenarios
>>> case = Scenarios.ResNet50FleetPlanH100
>>> value = case.accelerators_planned
>>> print(value)
36

Attributes

Name	Description
accelerator_price
accelerators	Integer accelerator count rounded up from the capacity requirement.
accelerators_exact	Fractional accelerator count before rounding, required rate divided by device capacity.
accelerators_planned	Integer planned fleet after applying the multiplicative headroom factor and rounding up.
batched_accelerators	Integer fleet count using batched throughput and the same utilization ceiling.
batched_accelerators_planned	Batched fleet count with multiplicative headroom and upward rounding.
batched_rate
exact_tail_factor	Exponential quantile multiplier via `calc_exponential_quantile_factor`.
example_utilization
headroom
max_utilization	Largest utilization meeting the tail target via `calc_mm1_max_utilization`; negative means even idle service misses it.
peak_rate
required_rate	Aggregate service rate in requests per second after utilization headroom.
saturated_accelerators_exact	Fractional fleet size at full utilization, ignoring queueing headroom.
service_rate	Per-accelerator service capacity in requests per second, reciprocal service time.
service_time
tail_budget
tail_factor
tail_quantile
utilization_after_failure	Dimensionless utilization after losing one planned accelerator; assumes equal load balancing.
utilization_points

Methods

Name	Description
cost_per_million	USD per million requests at peak load for the supplied accelerator count.
tail_time	Tail latency estimate in ms, configured tail factor times M/M/1 mean latency.
time_in_system	Mean request latency in ms via `calc_mm1_time_in_system` at the supplied utilization.

cost_per_million

engine.scenario_types.MM1CapacityPlanScenario.cost_per_million(accelerators)

USD per million requests at peak load for the supplied accelerator count.

Fleet cost per million requests served at peak_rate.

Equation: (accelerators * accelerator_price / peak_rate * 1000000.0).to('dollar').

tail_time

engine.scenario_types.MM1CapacityPlanScenario.tail_time(utilization)

Tail latency estimate in ms, configured tail factor times M/M/1 mean latency.

Equation: (tail_factor * time_in_system(utilization)).to('ms').

time_in_system

engine.scenario_types.MM1CapacityPlanScenario.time_in_system(utilization)

Mean request latency in ms via calc_mm1_time_in_system at the supplied utilization.

Calculation: mlsysim.physics.serving.calc_mm1_time_in_system.