physics.serving

physics.serving

Inference serving and queueing models.

Functions

Name	Description
calc_batch_formation_wait	Calculate mean delay while a fixed-size batch fills.
calc_batch_knee	Calculate where variable batch work equals fixed work.
calc_batch_throughput_share	Calculate batch throughput as a fraction of its affine-model ceiling.
calc_exponential_quantile_factor	Calculate an exponential quantile relative to its mean.
calc_kingman_wait_ratio	Approximate mean G/G/1 wait relative to mean service time.
calc_littles_law_occupancy	Calculate mean occupancy from arrival rate and residence time.
calc_littles_law_rate	Calculate average turnover rate from occupied slots and residence time.
calc_mg1_wait_ratio	Calculate mean M/G/1 queue wait relative to mean service time.
calc_mm1_max_utilization	Calculate the utilization ceiling implied by an M/M/1 tail budget.
calc_mm1_time_in_system	Calculate mean M/M/1 residence time including service.
calc_padding_waste_fraction	Calculate idle slot share when a batch waits for its longest member.
calc_queue_latency_mmc	Calculate M/M/c utilization and queue-wait percentiles.
calc_residency_factor	Calculate how retaining state during a wait expands residence time.
calc_speculative_expected_tokens	Calculate expected emitted tokens per speculative round.
calc_speculative_round_speedup	Calculate a speculative-round decode speedup under fixed step costs.

calc_batch_formation_wait

physics.serving.calc_batch_formation_wait(batch_size, arrival_rate)

Calculate mean delay while a fixed-size batch fills.

Parameters

Name	Type	Description	Default
batch_size	int	Positive batch count B; no timeout or partial-batch release.	required
arrival_rate	Quantity or float	Positive stationary arrival rate lambda; output has reciprocal-rate units.	required

Returns

Name	Type	Description
	Quantity or float	Mean formation delay, excluding execution and downstream queueing.

Notes

Equation: Wform = (B-1)/(2 lambda).

Assumptions: Average interarrival gap is 1/lambda and each request position within a batch is equally represented.

Excludes: Arrival bursts correlated with batch position, timeouts, priorities, and execution delay.

References

Derivation: batch positions wait for B-1, B-2, …, 0 subsequent arrivals; averaging those counts and multiplying by the mean gap gives the formula.

Examples

>>> calc_batch_formation_wait(1, 2 / ureg.second) == 0 * ureg.second
True

calc_batch_knee

physics.serving.calc_batch_knee(fixed_time, per_request_time)

Calculate where variable batch work equals fixed work.

Parameters

Name	Type	Description	Default
fixed_time	Quantity	Nonnegative fixed duration per batch.	required
per_request_time	Quantity	Positive incremental duration per request; compatible time units.	required

Returns

Name	Type	Description
	float	Continuous batch-size knee; an integer scheduler may need rounding.

Notes

Equation: Bknee = t0/treq; throughput B/(t0+B treq) is half its asymptotic ceiling at this point.

Assumptions: Batch service duration is affine in batch size with fixed coefficients.

Excludes: Queueing, capacity limits, nonlinear kernels, and discrete batch-size rounding.

References

Derivation: equate the fixed term t0 and variable term B treq in the affine batch-service model and solve for B.

Examples

>>> calc_batch_knee(4 * ureg.ms, 2 * ureg.ms) == 2
True

calc_batch_throughput_share

physics.serving.calc_batch_throughput_share(
    batch_size,
    fixed_time,
    per_request_time,
)

Calculate batch throughput as a fraction of its affine-model ceiling.

Parameters

Name	Type	Description	Default
batch_size	int or float	Nonnegative request count B, with a positive total service duration.	required
fixed_time	Quantity	Nonnegative fixed per-batch duration t0.	required
per_request_time	Quantity	Positive incremental duration treq per request.	required

Returns

Name	Type	Description
	float	Dimensionless share of the ceiling throughput 1/treq.

Notes

Equation: share = B treq / (t0 + B treq).

Assumptions: The same affine service model holds across batch sizes.

Excludes: Queue formation, memory-capacity limits, and changes in per-request execution efficiency.

References

Derivation: divide B/(t0+B treq) by its large-batch limit 1/treq.

Examples

>>> calc_batch_throughput_share(2, 4 * ureg.ms, 2 * ureg.ms) == 0.5
True

calc_exponential_quantile_factor

physics.serving.calc_exponential_quantile_factor(quantile)

Calculate an exponential quantile relative to its mean.

Parameters

Name	Type	Description	Default
quantile	float	Cumulative probability q in [0, 1).	required

Returns

Name	Type	Description
	float	Dimensionless quantile-to-mean ratio.

Notes

Equation: kq = -log(1-q).

Assumptions: An exponential distribution with positive finite mean.

Excludes: Other service-time distributions and mixtures with a point mass at zero.

References

Derivation: invert F(t)=1-exp(-t/mean) at F(t)=q, then divide by mean.

Examples

>>> calc_exponential_quantile_factor(0) == 0
True
>>> calc_exponential_quantile_factor(0.99) > calc_exponential_quantile_factor(0.5)
True

calc_kingman_wait_ratio

physics.serving.calc_kingman_wait_ratio(utilization, arrival_scv, service_scv)

Approximate mean G/G/1 wait relative to mean service time.

Parameters

Name	Type	Description	Default
utilization	float	Busy fraction rho in [0, 1).	required
arrival_scv	float	Nonnegative squared coefficient of variation of interarrival gaps.	required
service_scv	float	Nonnegative squared coefficient of variation of service durations.	required

Returns

Name	Type	Description
	float	Dimensionless approximate mean-wait to mean-service ratio.

Notes

Equation: Wq/E[S] approximately ((Ca**2 + Cs**2)/2) rho/(1-rho).

Assumptions: A single first-come server with independent renewal arrivals and service durations; the approximation is motivated by heavy traffic.

Excludes: Tail percentiles, correlated service/arrivals, finite buffers, and multiple-server routing.

References

Kingman (1961), The Single Server Queue in Heavy Traffic. https://doi.org/10.1017/S0305004100036094 (bibliography key: kingman1961single).

Examples

>>> calc_kingman_wait_ratio(0.5, 1, 1) == calc_mg1_wait_ratio(0.5, 1)
True

calc_littles_law_occupancy

physics.serving.calc_littles_law_occupancy(arrival_rate, time_in_system)

Calculate mean occupancy from arrival rate and residence time.

Parameters

Name	Type	Description	Default
arrival_rate	Quantity or float	Nonnegative average arrivals per unit time; use units reciprocal to time_in_system.	required
time_in_system	Quantity or float	Nonnegative mean total residence duration, including waiting and service.	required

Returns

Name	Type	Description
	Quantity or float	Mean resident request count; units follow the product of the inputs.

Notes

Equation: L = lambda W.

Assumptions: A stable system and consistently defined boundaries for arrivals, occupancy, and residence.

Excludes: Tail occupancy, burst reserves, and admission guarantees.

References

Little (1961), A Proof for the Queuing Formula: L = lambda W. https://doi.org/10.1287/opre.9.3.383 (bibliography key: little1961).

Examples

>>> calc_littles_law_occupancy(2 / ureg.second, 3 * ureg.second).dimensionless
True

calc_littles_law_rate

physics.serving.calc_littles_law_rate(occupied_slots, residence_time)

Calculate average turnover rate from occupied slots and residence time.

Parameters

Name	Type	Description	Default
occupied_slots	Quantity or float	Nonnegative mean count L of occupied slots.	required
residence_time	Quantity or float	Positive mean duration W for which each turn holds a slot.	required

Returns

Name	Type	Description
	Quantity or float	Average turnover rate L/W, reciprocal to the residence-time unit.

Notes

Equation: lambda = L/W.

Assumptions: Stable operation and a common definition of a turn across slot and duration measurements.

Excludes: Peak throughput, tail delays, and idle capacity not represented by occupied_slots.

References

Little (1961), A Proof for the Queuing Formula: L = lambda W. https://doi.org/10.1287/opre.9.3.383 (bibliography key: little1961).

Examples

>>> calc_littles_law_rate(6, 3 * ureg.second) == 2 / ureg.second
True

calc_mg1_wait_ratio

physics.serving.calc_mg1_wait_ratio(utilization, service_cv)

Calculate mean M/G/1 queue wait relative to mean service time.

Parameters

Name	Type	Description	Default
utilization	float	Nonnegative busy fraction rho; values at or above one return infinity.	required
service_cv	float	Nonnegative dimensionless coefficient of variation of service duration.	required

Returns

Name	Type	Description
	float	Mean waiting time divided by mean service time; dimensionless.

Notes

Equation: Wq/E[S] = rho (1 + Cs**2) / [2 (1-rho)].

Assumptions: Poisson arrivals, one first-come server, independent service times with finite second moment.

Excludes: Tail percentiles, correlated service, finite queue capacity, and multiple servers.

References

Harchol-Balter (2013), Performance Modeling and Design of Computer Systems: Queueing Theory in Action. https://www.cs.cmu.edu/~harchol/PerformanceModeling/book.html (bibliography key: harchol-balter2013).

Examples

>>> calc_mg1_wait_ratio(0.5, 1) == 1
True
>>> calc_mg1_wait_ratio(0.5, 0) < calc_mg1_wait_ratio(0.5, 1)
True

calc_mm1_max_utilization

physics.serving.calc_mm1_max_utilization(service_time, tail_budget, tail_factor)

Calculate the utilization ceiling implied by an M/M/1 tail budget.

Parameters

Name	Type	Description	Default
service_time	Quantity	Positive mean service duration S.	required
tail_budget	Quantity	Positive allowed tail duration T, in compatible units.	required
tail_factor	float	Positive tail-to-mean multiplier k, normally from the exponential quantile.	required

Returns

Name	Type	Description
	float	Unclamped dimensionless ceiling; a negative result means even idle service misses the budget.

Notes

Equation: rho_max = 1 - k S/T.

Assumptions: M/M/1 response time is exponential, and the tail factor matches the chosen quantile.

Excludes: Admission enforcement, finite buffers, or nonexponential latency tails.

References

Derivation: rearrange k S/(1-rho) <= T for rho. The underlying M/M/1 mean is documented in calc_mm1_time_in_system.

Examples

>>> calc_mm1_max_utilization(2 * ureg.ms, 2 * ureg.ms, 1) == 0
True

calc_mm1_time_in_system

physics.serving.calc_mm1_time_in_system(service_time, utilization)

Calculate mean M/M/1 residence time including service.

Parameters

Name	Type	Description	Default
service_time	Quantity or float	Positive mean service duration S; output preserves its unit.	required
utilization	float	Dimensionless busy fraction rho in [0, 1).	required

Returns

Name	Type	Description
	Quantity or float	Mean total residence time, in the service-time unit.

Notes

Equation: W = S/(1-rho).

Assumptions: One first-come server, Poisson arrivals, independent exponential service, and steady state.

Excludes: Finite buffer loss, nonexponential service, and tail percentiles.

References

Examples

>>> calc_mm1_time_in_system(2 * ureg.ms, 0) == 2 * ureg.ms
True

calc_padding_waste_fraction

physics.serving.calc_padding_waste_fraction(lengths)

Calculate idle slot share when a batch waits for its longest member.

Parameters

Name	Type	Description	Default
lengths	iterable of float	Nonempty sequence of nonnegative work lengths in common units, with a positive maximum.	required

Returns

Name	Type	Description
	float	Dimensionless fraction of reserved slot-time unused.

Notes

Equation: waste = 1 - sum(L)/(n max(L)).

Assumptions: Each member reserves one slot until the longest member finishes, and length is proportional to slot occupancy.

Excludes: Continuous replacement, variable work per token, and non-slot resource bottlenecks.

References

Derivation: useful slot-time is sum(L), while fixed-membership reservation consumes n max(L). Divide the unused difference by the reservation.

Examples

>>> calc_padding_waste_fraction([2, 2]) == 0
True
>>> calc_padding_waste_fraction([1, 2]) > 0
True

calc_queue_latency_mmc

physics.serving.calc_queue_latency_mmc(
    arrival_rate_hz,
    service_rate_hz,
    num_servers,
)

Calculate M/M/c utilization and queue-wait percentiles.

Parameters

Name	Type	Description	Default
arrival_rate_hz	Quantity or float	Nonnegative request arrival rate lambda; bare numbers mean Hz.	required
service_rate_hz	Quantity or float	Positive completion rate mu per replica; bare numbers mean Hz.	required
num_servers	int	Replica count c; implementation truncates to an integer and clamps to at least one.	required

Returns

Name	Type	Description
	tuple	Dimensionless utilization, median queue wait, and p99 queue wait; waits are second quantities.

Notes

Equation: rho = lambda/(c mu); P(W > t) = ErlangC(c, rho) exp[-c mu (1-rho)t]. The returned waits invert this tail at tail probabilities 0.5 and 0.01.

Assumptions: Poisson arrivals, independent exponential service, identical replicas, and a shared first-come queue. Stable operation requires lambda < c mu.

Excludes: Service duration in the returned waits, batching, admission limits, heterogeneous replicas, and burst correlations.

The Erlang C sum is evaluated in the log domain to limit overflow. Numerical breakdown falls back to rho as a proxy waiting probability, which is an approximation. At saturation or zero service rate the function returns utilization one and infinite waits; these are not finite latency estimates.

References

Examples

>>> rho, median, tail = calc_queue_latency_mmc(1, 2, 1)
>>> rho == 0.5 and median.magnitude == 0 and tail.magnitude > 0
True

calc_residency_factor

physics.serving.calc_residency_factor(generation_time, wait_time)

Calculate how retaining state during a wait expands residence time.

Parameters

Name	Type	Description	Default
generation_time	Quantity	Positive active generation duration.	required
wait_time	Quantity	Nonnegative additional duration for which state retains its slot.	required

Returns

Name	Type	Description
	float	Dimensionless residence multiplier; fixed-slot turnover is divided by this factor.

Notes

Equation: factor = (Tgen + Twait)/Tgen.

Assumptions: Generation and waiting are sequential, and both reserve the same slot.

Excludes: Releasing state during waits, overlapping work, and different memory sizes across phases.

References

Derivation: compare total reserved duration Tgen+Twait with generation-only duration Tgen; Little’s law converts this to the inverse turnover ratio.

Examples

>>> calc_residency_factor(2 * ureg.second, 0 * ureg.second) == 1
True

calc_speculative_expected_tokens

physics.serving.calc_speculative_expected_tokens(acceptance, draft_length)

Calculate expected emitted tokens per speculative round.

Parameters

Name	Type	Description	Default
acceptance	float	Independent per-token acceptance probability alpha in [0, 1); the alpha=1 limit is not accepted by this implementation.	required
draft_length	int	Nonnegative number K of draft proposals; caller must supply an integer count.	required

Returns

Name	Type	Description
	float	Expected total emitted tokens, including the target bonus or correction token.

Notes

Equation: E[tokens] = sum(alpha**i, i=0..K) = (1-alpha**(K+1))/(1-alpha).

Assumptions: Identical independent acceptance probabilities; verification stops at the first rejected draft, and the target supplies one additional token.

Excludes: Context-dependent acceptance, verification compute, draft cost, and output quality.

References

Leviathan, Kalman, and Matias (2023), Fast Inference from Transformers via Speculative Decoding. https://proceedings.mlr.press/v202/leviathan23a.html (bibliography key: leviathan2023fast).

Examples

>>> calc_speculative_expected_tokens(0, 4) == 1
True
>>> calc_speculative_expected_tokens(0.5, 4) > calc_speculative_expected_tokens(0.5, 2)
True

calc_speculative_round_speedup

physics.serving.calc_speculative_round_speedup(
    accepted_per_round,
    draft_length,
    draft_cost_ratio,
)

Calculate a speculative-round decode speedup under fixed step costs.

Parameters

Name	Type	Description	Default
accepted_per_round	float	Nonnegative mean number m of accepted draft tokens, excluding the target token.	required
draft_length	int	Nonnegative number gamma of draft steps per round.	required
draft_cost_ratio	float	Nonnegative draft-step duration divided by one target-step duration.	required

Returns

Name	Type	Description
	float	Dimensionless time-per-token speedup against serial target-only decoding.

Notes

Equation: speedup = (m+1)/(gamma r + 1).

Assumptions: Draft steps execute serially and one target verification costs one target step regardless of gamma; one target token is always emitted.

Excludes: Verification compute growth, KV-cache memory, communication, and changing context or acceptance.

References

Leviathan, Kalman, and Matias (2023), Fast Inference from Transformers via Speculative Decoding. https://proceedings.mlr.press/v202/leviathan23a.html (bibliography key: leviathan2023fast).

Examples

>>> calc_speculative_round_speedup(0, 0, 0) == 1
True
>>> calc_speculative_round_speedup(1, 4, 1) < 1
True