SingleAcceleratorLlmServiceScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
An LLM chat service on one accelerator: latency, KV residency, request rate, and token cost.
ServingModel gives TTFT and the weight footprint. TPOT combines the weight and request-cache transfer time with the solver’s ITL residual beyond its memory floor; that residual includes compute dominance as well as framework overhead. The KV reserve over cache bytes per token sets resident requests. Request rate is the smaller of prefill admission and residency turnover (Little’s law at the batch-one request time). The request-time calculation uses full-context TTFT; the separate ttft property uses prompt-only TTFT. Token billing counts prompt and output tokens.
Notes
Equation: ServingModel supplies prompt/full-context timing; request rate is min(prefill_rate/prompt_tokens, resident_requests/request_time).
Assumptions: Full-context cache size remains fixed throughout modeled decode; cache reserve and external admission rate are supplied assumptions.
Excludes: Queueing and dynamic batching. The ITL-minus-memory-floor residual may include compute cost, and request_time uses full-context TTFT rather than the prompt-only ttft property.
References
Derivation: the scenario composes mlsysim.physics.serving.calc_littles_law_rate, mlsysim.physics.transfer_time, mlsysim.solvers.ServingModel. Their contracts supply the underlying equations; property equations show the remaining unit conversions and accounting.
Examples
>>> from mlsysim import Scenarios
>>> scenario = Scenarios.Llama3_8BInt4ChatServiceH100
>>> value = scenario.total_tokens
>>> round(float(value), 3)
1256.0Attributes
| Name | Description |
|---|---|
| bandwidth | Registered memory bandwidth, treated as peak rather than measured sustained throughput. |
| cost_per_million_tokens | Hourly device cost divided by hourly token volume, expressed as USD per million total |
| decode_time | Generated token count times fixed-context TPOT, in seconds. |
| decode_tokens | |
| dispatch_cost | Serving ITL minus its memory floor; this residual includes compute dominance, not solely |
| hardware | |
| kv_bytes_per_value | |
| kv_capacity_tokens | Rounded cache-token slots within the declared KV reserve, not total device memory. |
| kv_per_request | Single-request KV bytes at prompt plus generated length. |
| kv_per_token | Single-request KV bytes for one context token at the chosen cache precision. |
| kv_reserve | |
| precision | |
| prefill_efficiency | |
| prefill_rate | |
| prefill_request_rate | Requests per second supported by the supplied prompt-token admission rate. |
| price | |
| prompt_tokens | |
| remaining_memory | Decimal GB left after weights alone; excludes KV reserve and runtime workspace. |
| request_rate | Smaller of prompt admission and residency turnover limits, in requests per second. |
| request_time | Seconds using full-context serving TTFT plus generated-token decode time; differs from |
| residency_request_rate | Requests per second from resident population over request time via Little’s law. |
| resident_requests | Whole full-context requests fitting the rounded KV-token budget. |
| tokens_per_hour | Prompt and output tokens served in an hour at the modeled request rate. |
| total_tokens | Prompt plus generated token count used to size the full resident request cache. |
| tpot | One-request weight-plus-cache transfer time plus the serving residual, in time units. |
| tpot_target | |
| ttft | Prompt-only model-side first-token milliseconds from ServingModel.solve. |
| ttft_target | |
| weight_floor | Seconds to read the weights once at registered bandwidth via transfer_time. |
| weights | Resident weight bytes reported by the serving solver at the full request context. |
| workload |
workload
Type: Workload.
Declared default: required.
hardware
Type: HardwareNode.
Declared default: required.
precision
Type: str.
Declared default: required.
kv_bytes_per_value
Type: Quantity.
Declared default: required.
prompt_tokens
Type: int.
Declared default: required.
decode_tokens
Type: int.
Declared default: required.
prefill_efficiency
Type: float.
Declared default: required.
kv_reserve
Type: Quantity.
Declared default: required.
prefill_rate
Type: Quantity.
Declared default: required.
price
Type: Quantity.
Declared default: required.
ttft_target
Type: Optional[Quantity].
Declared default: None.
tpot_target
Type: Optional[Quantity].
Declared default: None.
total_tokens
engine.scenario_types.SingleAcceleratorLlmServiceScenario.total_tokens
Prompt plus generated token count used to size the full resident request cache.
Equation: prompt_tokens + decode_tokens.
ttft
engine.scenario_types.SingleAcceleratorLlmServiceScenario.ttft
Prompt-only model-side first-token milliseconds from ServingModel.solve.
Equation: _serve(prompt_tokens).ttft.to('ms').
weights
engine.scenario_types.SingleAcceleratorLlmServiceScenario.weights
Resident weight bytes reported by the serving solver at the full request context.
Equation: _serve(total_tokens).model_weights_size.
bandwidth
engine.scenario_types.SingleAcceleratorLlmServiceScenario.bandwidth
Registered memory bandwidth, treated as peak rather than measured sustained throughput.
Equation: hardware.memory.bandwidth.
kv_per_token
engine.scenario_types.SingleAcceleratorLlmServiceScenario.kv_per_token
Single-request KV bytes for one context token at the chosen cache precision.
Equation: workload.get_kv_cache_size(seq_len=1, batch_size=1, precision=kv_bytes_per_value).
kv_per_request
engine.scenario_types.SingleAcceleratorLlmServiceScenario.kv_per_request
Single-request KV bytes at prompt plus generated length.
Equation: workload.get_kv_cache_size(seq_len=total_tokens, batch_size=1, precision=kv_bytes_per_value).
weight_floor
engine.scenario_types.SingleAcceleratorLlmServiceScenario.weight_floor
Seconds to read the weights once at registered bandwidth via transfer_time.
Equation: transfer_time(weights, bandwidth).
dispatch_cost
engine.scenario_types.SingleAcceleratorLlmServiceScenario.dispatch_cost
Serving ITL minus its memory floor; this residual includes compute dominance, not solely dispatch overhead.
Equation: res.itl - transfer_time(res.total_memory_required, bandwidth).
tpot
engine.scenario_types.SingleAcceleratorLlmServiceScenario.tpot
One-request weight-plus-cache transfer time plus the serving residual, in time units.
Equation: transfer_time(weights + kv_per_request, bandwidth) + dispatch_cost.
decode_time
engine.scenario_types.SingleAcceleratorLlmServiceScenario.decode_time
Generated token count times fixed-context TPOT, in seconds.
Equation: (decode_tokens * tpot).to('second').
kv_capacity_tokens
engine.scenario_types.SingleAcceleratorLlmServiceScenario.kv_capacity_tokens
Rounded cache-token slots within the declared KV reserve, not total device memory.
Equation: int(round((kv_reserve / kv_per_token).to('').magnitude)).
resident_requests
engine.scenario_types.SingleAcceleratorLlmServiceScenario.resident_requests
Whole full-context requests fitting the rounded KV-token budget.
Equation: int(kv_capacity_tokens / total_tokens).
request_time
engine.scenario_types.SingleAcceleratorLlmServiceScenario.request_time
Seconds using full-context serving TTFT plus generated-token decode time; differs from prompt-only ttft.
Equation: (_serve(total_tokens).ttft + decode_time).to('second').
prefill_request_rate
engine.scenario_types.SingleAcceleratorLlmServiceScenario.prefill_request_rate
Requests per second supported by the supplied prompt-token admission rate.
Equation: (prefill_rate / prompt_tokens).to('1/second').
residency_request_rate
engine.scenario_types.SingleAcceleratorLlmServiceScenario.residency_request_rate
Requests per second from resident population over request time via Little’s law.
Equation: calc_littles_law_rate(resident_requests, request_time).to('1/second').
request_rate
engine.scenario_types.SingleAcceleratorLlmServiceScenario.request_rate
Smaller of prompt admission and residency turnover limits, in requests per second.
Equation: min(prefill_request_rate, residency_request_rate).
tokens_per_hour
engine.scenario_types.SingleAcceleratorLlmServiceScenario.tokens_per_hour
Prompt and output tokens served in an hour at the modeled request rate.
Equation: (request_rate * Q_(1, 'hour')).to('').magnitude * total_tokens.
cost_per_million_tokens
engine.scenario_types.SingleAcceleratorLlmServiceScenario.cost_per_million_tokens
Hourly device cost divided by hourly token volume, expressed as USD per million total tokens.
Equation: (price * Q_(1, 'hour') / (tokens_per_hour / 1000000.0)).to('dollar').
remaining_memory
engine.scenario_types.SingleAcceleratorLlmServiceScenario.remaining_memory
Decimal GB left after weights alone; excludes KV reserve and runtime workspace.
Equation: (hardware.memory.capacity - weights).to('GB').