KvDecodeBatchScenario

Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.

Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.

Batched decode on one accelerator when every sequence also reads its own KV cache.

Each step reads the weights once and each sequence’s cache of context_tokens: bytes W + B m_seq, time (W + B m_seq) / B_mem (a bandwidth floor), aggregate rate B / step. The rate approaches B_mem / m_seq as caches dominate. A TPOT target caps the batch at floor((t B_mem - W) / m_seq); memory caps it at floor((M - W) / m_seq). Prefill of prompt_tokens runs at prefill_efficiency of the peak_precision peak.

Notes

Equation: Each memory step reads W+B*KV_sequence bytes; rate is B/step_time; capacity and latency ceilings solve those inequalities for B.

Assumptions: Every sequence reads a fixed full-length cache, weights are read once per batch, and memory transfers use peak bandwidth.

Excludes: Decode compute, collectives, launch overhead, workspaces, and evolving sequence lengths; prefill uses only the registered per-token forward count.

References

Derivation: the scenario composes mlsysim.physics.transfer_time. Their contracts supply the underlying equations; property equations show the remaining unit conversions and accounting.

Examples

>>> from mlsysim import Scenarios
>>> scenario = Scenarios.Gpt2XlChatDecodeBatchesH100
>>> value = scenario.weights
>>> str(value.units)
'GB'

Attributes

Name	Description
bandwidth	Registered device memory bandwidth, without a sustained-efficiency adjustment.
batch_sizes
context_tokens
hardware
kv_bytes_per_value
kv_per_sequence	Single-request cache in decimal MB at the fixed full context length.
kv_per_token	Single-request KV-cache bytes per context token from the workload’s cache geometry.
max_batch_in_memory	Integer weight-plus-cache capacity ceiling; negative values indicate weights already
peak_precision
peak_rate	Registered FLOP/s at the requested precision key, with no efficiency adjustment.
prefill_efficiency
prefill_ops	Prompt FLOPs from the workload’s per-token forward count, without context-dependent
prefill_time	Compute-only prompt milliseconds at the declared fraction of registered peak FLOP/s.
prompt_tokens
rate_ceiling	Aggregate tokens per second approached when cache traffic dominates the memory step.
ridge	Peak compute divided by memory bandwidth in FLOP/byte.
weight_bytes_per_param
weights	Resident model weights in decimal GB at the selected bytes per parameter.
workload

Methods

Name	Description
cache_share	Fraction of modeled step bytes attributable to KV reads rather than weights.
max_batch_within	Integer batch ceiling from the TPOT memory budget; negative values indicate no feasible
step_bytes	Decimal GB read per decode step: weights once plus a full cache per active sequence.
step_time	Memory-transfer floor in milliseconds via `transfer_time`; compute and launch costs
token_rate	Aggregate tokens per second from one output token per sequence per modeled memory step.

cache_share

engine.scenario_types.KvDecodeBatchScenario.cache_share(batch_size)

Fraction of modeled step bytes attributable to KV reads rather than weights.

Equation: (batch_size * kv_per_sequence / step_bytes(batch_size)).to('').magnitude.

max_batch_within

engine.scenario_types.KvDecodeBatchScenario.max_batch_within(tpot)

Integer batch ceiling from the TPOT memory budget; negative values indicate no feasible positive batch.

Equation: _math.floor(((tpot * bandwidth - weights) / kv_per_sequence).to('').magnitude).

step_bytes

engine.scenario_types.KvDecodeBatchScenario.step_bytes(batch_size)

Decimal GB read per decode step: weights once plus a full cache per active sequence.

Equation: (weights + batch_size * kv_per_sequence).to('GB').

step_time

engine.scenario_types.KvDecodeBatchScenario.step_time(batch_size)

Memory-transfer floor in milliseconds via transfer_time; compute and launch costs are excluded.

Equation: transfer_time(step_bytes(batch_size), bandwidth).to('ms').

token_rate

engine.scenario_types.KvDecodeBatchScenario.token_rate(batch_size)

Aggregate tokens per second from one output token per sequence per modeled memory step.

Equation: (batch_size / step_time(batch_size)).to('1/second').