Skip to main content

Serving Scenarios

Inference work, request scheduling, and token-cache capacity.

All domains · Classes A–Z

Source locations

  • Inputs and calculations: mlsysim/engine/scenario_definitions/serving.py.
  • Named configurations: mlsysim/engine/scenario_catalog/serving/.
  • Result schemas: mlsysim/engine/result_types/serving.py.

Public imports and existing class reference URLs remain stable.

Scenario contracts

  • BatchCapChoiceScenario: Two batch caps at one arrival rate, batches waiting until full: throughput against formation wait.
  • BatchDispatchScenario: A dual-trigger batch: it may leave at timeout after its oldest arrival, then waits for a worker.
  • BatchEnergyPerImageScenario: Bytes, arithmetic intensity, and energy per image when a batch shares one weight read.
  • BatchWindowScenario: Requests arriving as a Poisson stream into a batching window of length window.
  • BatchingKneeScenario: Batch service time S(B) = t_0 + B t_req: where throughput turns over.
  • ChunkedPrefillScenario: A long prompt advanced chunk_tokens per iteration beside decode.
  • ColdStartScenario: A new replica: provision, fetch the weight artifact, load and warm up.
  • CompilationPaybackScenario: A compiled decode step and the executions that repay its build.
  • ContextGrowthScenario: A context window grown from base_tokens to tokens: a kept cache grows with the ratio, attention’s score work and stored scores with its square.
  • ContextMemoryCrossingScenario: Three memories of workload as context grows on hardware: fixed weights, a KV cache linear in tokens, and one layer’s stored scores (all heads) quadratic in tokens.
  • DecodeBudgetScenario: A constructed decode-step budget, cut interval by interval, against a TPOT target.
  • DecodeScenario: One decode step of workload on hardware (weight-streaming bound).
  • EmbeddingGatherScenario: One recommender request on hardware: one row of width values per categorical feature of dataset is gathered, then the rows and the bottom MLP’s output interact by pairwise dot products.
  • EmbeddingTableScenario: tables embedding tables of rows rows, each width values wide, against hardware.
  • FixedBatchScheduleScenario: Requests in one batch of fixed membership, one token per iteration.
  • FleetMfuScenario: Program goodput (MFU) of a training step on one node and on a fleet of fleet_devices.
  • GoodputScenario: ML productivity goodput of a job on a pool: scheduling, runtime, and program goodput.
  • KvAdmissionScenario: A replica’s KV budget against one context length (KvAdmissionModel).
  • KvAttentionDesignScenario: Bytes per token across registry models, with one MHA, one GQA, and one MLA reference.
  • KvBlockSharingScenario: Requests with an output allowance in whole blocks, with and without a shared prefix, as a turn rate.
  • KvDecodeBatchScenario: Batched decode on one accelerator when every sequence also reads its own KV cache.
  • KvFootprintScenario: workload’s KV cache for tokens kept tokens; with hardware and kv_share, how many such requests the cache’s share of the labeled device memory holds. kv_width_factor scales the key-value width (one half: half the key-value heads).
  • KvForkCopyScenario: Candidates forked from one prefix, each copying the prefix’s KV state without block sharing.
  • KvResidencyArchetypeScenario: One agent workload’s sharing gain, residency tax, and reload against recompute.
  • LittlesLawScenario: Requests in flight at arrival_rate, and replicas sized from a batch service time.
  • MM1CapacityPlanScenario: Accelerators for peak_rate when each is an M/M/1 server whose tail must meet tail_budget.
  • MM1ServiceSpeedupScenario: The same traffic on an M/M/1 server before and after its service time falls.
  • MaterializedScoresScenario: Attention scores stored in full: seq_len^2 per head, heads per layer, layers layers.
  • NoisyNeighborScenario: Tenants sharing a KV pool and a replica, one of them bursting, with and without quotas.
  • PackedQkvProjectionScenario: tokens rows through workload’s packed query-key-value projection, one d x 3d matrix.
  • PagedAllocationScenario: Requests under a length cap in pool’s KV pool: full-cap reservation against paged blocks.
  • PausedKvStateScenario: A request paused on a tool: keep, move over the host links, or drop and recompute its KV state.
  • PrecisionDividendScenario: Maximum batch before and after lower-precision weights and KV (PrecisionDividendModel).
  • PrecomputeVersusOnDemandScenario: items inputs answered one at a time on demand, or precomputed at a large-batch offline_rate.
  • PrefixRebuildScenario: A cached prefix on node: the prefill a miss rebuilds against one short suffix pass on a hit.
  • PrefixSharingScenario: KV cache for users who share one prompt prefix, stored per user or once.
  • QueueWaitScenario: A fleet modeled as one M/G/1 server at offered load utilization.
  • ReasoningLatencyScenario: Serial reasoning tokens before the answer, at a fixed time per token.
  • ReplicaStartupScenario: A serving replica’s startup phases; warm_path and cold_path name the phases each start pays.
  • RequestZonesScenario: One serialized request laid out in zones, the leading ones an invariant shared prefix.
  • RuntimeGoodputScenario: Runtime goodput of a job on failures.n_devices devices over a held window.
  • SchedulingSnapshotScenario: One snapshot of a shared pool’s in-service capacity, read at the scheduling boundary.
  • ServingScenario: Serving workload on hardware at a stated context and batch (ServingModel).
  • ServingTaxScenario: A request’s latency target split between model execution and everything outside it.
  • SingleAcceleratorLlmServiceScenario: An LLM chat service on one accelerator: latency, KV residency, request rate, and token cost.
  • SpeculativeRoundScenario: A draft proposes draft_length tokens; the target verifies them in one step.
  • TailEventRateScenario: How often an event of probability event_probability per request occurs at request_rate.
  • TensorParallelSpeedupScenario: Stated per-layer compute split tp_size ways, plus the AllReduces each layer adds.
  • TraceVolumeScenario: A profiler trace of window on every one of devices devices.
  • TrafficSurgeScenario: A service sized for normal_rate that receives surge_rate; latency in each regime.
  • TransformerBlockScenario: workload’s decoder blocks at a context of context_tokens: the classic block’s weights (4 d^2 attention projections, 2 x ffn_multiple d^2 MLP), its dense and attention multiply-adds, and, with hardware, the attention arithmetic of one prompt at peak.
  • WeightedRoutingScenario: Traffic split over server pools in proportion to each pool’s per-server capacity.
Back to top