PrefixSharingScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
KV cache for users who share one prompt prefix, stored per user or once.
Notes
Equation: Independent storage=users*(prefix+unique)*KV_bytes_per_token; shared storage=(prefix+users*unique)*KV_bytes_per_token.
Assumptions: Uniform KV width and token storage, one physically shared prefix, positive user count.
Excludes: Allocator fragmentation, copy-on-write suffix blocks, and prefix cache misses.
References
Implementation references: mlsysim.physics.memory.calc_kv_cache_size. These helper contracts state the source assumptions; the scenario binds their inputs and derives the displayed ratios and budgets.
Examples
The named registry scenario supplies the stated inputs; this is a modeled calculation, not a new measurement.
>>> from mlsysim import Scenarios
>>> scenario = Scenarios.ChatbotSharedSystemPrompt
>>> round(scenario.capacity_ratio, 6)
4.98008Attributes
| Name | Description |
|---|---|
| bytes_per_elem | |
| capacity_ratio | Dimensionless storage ratio kv_unshared / kv_shared; a capacity gain under identical requests, not measured throughput. |
| kv_shared | KV storage in bytes for one prefix and all suffixes; uses the same calc_kv_cache_size accounting as kv_unshared. |
| kv_unshared | KV storage in bytes for all independent contexts; _kv delegates to calc_kv_cache_size with the declared cache width. |
| kv_width | |
| prefill_tokens_skipped | Prefix tokens avoided after the first user has populated the cache: (users - 1) * prefix_tokens. |
| prefix_tokens | |
| reduction | Fraction of KV storage removed by prefix sharing: 1 - kv_shared / kv_unshared. |
| tokens_per_user | Context tokens for one user: shared prefix plus that user’s unique suffix. |
| tokens_shared | Total stored token positions with one prefix copy and one unique suffix per user. |
| tokens_unshared | Total stored token positions without sharing: users * tokens_per_user. |
| unique_tokens_per_user | |
| users | |
| workload |
workload
Type: Workload.
Declared default: required.
prefix_tokens
Type: int.
Declared default: required.
users
Type: int.
Declared default: required.
unique_tokens_per_user
Type: int.
Declared default: required.
kv_width
Type: int.
Declared default: required.
bytes_per_elem
Type: Quantity.
Declared default: required.
tokens_per_user
engine.scenario_types.PrefixSharingScenario.tokens_per_user
Context tokens for one user: shared prefix plus that user’s unique suffix.
Equation: self.prefix_tokens + self.unique_tokens_per_user
reduction
engine.scenario_types.PrefixSharingScenario.reduction
Fraction of KV storage removed by prefix sharing: 1 - kv_shared / kv_unshared.
Equation: 1 - (self.kv_shared / self.kv_unshared).to(’’).magnitude
capacity_ratio
engine.scenario_types.PrefixSharingScenario.capacity_ratio
Dimensionless storage ratio kv_unshared / kv_shared; a capacity gain under identical requests, not measured throughput.
Equation: (self.kv_unshared / self.kv_shared).to(’’).magnitude
prefill_tokens_skipped
engine.scenario_types.PrefixSharingScenario.prefill_tokens_skipped
Prefix tokens avoided after the first user has populated the cache: (users - 1) * prefix_tokens.
Equation: (self.users - 1) * self.prefix_tokens