Skip to main content

KvForkCopyScenario

Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.

Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.

Candidates forked from one prefix, each copying the prefix’s KV state without block sharing.

Notes

Equation: Copied bytes=branches*prefix_tokens*pool.result.kv_per_token.

Assumptions: Every candidate receives a separate full prefix copy.

Excludes: Copy-on-write blocks, transfer time, and generated suffix storage.

References

Derivation: Each independent branch stores prefix_tokens positions at the admission model’s bytes per position. Multiplying by the number of branches counts the copied state without any sharing.

Examples

The named registry scenario supplies the stated inputs; this is a modeled calculation, not a new measurement.

>>> from mlsysim import Scenarios
>>> scenario = Scenarios.ForkedCandidatesPrefixCopy
>>> round((scenario.copied).to('B').magnitude, 6)
10737418240

Attributes

Name Description
branches
copied KV bytes copied when each branch receives the complete prefix: branches * prefix_tokens * pool.result.kv_per_token; excludes block sharing.
pool
prefix_tokens

pool

Type: KvAdmissionScenario.

Declared default: required.

prefix_tokens

Type: int.

Declared default: required.

branches

Type: int.

Declared default: required.

copied

engine.scenario_types.KvForkCopyScenario.copied

KV bytes copied when each branch receives the complete prefix: branches * prefix_tokens * pool.result.kv_per_token; excludes block sharing.

Equation: self.branches * self.prefix_tokens * self.pool.result.kv_per_token

Back to top