ColdStartScenario
Serving scenarios · Implementation: mlsysim/engine/scenario_definitions/serving.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
A new replica: provision, fetch the weight artifact, load and warm up.
Notes
Equation: Start time=provisioning_time+artifact_bytes/fetch_bandwidth+load_warmup_time.
Assumptions: Sequential startup phases and constant available transfer bandwidth; peak replica counts are supplied.
Excludes: Overlap, network contention, and forecasting the predictive fleet size.
References
Derivation: Nonoverlapping startup phases add in elapsed time. The fetch phase is artifact bytes divided by available bandwidth. Replica saving is an arithmetic comparison of two supplied peak counts, not a predictive scaling model.
Examples
The named registry scenario supplies the stated inputs; this is a modeled calculation, not a new measurement.
>>> from mlsysim import Scenarios
>>> scenario = Scenarios.Llama70BReplicaColdStart
>>> round((scenario.cold_start_time).to('s').magnitude, 6)
400.0Attributes
| Name | Description |
|---|---|
| artifact | Model weight artifact size in decimal GB via workload.size_in_bytes at the declared storage width. |
| cold_start_time | Serial start delay in seconds: provisioning plus artifact fetch plus load/warmup; no overlap is modeled. |
| fetch_bandwidth | |
| fetch_time | Ideal transfer time in seconds: artifact bytes divided by available fetch bandwidth. |
| load_warmup_time | |
| peak_saving | Fraction of reactive peak replicas avoided by the supplied predictive peak count; the counts are inputs, not forecasts made here. |
| predictive_peak_replicas | |
| prewarm_lead | |
| provisioning_time | |
| reactive_peak_replicas | |
| weight_bytes_per_param | |
| workload |
workload
Type: Workload.
Declared default: required.
weight_bytes_per_param
Type: Quantity.
Declared default: required.
fetch_bandwidth
Type: Quantity.
Declared default: required.
provisioning_time
Type: Quantity.
Declared default: required.
load_warmup_time
Type: Quantity.
Declared default: required.
reactive_peak_replicas
Type: int.
Declared default: required.
predictive_peak_replicas
Type: int.
Declared default: required.
prewarm_lead
Type: Quantity.
Declared default: required.
artifact
engine.scenario_types.ColdStartScenario.artifact
Model weight artifact size in decimal GB via workload.size_in_bytes at the declared storage width.
Equation: self.workload.size_in_bytes(self.weight_bytes_per_param).to(‘GB’)
fetch_time
engine.scenario_types.ColdStartScenario.fetch_time
Ideal transfer time in seconds: artifact bytes divided by available fetch bandwidth.
Equation: (self.artifact / self.fetch_bandwidth).to(‘second’)
cold_start_time
engine.scenario_types.ColdStartScenario.cold_start_time
Serial start delay in seconds: provisioning plus artifact fetch plus load/warmup; no overlap is modeled.
Equation: (self.provisioning_time + self.fetch_time + self.load_warmup_time).to(‘second’)
peak_saving
engine.scenario_types.ColdStartScenario.peak_saving
Fraction of reactive peak replicas avoided by the supplied predictive peak count; the counts are inputs, not forecasts made here.
Equation: 1 - self.predictive_peak_replicas / self.reactive_peak_replicas