ServingConsolidationScenario
Optimization scenarios · Implementation: mlsysim/engine/scenario_definitions/optimization.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Small and large serving models on whole devices, exclusive against shared in pairs.
Each small model occupies small_footprint (weights, KV cache, and activations) of a device_memory budget; tenants small models share one device when load tests allow. Each large model takes gpus_per_large devices for tensor parallelism either way.
Notes
Equation: shared devices = floor(small models / tenants) + large-model devices; savings = exclusive minus shared devices.
Assumptions: Small-model count is divisible by tenants for complete accounting; tenant throughput and memory isolation are supplied assumptions.
Excludes: Partial tenant groups, interference, latency-tail guarantees, and workspace in the large-model weight-only floor.
References
Derivation: shared devices = floor(small models / tenants) + large-model devices; savings = exclusive minus shared devices. The individual properties state units and intermediate accounting; this class introduces no fitted coefficient.
Examples
The named scenario supplies illustrative inputs; the output is calculated.
>>> from mlsysim import Scenarios
>>> case = Scenarios.ServingConsolidationA100
>>> round(float(case.exclusive_unused), 3)
0.675Attributes
| Name | Description |
|---|---|
| annual_value | Annual USD value of released devices at the scenario’s fixed hourly rate. |
| bytes_per_param | |
| device_hour_rate | |
| device_memory | |
| exclusive_gpus | Total device count when every small model owns a device and large models retain |
| exclusive_unused | Unused fraction of a device reserved for one small model. |
| exclusive_utilization | Memory occupancy fraction for one small model, not a compute-utilization estimate. |
| gpus_per_large | |
| large_gpus | Devices reserved for all large models at their declared tensor-parallel degree. |
| large_model | |
| large_weights | Large-model weight memory, parameter count times stored bytes per parameter. |
| min_devices_large | Whole-device weight-residency floor, rounding weight bytes divided by device |
| n_large | |
| n_small | |
| partition_memory | |
| partition_unused | Unused fraction of a hardware partition containing one small model; requires |
| saved_gpus | Devices released by sharing, relative to exclusive allocation. |
| saved_share | Fraction of the exclusive device allocation released by sharing. |
| shared_footprint | Combined memory quantity for tenants copies of the small model footprint. |
| shared_gpus | Total device count with shared small models and unchanged large-model allocation. |
| shared_small_gpus | Device count for small models grouped into complete tenant sets; partial sets are |
| shared_unused | Unused fraction after packing the declared tenants; negative indicates a memory |
| shared_utilization | Memory occupancy fraction for the declared tenant group, not a throughput |
| small_footprint | |
| small_model | |
| tenants |
device_memory
Type: Quantity.
Declared default: required.
partition_memory
Type: Optional[Quantity].
Declared default: None.
small_model
Type: Workload.
Declared default: required.
small_footprint
Type: Quantity.
Declared default: required.
tenants
Type: int.
Declared default: required.
n_small
Type: int.
Declared default: required.
large_model
Type: Workload.
Declared default: required.
n_large
Type: int.
Declared default: required.
gpus_per_large
Type: int.
Declared default: required.
bytes_per_param
Type: Quantity.
Declared default: required.
device_hour_rate
Type: Quantity.
Declared default: required.
exclusive_unused
engine.scenario_types.ServingConsolidationScenario.exclusive_unused
Unused fraction of a device reserved for one small model.
Equation: return _ratio(device_memory - small_footprint, device_memory).
partition_unused
engine.scenario_types.ServingConsolidationScenario.partition_unused
Unused fraction of a hardware partition containing one small model; requires partition_memory.
Equation: return _ratio(partition_memory - small_footprint, partition_memory).
large_weights
engine.scenario_types.ServingConsolidationScenario.large_weights
Large-model weight memory, parameter count times stored bytes per parameter.
Equation: return large_model.parameters * bytes_per_param.
min_devices_large
engine.scenario_types.ServingConsolidationScenario.min_devices_large
Whole-device weight-residency floor, rounding weight bytes divided by device capacity upward.
Equation: return math.ceil(_ratio(large_weights, device_memory)).
large_gpus
engine.scenario_types.ServingConsolidationScenario.large_gpus
Devices reserved for all large models at their declared tensor-parallel degree.
Equation: return n_large * gpus_per_large.
exclusive_gpus
engine.scenario_types.ServingConsolidationScenario.exclusive_gpus
Total device count when every small model owns a device and large models retain their allocation.
Equation: return n_small + large_gpus.
saved_gpus
engine.scenario_types.ServingConsolidationScenario.saved_gpus
Devices released by sharing, relative to exclusive allocation.
Equation: return exclusive_gpus - shared_gpus.
exclusive_utilization
engine.scenario_types.ServingConsolidationScenario.exclusive_utilization
Memory occupancy fraction for one small model, not a compute-utilization estimate.
Equation: return _ratio(small_footprint, device_memory).
annual_value
engine.scenario_types.ServingConsolidationScenario.annual_value
Annual USD value of released devices at the scenario’s fixed hourly rate.
Equation: return saved_gpus * _usd_per_hour(device_hour_rate) * HOURS_PER_YEAR.