BatchVersusStreamingIngestScenario
Data scenarios · Implementation: mlsysim/engine/scenario_definitions/data.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
One event stream ingested by always-on streaming cores or by a batch job each hour.
Streaming runs stream_cores (redundancy_factor cores per primary) for stream_hours_per_day; batch runs batch_cores for batch_window of each clock hour. Both pay core_hour_price.
Notes
Equation: daily bill = occupied core-hours * price; per-core throughput = required aggregate bandwidth / productive cores.
Assumptions: A batch drains one hour of arrivals inside each batch window; streaming redundancy is already included in reserved core count.
Excludes: Traffic bursts, queue latency, storage I/O bottlenecks, and scheduling overhead.
References
Derivation: daily bill = occupied core-hours * price; per-core throughput = required aggregate bandwidth / productive cores. The individual properties state units and intermediate accounting; this class introduces no fitted coefficient.
Examples
The named scenario supplies illustrative inputs; the output is calculated.
>>> from mlsysim import Scenarios
>>> case = Scenarios.MillionEventBatchVersusStreaming
>>> round(float(case.throughput.to('GB/s').magnitude), 3)
1.0Attributes
| Name | Description |
|---|---|
| batch_aggregate | GB/s needed to drain one hour’s arrivals inside the batch window. |
| batch_core_hours_per_day | Daily batch core-hours when one batch runs in every clock hour. |
| batch_cores | |
| batch_cost_per_day | Daily batch bill in USD from its occupied core-hours. |
| batch_per_core | Batch decode rate in MB/s per core under equal work partitioning. |
| batch_window | |
| core_hour_price | |
| cost_ratio | Streaming bill divided by batch bill, a dimensionless cost ratio. |
| event_rate | |
| event_size | |
| hourly_data | Data arriving in one clock hour, expressed in TB. |
| redundancy_factor | |
| stream_core_hours_per_day | Reserved streaming core-hours per day, including redundant cores already in the |
| stream_cores | |
| stream_cost_per_day | Daily streaming bill in USD from reserved core-hours and price. |
| stream_hours_per_day | |
| stream_per_core | Incoming MB/s per primary streaming core, assuming equal traffic partitioning. |
| stream_primary_cores | Primary streaming cores after integer division by the redundancy factor. |
| throughput | Incoming data rate in GB/s, event rate times bytes per event. |
event_rate
Type: Quantity.
Declared default: required.
event_size
Type: Quantity.
Declared default: required.
stream_cores
Type: int.
Declared default: required.
stream_hours_per_day
Type: Quantity.
Declared default: required.
batch_cores
Type: int.
Declared default: required.
batch_window
Type: Quantity.
Declared default: required.
core_hour_price
Type: Quantity.
Declared default: required.
redundancy_factor
Type: int.
Declared default: 1.
throughput
engine.scenario_types.BatchVersusStreamingIngestScenario.throughput
Incoming data rate in GB/s, event rate times bytes per event.
Equation: return (event_rate * event_size).to('GB/s').
hourly_data
engine.scenario_types.BatchVersusStreamingIngestScenario.hourly_data
Data arriving in one clock hour, expressed in TB.
Equation: return (throughput * Q_(1, 'hour')).to('TB').
stream_core_hours_per_day
engine.scenario_types.BatchVersusStreamingIngestScenario.stream_core_hours_per_day
Reserved streaming core-hours per day, including redundant cores already in the count.
Equation: return stream_cores * stream_hours_per_day.to('hour').magnitude.
stream_cost_per_day
engine.scenario_types.BatchVersusStreamingIngestScenario.stream_cost_per_day
Daily streaming bill in USD from reserved core-hours and price.
Equation: return (stream_core_hours_per_day * Q_(1, 'hour') * core_hour_price).to('USD').
batch_core_hours_per_day
engine.scenario_types.BatchVersusStreamingIngestScenario.batch_core_hours_per_day
Daily batch core-hours when one batch runs in every clock hour.
Equation: hours_per_day = Q_(1, 'day').to('hour').magnitude; return batch_cores * batch_window.to('hour').magnitude * hours_per_day.
batch_cost_per_day
engine.scenario_types.BatchVersusStreamingIngestScenario.batch_cost_per_day
Daily batch bill in USD from its occupied core-hours.
Equation: return (batch_core_hours_per_day * Q_(1, 'hour') * core_hour_price).to('USD').
cost_ratio
engine.scenario_types.BatchVersusStreamingIngestScenario.cost_ratio
Streaming bill divided by batch bill, a dimensionless cost ratio.
Equation: return (stream_cost_per_day / batch_cost_per_day).to('').magnitude.
stream_primary_cores
engine.scenario_types.BatchVersusStreamingIngestScenario.stream_primary_cores
Primary streaming cores after integer division by the redundancy factor.
Equation: return stream_cores // redundancy_factor.
stream_per_core
engine.scenario_types.BatchVersusStreamingIngestScenario.stream_per_core
Incoming MB/s per primary streaming core, assuming equal traffic partitioning.
Equation: return (throughput / stream_primary_cores).to('MB/s').
batch_aggregate
engine.scenario_types.BatchVersusStreamingIngestScenario.batch_aggregate
GB/s needed to drain one hour’s arrivals inside the batch window.
Equation: return (hourly_data / batch_window).to('GB/s').
batch_per_core
engine.scenario_types.BatchVersusStreamingIngestScenario.batch_per_core
Batch decode rate in MB/s per core under equal work partitioning.
Equation: return (batch_aggregate / batch_cores).to('MB/s').