Skip to main content

engine.engine.Engine

engine.engine.Engine()

The core analytical engine for single-node Roofline performance modeling.

Maps a (Workload, HardwareNode) pair to a PerformanceProfile by applying the Roofline model: the workload is “lowered” into a hardware-agnostic ComputationGraph, then bounded by the hardware’s compute ceiling and memory bandwidth ceiling.

The Iron Law of ML Training

Compute-dominated training time can be organized as:

Time = FLOPs / (N × Peak_FLOPS × MFU × η_scaling × Goodput)

where N is device count, MFU is Model FLOPs Utilization, η_scaling is scaling efficiency, and Goodput accounts for failures and checkpointing. This accounting view requires mutually consistent definitions of the efficiency factors. Engine.solve evaluates compute and memory bounds on one device; DistributedModel adds fleet coordination terms. Neither this expression nor a single efficiency factor replaces those bounds.

Source: Williams et al. (2009), “Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures.”

Methods

Name Description
solve Estimate one device’s compute, memory, and capacity limits.
sweep Evaluate feasible inference configurations over a Cartesian product.

solve

engine.engine.Engine.solve(
    model,
    hardware,
    batch_size=1,
    precision='fp16',
    efficiency=0.5,
    raise_errors=False,
    is_training=False,
    seq_len=2048,
    zero_stage=0,
    dp_size=1,
    is_lora=False,
    activation_recomputation=False,
)

Estimate one device’s compute, memory, and capacity limits.

Parameters

Name Type Description Default
model Workload Workload whose lower method supplies FLOPs and weight bytes. A dense autoregressive Transformer graph represents one token; a vision graph represents one input. Training multiplies Transformer work by seq_len before forming a batch. required
hardware HardwareNode Device specification with FLOP/s, byte/s, byte capacity, dispatch time, and optional TDP in watts and alternate memory tiers. required
batch_size int Positive number of samples per batch, or sequences during training. 1
precision str Registered numeric format controlling bytes per weight and the precision-specific compute ceiling. Missing device precision throughput falls back to the device’s default peak. 'fp16'
efficiency float Declared compute efficiency in [1e-9, 1]. It derates FLOP/s only; bandwidth remains the selected tier’s peak specification. This input is an assumption, not a fitted bandwidth efficiency. 0.5
raise_errors bool Raise OOMError for a footprint above device capacity. Otherwise retain feasible=False and estimate offload-limited latency. False
is_training bool Include backward work and training state. Inference capacity checks weights only; use ServingModel for KV-aware capacity. False
seq_len int Positive tokens per training sequence. Scales Transformer work and activation storage; generic inference remains one forward step. 2048
zero_stage int ZeRO stage 0, 1, 2, or 3, passed to Transformer training memory. 0
dp_size int Positive data-parallel count used for ZeRO state sharding. Does not add distributed execution or communication to this solver. 1
is_lora bool Use the training-memory adapter approximation. The generic non-Transformer fallback uses a separate 1.1-times-weights heuristic. False
activation_recomputation bool Recompute a forward pass during training: four rather than three forward-equivalent work units, paired with full activation recompute. False

Returns

Name Type Description
PerformanceProfile Modeled batch latency in milliseconds, throughput in samples/s, energy in joules, byte footprint, utilization fractions, selected resource bound, and feasibility trace. A false capacity flag remains false even when an offload-latency estimate is returned.

See Also

mlsysim.physics.performance.calc_bottleneck : Compute and memory bounds. mlsysim.solvers.ServingModel : KV-aware prefill and decode estimates. mlsysim.models.types.TransformerWorkload.training_memory : Training state.

Notes

Equation: T = max(F_batch / (eta * F_peak), W * (1 + 0.1 * B) / BW) + T_dispatch + L * T_layer * launch_multiplier. Here W is weight bytes, B is batch size, and launch_multiplier is three for training and one for inference. Training work is three forward passes, or four with full recomputation. Throughput is B / T; energy is average_power * T.

Assumptions: activation traffic is the declared 0.1 * W per-sample heuristic, not an architecture trace. Compute and memory bounds overlap ideally. The bandwidth path selects host offload, resident SRAM, flash, or primary memory. Offload charges all traffic at the limiting link rate, not only the spilled fraction. Power is TDP times an idle-plus-HFU model with fractions in engine.calibration; missing TDP yields zero estimated energy. Fixed dispatch and layer taxes are modeling defaults.

Excludes: KV-cache capacity, attention-context work in generic inference, allocator reserve, kernel workspaces, queueing, serving concurrency, and communication between devices. Feasibility is a footprint check, not a deployment guarantee. The bottleneck label compares compute and memory terms; overhead_dominated separately flags a larger additive tax.

References

Williams et al. (2009), “Roofline: An Insightful Visual Performance Model for Multicore Architectures.” https://doi.org/10.1145/1498765.1498785 (bibliography key: williams2009). The activation traffic, layer tax, and power coefficients are separate implementation assumptions, not constants established by Roofline.

Examples

>>> from mlsysim import Engine, Hardware, Models
>>> result = Engine.solve(Models.Vision.ResNet50, Hardware.Cloud.H100)
>>> result.feasible
True
>>> result.latency >= max(result.latency_compute, result.latency_memory)
True

sweep

engine.engine.Engine.sweep(
    model,
    hardware_list,
    batch_sizes=None,
    precisions=None,
    efficiency=0.5,
)

Evaluate feasible inference configurations over a Cartesian product.

Parameters

Name Type Description Default
model Workload Fixed workload to lower for each device and precision. required
hardware_list list of HardwareNode Devices to evaluate in their supplied order; an empty list gives no results. required
batch_sizes list of int Positive sample counts. None or an empty list uses [1]. None
precisions list of str Registered numeric formats. None or an empty list uses [‘fp16’]. None
efficiency float Declared compute efficiency in [1e-9, 1], shared by all configurations. 0.5

Returns

Name Type Description
list of dict One entry per capacity-feasible combination, containing hardware name, batch_size, precision, and a PerformanceProfile with units. Capacity failures are omitted rather than returned as failed rows.

See Also

mlsysim.engine.engine.Engine.solve : Per-configuration model and limits.

Notes

Equation: results = [solve(model, h, b, p) for h, b, p in hardware_list x batch_sizes x precisions if the footprint fits]. The latency and energy equations are those of Engine.solve.

Assumptions: every point uses generic inference and the same efficiency; input order determines result order. OOMError alone is caught; invalid parameters and other errors still propagate.

Excludes: training configuration sweeps, optimization, queueing, measured hardware execution, and a record of omitted infeasible configurations. No runtime bound follows from the number of configurations alone.

References

Derivation: enumeration of the Cartesian product delegates each point to Engine.solve and filters its capacity failures. There is no separate performance equation or cross-configuration scheduling model.

Examples

>>> from mlsysim import Engine, Hardware, Models
>>> rows = Engine.sweep(Models.Vision.ResNet50, [Hardware.Cloud.H100], [1, 2])
>>> [row['batch_size'] for row in rows]
[1, 2]
>>> all(row['profile'].feasible for row in rows)
True
Back to top