Skip to main content

Starving the GPU

The GPU is ready for the next batch. The CPU is still decoding JPEGs.

data
intermediate
Discover that the data pipeline, not the GPU, is often the binding constraint in training. Use DataModel and TransformationModel to find the crossover where CPU preprocessing stalls the accelerator.

The Question

An engineer launches ResNet-50 training on an NVIDIA A100 at batch size 256 and watches nvidia-smi. Active GPU duty cycle reads 21.8%, far below full saturation. The model is compute-bound. The hardware is top-tier. Why is the GPU sitting idle 78.2% of the time?

The answer is almost never the model or the GPU. It is the invisible pipeline upstream: JPEG decoding, random cropping, color jitter, and normalization, all running on the CPU. When the CPU cannot prepare batches fast enough, the GPU starves.

NotePrerequisites

Complete Tutorial 0: Hello, Roofline and Tutorial 1: The Memory Wall. This tutorial assumes familiarity with memory-bound vs. compute-bound regimes and how Engine.solve reports bottlenecks.

NoteLearning Objectives
  • Measure the GPU’s step time in isolation using SingleNodeModel
  • Calculate the data pipeline’s throughput using DataModel and TransformationModel
  • Identify the batch size crossover where the CPU becomes the binding constraint
  • Predict how many CPU workers are needed to eliminate the data bottleneck
TipBackground: The Three Stages of a Training Step

Every training step has three sequential stages. The slowest one, not the GPU alone, determines actual throughput:

  1. Storage I/O (Wall 8): Read raw data from disk into CPU memory
  2. CPU Preprocessing (Wall 9): Decode, resize, augment, normalize
  3. Accelerator Compute (Wall 1): Forward pass, backward pass, weight update

The GPU cannot start until stages 1 and 2 finish. If either is slower than the GPU, the accelerator duty cycle drops below 100%. This is the data pipeline bottleneck.

1. Setup

import mlsysim
from mlsysim.solvers import SingleNodeModel, DataModel
from mlsysim.solvers import TransformationModel

2. GPU Compute Time: The Apparent Ceiling

This tutorial switches from LLM serving (Tutorials 2–3) to CNN training because the data pipeline bottleneck is most visible there. LLM training on tokenized text has a tiny data footprint (~8 MB/s, as Tutorial 12 shows). Image training with JPEG decoding, resizing, and augmentation can demand 10–100× more CPU work per sample, and that is where the GPU starves.

First, establish how fast the NVIDIA A100 processes a ResNet-50 step at batch 256 in isolation, with no data loading and no preprocessing, only compute:

from mlsysim.solvers import SingleNodeModel
from mlsysim.core.units import Q_
from mlsysim.show import table, info

model = mlsysim.Models.Vision.ResNet50
hardware = mlsysim.Hardware.Cloud.A100
solver = SingleNodeModel()

# Baseline: ResNet-50 on A100, batch 256, FP16
profile = solver.solve(model=model, hardware=hardware, batch_size=256, precision="fp16")

info("GPU Compute Baseline",
     Model=model.name,
     Hardware=hardware.name,
     Batch_size=256,
     Step_latency=profile.latency.to('ms'),
     Throughput=f"{profile.throughput.magnitude:.0f} img/s",
     Bottleneck=profile.bottleneck)
── GPU Compute Baseline ────────────────────
Model:         ResNet-50
Hardware:      NVIDIA A100
Batch size:    256
Step latency:  13.97 ms
Throughput:    18323 img/s
Bottleneck:    Compute

The GPU finishes this 256-image batch in 13.97 ms, setting an accelerator compute ceiling of 18,323 img/s. The next step checks whether the data pipeline can keep up.

3. Storage I/O Check: Disk Supply vs. Demand

ImageNet images average 500 KB each (JPEG compressed). At batch 256 (128 MB per step over 13.97 ms), the GPU demands 9.16 GB/s of raw image data, and the storage subsystem must supply it:

from mlsysim.solvers import DataModel

sample_size = Q_("500 KB")  # Average ImageNet JPEG
batch_size = 256

# Data demand = batch_size x sample_size / step_time
step_time_s = profile.latency.to("s").magnitude
data_per_step = (batch_size * sample_size.to("GB")).magnitude
demand_rate = Q_(data_per_step / step_time_s, "GB/s")

data_solver = DataModel()
data_result = data_solver.solve(workload_data_rate=demand_rate, hardware=hardware)

info("Storage I/O Check",
     Data_demand=f"{demand_rate:.3f}",
     Storage_supply=f"{data_result.supply_bw:.2f}",
     Utilization=f"{data_result.utilization:.1%}",
     Is_stalled=data_result.is_stalled)
── Storage I/O Check ───────────────────────
Data demand:     9.162 GB / second
Storage supply:  32.00 GB / second
Utilization:     28.6%
Is stalled:      False

Storage I/O is not the problem: the 9.16 GB/s demand consumes 28.6% of the 32.0 GB/s storage supply. The bottleneck is not reading the bytes. It is transforming them.

4. The Reveal: CPU Preprocessing Is the Wall

Even with fast storage, the CPU must decode JPEGs, apply random crops, color jitter, and normalization. A typical CPU worker processes ImageNet images at 250 MB/s. With 8 workers, total CPU ingestion throughput is 2 GB/s (4,000 img/s at 500 KB per image):

from mlsysim.solvers import TransformationModel

transform_solver = TransformationModel()
cpu_throughput = Q_("2 GB/s")  # 8 workers x 250 MB/s each

t = transform_solver.solve(
    batch_size=256,
    sample_size_bytes=sample_size,
    cpu_throughput=cpu_throughput,
    accelerator_step_time=profile.latency
)

info("CPU vs GPU Pipeline",
     CPU_transform_time=t.transform_time,
     GPU_step_time=t.accelerator_step_time,
     CPU_is_bottleneck=t.is_bottleneck,
     GPU_utilization=f"{t.accelerator_utilization:.1%}",
     Slowdown_factor=f"{t.slowdown_factor:.2f}x")
── CPU vs GPU Pipeline ─────────────────────
CPU transform time:  64 ms
GPU step time:       13.97 ms
CPU is bottleneck:   True
GPU utilization:     21.8%
Slowdown factor:     4.58x
ImportantKey Insight

The binding constraint is not silicon but JPEG decoding on the CPU. The data pipeline (Wall 9: Transformation) becomes the bottleneck before the GPU (Wall 1: Compute). The two step times above show that the NVIDIA A100 finishes a 256-image batch in 13.97 ms (18,323 img/s ceiling), whereas the 8 CPU workers require 64.0 ms (4,000 img/s ceiling), a 4.58× slowdown. The GPU is active for only 21.8% of each step and sits idle for 78.2% waiting for data. This is why production training pipelines use GPU-accelerated preprocessing (NVIDIA DALI), pre-decoded datasets, or aggressive prefetching.

5. Batch Size Sweep: Finding the Crossover

A sweep over batch sizes locates the point where the CPU becomes the binding constraint. At small batches, the GPU is slower and data arrives in time. At large batches, the GPU becomes more efficient but the CPU falls behind:

rows = []
for bs in [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]:
    p = solver.solve(model=model, hardware=hardware, batch_size=bs, precision="fp16")

    t = transform_solver.solve(
        batch_size=bs,
        sample_size_bytes=sample_size,
        cpu_throughput=cpu_throughput,
        accelerator_step_time=p.latency
    )

    binding = "Transformation" if t.is_bottleneck else p.bottleneck
    rows.append([
        bs,
        f"{p.latency.to('ms').magnitude:.2f} ms",
        f"{t.transform_time.to('ms').magnitude:.2f} ms",
        binding,
        f"{t.accelerator_utilization:.1%}"
    ])

table(["Batch", "GPU Step", "CPU Xform", "Binding", "GPU Util"], rows)

crossover = next(r[0] for r in rows if r[3] == "Transformation")
info("Crossover", First_CPU_bound_batch=crossover)
Batch  GPU Step  CPU Xform         Binding  GPU Util
────────────────────────────────────────────────────
1       0.57 ms    0.25 ms         Compute    100.0%
2       0.62 ms    0.50 ms         Compute    100.0%
4       0.73 ms    1.00 ms  Transformation     72.5%
8       0.94 ms    2.00 ms  Transformation     46.8%
16      1.36 ms    4.00 ms  Transformation     33.9%
32      2.20 ms    8.00 ms  Transformation     27.5%
64      3.88 ms   16.00 ms  Transformation     24.2%
128     7.24 ms   32.00 ms  Transformation     22.6%
256    13.97 ms   64.00 ms  Transformation     21.8%
512    27.43 ms  128.00 ms  Transformation     21.4%
1024   54.34 ms  256.00 ms  Transformation     21.2%
── Crossover ───────────────────────────────
First CPU bound batch:  4

The crossover sits at batch size 4. At batch size 1 (0.57 ms GPU step vs. 0.25 ms CPU transform) and batch size 2, the CPU keeps up and the GPU runs at 100% duty cycle, yet by batch size 4 Wall 9 binds and never lets go. CPU transform time is strictly proportional to batch size. The GPU step is a fixed dispatch overhead plus a per-sample compute cost, and that fixed overhead only shields the GPU while batches are tiny. Once it is amortized, both costs grow linearly, so the GPU duty cycle levels off near 21.2% (at batch 1024: 54.34 ms GPU step vs. 256.0 ms CPU transform) instead of falling to zero.

6. The Fix: Adding CPU Workers

The simplest fix for a CPU bottleneck is more workers. The sweep below compares 8, 16, and 32 workers at batch size 512:

rows = []
for n_workers in [8, 16, 32]:
    cpu_tp = Q_(f"{n_workers * 250} MB/s")

    p = solver.solve(model=model, hardware=hardware, batch_size=512, precision="fp16")

    t = transform_solver.solve(
        batch_size=512,
        sample_size_bytes=sample_size,
        cpu_throughput=cpu_tp,
        accelerator_step_time=p.latency
    )

    rows.append([n_workers, cpu_tp.to('GB/s'), f"{t.accelerator_utilization:.1%}"])

table(["Workers", "Throughput", "GPU Util @ bs=512"], rows)
Workers  Throughput  GPU Util @ bs=512
──────────────────────────────────────
8            2 GB/s              21.4%
16           4 GB/s              42.9%
32           8 GB/s              85.7%

Doubling workers from 8 to 16 doubles active GPU duty cycle from 21.4% to 42.9%, and 32 workers (8 GB/s) raise it to 85.7%, before the pipeline eventually hits either storage I/O limits (Wall 8) or PCIe bandwidth. The lesson is to check all three stages of the pipeline every time.

Next Steps

Back to top