Skip to main content

Quantization: Not a Free Lunch

INT4 gives 3.4× speedup for memory-bound decode. For compute-bound weight-only quantization, it gives 1.0×.

algorithm
intermediate
Discover that quantization helps only when a workload is memory-bound (or when native low-precision Tensor Cores boost compute). Compare the effect of INT4 on LLM decode vs. large-batch training.

The Question

INT4 quantization reduces model weight size by 4× (from 16.06 GB in FP16 to 4.02 GB in INT4 for Llama-3.1-8B). Intuitively, 4× fewer bytes should mean 4× faster execution. Does INT4 always give a 4× speedup?

Tutorial 1 already supplies the tools to predict the answer: performance is governed by whichever ceiling binds first. When quantization reduces bytes transferred from HBM without increasing arithmetic throughput, it accelerates memory-bound workloads and leaves compute-bound workloads unchanged.

This tutorial makes the prediction, runs the experiment, and shows that the outcome depends entirely on the operating regime.

NotePrerequisites

Complete Tutorial 1: The Memory Wall and Tutorial 2: Two Phases, One Request. This tutorial assumes familiarity with memory-bound vs. compute-bound regimes and why LLM decode is memory-bound.

NoteLearning Objectives
  • Predict whether quantization will help a given workload based on its regime
  • Measure the speedup from INT8 and INT4 for both memory-bound and compute-bound workloads
  • Explain why weight-only INT4 yields 3.4× in one regime and 1.0× in another
  • Evaluate the accuracy-compression trade-off using CompressionModel
TipBackground: What Quantization Actually Does

Quantization reduces the number of bytes per parameter:

Precision Bytes/Param Relative to FP16 Llama-3.1-8B Weights
FP16 2 1.0× (baseline) 16.06 GB
INT8 1 0.5× 8.03 GB
INT4 0.5 0.25× 4.02 GB

For memory-bound workloads, performance scales with bytes loaded from HBM per step. Halving bytes nearly halves latency. For compute-bound workloads, performance scales with FLOP/s. Fewer weight bytes do not change the number of FLOPs, so latency stays the same unless the hardware also provides higher-throughput low-precision Tensor Cores.

The key criterion is always which ceiling the workload is hitting.

1. Setup

import mlsysim
from mlsysim.solvers import ServingModel, SingleNodeModel
from mlsysim.solvers import CompressionModel

2. Memory-Bound Case: LLM Decode at Batch 1

LLM decoding at batch 1 is the textbook memory-bound workload. Each token generation must reload the entire model from HBM. Fewer bytes per parameter means fewer bytes to load, and therefore lower inter-token latency:

from mlsysim.solvers import ServingModel
from mlsysim.show import table, info

model = mlsysim.Models.Language.Llama3_8B
hardware = mlsysim.Hardware.Cloud.H100
solver = ServingModel()

rows = []
baseline_itl = None
for prec in ["fp16", "int8", "int4"]:
    r = solver.solve(
        model=model, hardware=hardware,
        seq_len=2048, batch_size=1, precision=prec
    )
    itl_ms = r.itl.to("ms").magnitude
    if baseline_itl is None:
        baseline_itl = itl_ms
    speedup = baseline_itl / itl_ms
    rows.append([prec, r.itl.to('ms'), r.model_weights_size, f"{speedup:.1f}x"])

table(["Precision", "ITL", "Weights", "Speedup vs FP16"], rows)
Precision      ITL   Weights  Speedup vs FP16
─────────────────────────────────────────────
fp16       5.19 ms  16.06 GB             1.0x
int8       2.76 ms   8.03 GB             1.9x
int4       1.54 ms   4.02 GB             3.4x

INT8 cuts weights from 16.06 GB to 8.03 GB and lowers ITL from 5.19 ms to 2.76 ms, a 1.9× speedup. INT4 cuts weights to 4.02 GB and lowers ITL to 1.54 ms, a 3.4× speedup. The speedup tracks the byte reduction closely (falling slightly short of 2.0× and 4.0× only because per-layer kernel launch overhead does not shrink with weight precision). Every weight byte removed directly reduces HBM transfer time.

3. Compute-Bound Case: Large-Batch Execution

The same optimization behaves differently on a compute-bound workload, ResNet-50 at batch 256 on the NVIDIA A100:

from mlsysim.solvers import SingleNodeModel

train_model = mlsysim.Models.Vision.ResNet50
train_hw = mlsysim.Hardware.Cloud.A100
train_solver = SingleNodeModel()

rows = []
baseline_lat = None
for prec in ["fp16", "int8", "int4"]:
    p = train_solver.solve(
        model=train_model, hardware=train_hw,
        batch_size=256, precision=prec
    )
    lat_ms = p.latency.to("ms").magnitude
    if baseline_lat is None:
        baseline_lat = lat_ms
    speedup = baseline_lat / lat_ms if lat_ms > 0 else 0
    rows.append([prec, p.latency.to('ms'), f"{p.throughput.magnitude:.0f} img/s", f"{speedup:.1f}x"])

table(["Precision", "Latency", "Throughput", "Speedup"], rows)
Precision   Latency   Throughput  Speedup
─────────────────────────────────────────
fp16       13.97 ms  18323 img/s     1.0x
int8        7.24 ms  35343 img/s     1.9x
int4       13.97 ms  18323 img/s     1.0x

At batch 256, ResNet-50 on the NVIDIA A100 is compute-bound: weight-only INT4 (13.97 ms, 18,323 img/s) delivers 1.0× speedup relative to FP16 (13.97 ms, 18,323 img/s). Because the A100 lacks native INT4 Tensor Cores in the Zoo, weight-only INT4 dequantizes weights to FP16 in registers and executes at the 312 TFLOP/s FP16 compute ceiling; shrinking weight traffic by 4× does nothing when arithmetic is the bottleneck. By contrast, INT8 reaches 7.24 ms (1.9× speedup) only because the A100 provides dedicated INT8 Tensor Cores at 624 TFLOP/s (2× the FP16 compute peak).

WarningNuance: INT8 Tensor Cores vs. Weight-Only INT4

GPUs with dedicated INT8 Tensor Cores (such as the A100 at 624 TFLOP/s INT8 vs. 312 TFLOP/s FP16, a 2× compute boost) raise both the memory ceiling (fewer bytes) and the compute ceiling (more INT ops/sec) when activations and weights are quantized together (W8A8). Weight-only INT4 (W4A16), by contrast, reduces HBM weight traffic by 4× while keeping arithmetic on FP16 Tensor Cores, so it accelerates memory-bound decode (3.4×) while leaving compute-bound steps unchanged (1.0×).

4. The Reveal: Same Optimization, Two Regimes

Placing the results side by side makes the contrast stark:

rows = []

# Memory-bound: LLM decode
decode_row = ["Llama-3 8B decode"]
for prec in ["fp16", "int8", "int4"]:
    r = solver.solve(model=model, hardware=hardware, seq_len=2048, batch_size=1, precision=prec)
    decode_row.append(r.itl.to('ms'))
decode_row.append("Memory-bound")
rows.append(decode_row)

# Compute-bound: large-batch CNN execution
train_row = ["ResNet-50 bs=256"]
for prec in ["fp16", "int8", "int4"]:
    p = train_solver.solve(model=train_model, hardware=train_hw, batch_size=256, precision=prec)
    train_row.append(p.latency.to('ms'))
train_row.append("Compute-bound")
rows.append(train_row)

table(["Workload", "FP16", "INT8", "INT4", "Regime"], rows)
Workload               FP16     INT8      INT4         Regime
─────────────────────────────────────────────────────────────
Llama-3 8B decode   5.19 ms  2.76 ms   1.54 ms   Memory-bound
ResNet-50 bs=256   13.97 ms  7.24 ms  13.97 ms  Compute-bound
ImportantKey Insight

Weight-only quantization reduces bytes loaded from memory. For a memory-bound workload, fewer bytes mean proportional speedup. For a compute-bound workload, fewer bytes alone mean no speedup, because compute, not memory, is the ceiling. The regime determines whether byte reduction works: INT4 lowers Llama-3.1-8B decode latency from 5.19 ms to 1.54 ms (3.4× speedup, memory-bound) yet leaves ResNet-50 at batch 256 at 13.97 ms (1.0×, compute-bound). Always check the regime before choosing an optimization.

5. The Accuracy Tax: CompressionModel

Quantization is not free; it trades accuracy for speed and memory capacity. The CompressionModel quantifies this trade-off relative to an FP32 baseline:

from mlsysim.solvers import CompressionModel

comp_solver = CompressionModel()

rows = []
for bits in [16, 8, 4]:
    c = comp_solver.solve(
        model=model, hardware=hardware,
        method="quantization", target_bitwidth=bits
    )
    rows.append([
        bits,
        c.compressed_size_gb,
        f"{c.compression_ratio:.1f}x",
        f"{c.estimated_accuracy_delta:+.1%}",
        f"{c.memory_savings_pct:.1f}%"
    ])

table(["Bits", "Compressed", "Compression", "Accuracy", "Savings"], rows)
Bits  Compressed  Compression  Accuracy  Savings
────────────────────────────────────────────────
16      16.06 GB         2.0x     +0.0%    50.0%
8        8.03 GB         4.0x     -0.2%    75.0%
4        4.02 GB         8.0x     -2.5%    87.5%

Relative to FP32, 8-bit quantization shrinks Llama-3.1-8B to 8.03 GB (4.0× compression, 75.0% memory savings) with an estimated accuracy change of -0.2%. Going to 4-bit shrinks weights to 4.02 GB (8.0× compression, 87.5% savings) with an estimated accuracy change of -2.5%. The decision is not “always quantize” but “quantize when the workload is memory-bound AND the accuracy cost is acceptable for the application.”

WarningWhen NOT to Quantize
  • Training: Training is compute-bound at large batch sizes. Weight-only quantization does not help and low-precision gradients can harm convergence.
  • High-accuracy applications: Medical, financial, and safety-critical systems may not tolerate even a 0.2% to 2.5% accuracy loss.
  • Already compute-bound inference: An inference workload that runs at large batch sizes (e.g., offline batch processing) is already compute-bound unless hardware supports faster low-precision arithmetic.

Next Steps

  • KV-Cache: The Hidden Tax: Quantization also shrinks the KV-cache, allowing more concurrent users
  • The Memory Wall: Revisit the memory wall to see how quantization shifts the bandwidth bottleneck
  • Starving the GPU: Another case where the bottleneck is not where it is expected
  • Where to Invest: Quantify how much quantization buys compared to hardware upgrades
Back to top