Two Phases, One Request

The same model on the same GPU hits two different ceilings, and that changes everything.

node

intermediate

Discover why LLM inference has two distinct performance regimes (prefill and decode) with different bottlenecks. The foundation for all LLM serving analysis.

The Question

A CNN processes one image in one pass. An LLM generates text one token at a time, yet the first token and the hundredth token are bottlenecked by completely different hardware resources. Why does the same model on the same GPU have two different speed limits?

Understanding this two-phase structure is what separates a systems engineer who can predict serving costs from one who has to discover them in production.

Prerequisites

Complete Tutorial 0: Hello, Roofline and Tutorial 1: The Memory Wall. This tutorial assumes familiarity with memory-bound vs. compute-bound regimes and the roofline model.

Learning Objectives

Distinguish the two phases of LLM inference: prefill (TTFT) and decode (ITL)
Explain why prefill is compute-bound and decode is memory-bound
Predict which hardware spec (FLOP/s or bandwidth) matters for each phase
Compare GPUs based on their serving characteristics, not only peak specs

Background: How LLM Inference Works

Unlike a CNN that processes a fixed input in one forward pass, an LLM generates output autoregressively, one token at a time:

Prefill (Time to First Token, TTFT): The model processes the entire input prompt in a single forward pass. All prompt tokens are processed in parallel, saturating the GPU’s compute units. This phase is compute-bound, so optimizing TTFT means getting more TFLOP/s.
Decode (Inter-Token Latency, ITL): Each subsequent token requires a full forward pass through the model but processes only one token of new input. The model weights (8.03 billion parameters × 2 bytes per FP16 parameter = 16.06 GB for Llama-3.1-8B) must be loaded from HBM for each token, yet only a small amount of arithmetic is performed. This phase is memory-bound, so optimizing ITL means getting more GB/s of HBM bandwidth.

The same GPU and the same model face two completely different bottlenecks.

1. Setup

import mlsysim
from mlsysim.solvers import ServingModel

The previous tutorials used Engine.solve, which models inference as a single forward pass. LLM serving, however, is not a single pass; it has two distinct phases with different bottlenecks. The ServingModel models each phase separately and reports TTFT (time to first token) and ITL (inter-token latency) instead of a single latency number.

2. First Serving Prediction

Load Llama-3.1-8B (8.03B parameters, 32 layers, hidden dimension 4096) and the NVIDIA H100 (989 TFLOP/s FP16, 3.35 TB/s HBM3, 80 GiB / 85.9 GB capacity) from the Zoo:

from mlsysim.solvers import ServingModel

model = mlsysim.Models.Language.Llama3_8B
hardware = mlsysim.Hardware.Cloud.H100

solver = ServingModel()
result = solver.solve(
    model=model,
    hardware=hardware,
    seq_len=2048,       # 2K token context window
    batch_size=1,       # single user
    precision="fp16"
)

from mlsysim.show import table, info

info("Phase Analysis",
     TTFT_prefill=result.ttft.to('ms'),
     ITL_per_token=result.itl.to('ms'))

info("Memory Budget",
     Model_weights=result.model_weights_size,
     KV_cache=result.kv_cache_size,
     Memory_utilization=f"{result.memory_utilization:.1%}")

── Phase Analysis ──────────────────────────
TTFT prefill:   70.97 ms
ITL per token:  5.19 ms
── Memory Budget ───────────────────────────
Model weights:       16.06 GB
KV cache:            0.268 GB
Memory utilization:  19.0%

The two numbers tell different stories:

TTFT is 71.0 ms, governed by the 989 TFLOP/s compute ceiling (evaluated at ServingModel’s default 50% compute efficiency)
ITL is 5.19 ms, governed by the 3.35 TB/s HBM bandwidth ceiling

The asymmetry comes from parallelism. Prefill processes all 2048 prompt tokens in parallel, performing 2048× more linear arithmetic per weight load than decode, which processes one token at a time. Across the linear layers, prefill’s arithmetic intensity is 2048 FLOP/byte, well above the H100’s 295.2 FLOP/byte ridge point. Decode’s linear intensity is 1.0 FLOP/byte, far below that ridge point. The weights (16.06 GB) are the same and are loaded the same way, yet the two phases operate in completely different regimes.

3. Why They Respond to Different Optimizations

The same asymmetry appears across GPU generations. Because TTFT and ITL operate in different regimes, they respond to different hardware specs:

gpus = [
    ("A100",  mlsysim.Hardware.Cloud.A100),
    ("H100",  mlsysim.Hardware.Cloud.H100),
    ("H200",  mlsysim.Hardware.Cloud.H200),
]

rows = []
for name, hw in gpus:
    r = solver.solve(model=model, hardware=hw, seq_len=2048, batch_size=1, precision="fp16")
    rows.append([
        name,
        hw.compute.peak_flops.to("TFLOPs/s"),
        hw.memory.bandwidth.to("TB/s"),
        r.ttft.to('ms'),
        r.itl.to('ms'),
    ])

table(["GPU", "TFLOP/s", "BW (TB/s)", "TTFT (ms)", "ITL (ms)"], rows)

GPU        TFLOP/s  BW (TB/s)  TTFT (ms)  ITL (ms)
──────────────────────────────────────────────────
A100  312 TFLOPs/s  2.04 TB/s   225.0 ms   8.33 ms
H100  989 TFLOPs/s  3.35 TB/s   70.97 ms   5.19 ms
H200  989 TFLOPs/s  4.80 TB/s   70.97 ms   3.72 ms

Compare the ratios:

A100 → H100: FLOP/s increases 3.2× (312 to 989 TFLOP/s), and TTFT improves 3.2× (225.0 ms to 71.0 ms). Bandwidth increases 1.6× (2.04 to 3.35 TB/s), and ITL improves 1.6× (8.33 ms to 5.19 ms).
H100 → H200: FLOP/s stays at 989 TFLOP/s, and TTFT stays at 71.0 ms. Bandwidth increases 1.4× (3.35 to 4.80 TB/s), and ITL improves 1.4× (5.19 ms to 3.72 ms).

Each metric tracks its own ceiling. TTFT scales with compute. ITL scales with bandwidth.

4. The Asymmetry: Where Quantization Helps

Quantization (reducing numerical precision) shrinks the model weights from 16.06 GB (FP16) to 8.03 GB (INT8) and 4.02 GB (INT4). Because decode must load all weights from HBM at every step, fewer bytes directly reduce ITL (from 5.19 ms in FP16 to 2.76 ms in INT8 and 1.54 ms in INT4). Weight-only INT4 quantization, which dequantizes weights to FP16 in registers, leaves prefill TTFT unchanged at 71.0 ms, whereas native INT8 Tensor Cores on the H100 also double prefill compute throughput (35.5 ms):

rows = []
for prec in ["fp16", "int8", "int4"]:
    r = solver.solve(model=model, hardware=hardware, seq_len=2048, batch_size=1, precision=prec)
    rows.append([prec, r.ttft.to('ms'), r.itl.to('ms'), r.model_weights_size])

table(["Precision", "TTFT (ms)", "ITL (ms)", "Weights"], rows)

Precision  TTFT (ms)  ITL (ms)   Weights
────────────────────────────────────────
fp16        70.97 ms   5.19 ms  16.06 GB
int8        35.47 ms   2.76 ms   8.03 GB
int4        70.97 ms   1.54 ms   4.02 GB

Key Insight

LLM serving is not one problem but two problems in sequence. Prefill (TTFT) is compute-bound and scales with FLOP/s. Decode (ITL) is memory-bound and scales with bandwidth. Three consequences follow:

Quantization is a decode optimization (reduces bytes loaded per step)
More TFLOP/s is a prefill optimization (processes prompt tokens faster)
The right GPU depends on which phase dominates the latency budget

A chatbot (short prompts, long responses) is ITL-dominated → buy bandwidth. A summarization service (long documents, short outputs) is TTFT-dominated → buy compute.

Going Further: Speculative Decoding

This two-phase asymmetry also explains why speculative decoding works: a small draft model generates candidate tokens cheaply, then the large model verifies them in a single parallel pass (like prefill). It converts the large model’s spare compute into reduced memory loads, attacking the decode bottleneck at the algorithmic level.

5. Putting It Together: SLA-Based Hardware Selection

Suppose the production SLA is TTFT < 200 ms and ITL < 50 ms/token at a 4096-token context. The sweep below shows which GPUs qualify:

gpus_all = [
    ("T4",    mlsysim.Hardware.Cloud.T4),
    ("A100",  mlsysim.Hardware.Cloud.A100),
    ("H100",  mlsysim.Hardware.Cloud.H100),
    ("H200",  mlsysim.Hardware.Cloud.H200),
]

TTFT_SLA = 200   # ms
ITL_SLA = 50     # ms

rows = []
for name, hw in gpus_all:
    r = solver.solve(model=model, hardware=hw, seq_len=4096, batch_size=1, precision="fp16")
    ttft = r.ttft.to("ms").magnitude
    itl = r.itl.to("ms").magnitude
    ttft_ok = ttft <= TTFT_SLA
    itl_ok = itl <= ITL_SLA
    rows.append([
        name,
        f"{ttft:.1f} ms",
        f"{itl:.2f} ms",
        "✓" if ttft_ok else "✗",
        "✓" if itl_ok else "✗",
        "PASS" if ttft_ok and itl_ok else "FAIL",
    ])

table(["GPU", "TTFT", "ITL", "TTFT OK?", "ITL OK?", "Verdict"], rows)

GPU        TTFT       ITL  TTFT OK?  ITL OK?  Verdict
─────────────────────────────────────────────────────
T4    2294.7 ms  52.19 ms         ✗        ✗     FAIL
A100   478.1 ms   8.46 ms         ✗        ✓     FAIL
H100   150.8 ms   5.27 ms         ✓        ✓     PASS
H200   150.8 ms   3.78 ms         ✓        ✓     PASS

This is the analysis every ML engineer should run before choosing serving infrastructure. The answer depends not only on the GPU but on the model size, context length, batch size, and precision, all of which the ServingModel captures analytically.

Next Steps

KV-Cache: The Hidden Tax: Discover what limits how many concurrent users a GPU can serve
The Memory Wall: Why GPU generations do not deliver the expected speedup
Quantization: Not a Free Lunch: When reducing precision helps and when it does not
The $9M Question: How chain-of-thought reasoning multiplies serving costs