The $9M Question

Chain-of-thought reasoning multiplies the serving bill, turning an algorithm choice into a capital decision.

ops

intermediate

Use InferenceScalingModel and EconomicsModel to quantify how chain-of-thought reasoning multiplies per-query latency, GPU fleet size, and annual infrastructure cost.

The Question

A team wants to add chain-of-thought (CoT) reasoning to its production serving pipeline. The accuracy improvement is clear: $K=8$ reasoning steps (400 reasoning tokens vs. 50 tokens at $K=1$) measurably improve answer quality on hard queries. The open matter is what $K=8$ costs, measured not in tokens but in dollars, GPUs, and annual infrastructure budget. Is this an algorithm decision or a capital expenditure decision?

Prerequisites

Complete Tutorial 2: Two Phases, One Request and Tutorial 3: The KV Cache Wall. This tutorial assumes familiarity with TTFT, ITL, and the two-phase serving model.

Learning Objectives

Calculate per-query latency and energy cost across reasoning depths $K=1$ to $K=16$
Compose InferenceScalingModel with EconomicsModel for annualized fleet cost
Quantify the cost multiplier of $K=8$ reasoning at 100 QPS fleet scale
Evaluate a routing strategy that sends easy queries to a smaller model and hard queries to an expensive reasoning model

Background: Inference-Time Compute Scaling

Standard LLM inference generates one answer directly: prefill the prompt (TTFT), then decode tokens one at a time (ITL). Chain-of-thought reasoning changes this: the model generates $K$ intermediate “thinking” steps, each producing 50 tokens, before the final answer. The cost model becomes:

\[T_{\text{reason}} = \text{TTFT} + K \times T_{\text{step}}\]

where $T_{\text{step}} = \text{tokens\_per\_step} \times \text{ITL}$. Each step is memory-bound (decoding), so the cost scales linearly with $K$ at the decode rate, which belongs to the memory-bandwidth-limited phase. A seemingly algorithmic choice (add more reasoning) translates directly into GPU-hours and dollars.

1. Setup

import mlsysim
from mlsysim import Engine

2. Baseline Serving: Single-Query Cost

The baseline is GPT-4 (1,760B parameters) served on NVIDIA H100 at context length 2048 with no multi-step reasoning. It gives the per-query TTFT and ITL that everything else builds on.

from mlsysim import Models, Hardware
from mlsysim.solvers import ServingModel
from mlsysim.show import table, info

model = Models.Language.GPT4
hardware = Hardware.Cloud.H100

serving = ServingModel()
baseline = serving.solve(
    model=model, hardware=hardware,
    seq_len=2048, batch_size=1, precision="fp16"
)

info("Baseline Serving",
     Model=f"{model.name} ({model.parameters.to('Bparam').magnitude:.0f}B params)",
     Hardware=hardware.name,
     TTFT=baseline.ttft.to('ms'),
     ITL=baseline.itl.to('ms'),
     Memory_Used=f"{baseline.memory_utilization:.0%}",
     Feasible=f"{baseline.feasible}")

── Baseline Serving ────────────────────────
Model:        GPT-4 (1760B params)
Hardware:     NVIDIA H100
TTFT:         14,645.0 ms
ITL:          1,056.8 ms
Memory Used:  4117%
Feasible:     False

For GPT-4 on NVIDIA H100, prefill TTFT is 14,645.0 ms and per-token decode ITL is 1,056.8 ms. Every reasoning step generates 50 tokens at this decode rate (52,837.7 ms per step), so the latency and energy costs accumulate in decode.

3. CoT Sweep: K=1 to K=16

The next cell sweeps reasoning depth $K \in \{1, 4, 8, 16\}$ with the InferenceScalingModel. Each step generates 50 tokens of intermediate reasoning, and the cost multiplier grows with $K$.

from mlsysim.solvers import InferenceScalingModel

cot_solver = InferenceScalingModel()
K_values = [1, 4, 8, 16]

baseline_time = None
results = {}
rows = []

for K in K_values:
    result = cot_solver.solve(
        model=model, hardware=hardware,
        reasoning_steps=K, context_length=2048, precision="fp16"
    )
    total_ms = result.total_reasoning_time.to("ms").magnitude
    energy_j = result.energy_per_query.to("J").magnitude

    if baseline_time is None:
        baseline_time = total_ms

    multiplier = total_ms / baseline_time
    results[K] = result

    rows.append([K, f"{total_ms:.1f}ms", result.tokens_generated, f"{energy_j:.1f} J", f"{multiplier:.1f}x"])

table(["K", "Total Time", "Tokens", "Energy (J)", "Multiplier"], rows)

K   Total Time  Tokens  Energy (J)  Multiplier
──────────────────────────────────────────────
1    67482.7ms      50   47237.9 J        1.0x
4   225995.8ms     200  158197.1 J        3.3x
8   437346.6ms     400  306142.6 J        6.5x
16  860048.2ms     800  602033.8 J       12.7x

Going from $K=1$ (50 tokens, 67,482.7 ms, 47,237.9 J) to $K=8$ (400 tokens, 437,346.6 ms, 306,142.6 J) multiplies per-query time and energy by 6.5× (and by 12.7× at $K=16$ with 800 tokens). The multiplier is slightly below $K$ because each query pays one fixed TTFT (14,645.0 ms) plus $K$ decode steps (52,837.7 ms per step).

4. Annualized Fleet Cost

Per-query cost explains the mechanism, while fleet-level cost drives the budget. The cell below computes the annual infrastructure cost of serving 100 queries per second across $K=1$, $K=4$, $K=8$, and $K=16$.

from mlsysim.solvers import EconomicsModel
from mlsysim.systems.types import Fleet, Node, NetworkFabric
from mlsysim.core.units import Q_

econ = EconomicsModel()
target_qps = 100

fleet_results = {}
fleet_objects = {}
rows = []
for K in K_values:
    r = results[K]
    qt_s = r.total_reasoning_time.to("s").magnitude

    # Each GPU serves one query at a time (batch_size=1)
    qps_per_gpu = 1.0 / qt_s if qt_s > 0 else 0
    gpus_needed = int(target_qps / qps_per_gpu) + 1

    # Build fleet: 8 GPUs per node
    fleet = Fleet(
        name=f"K={K} Serving",
        node=Node(
            name="H100 Node",
            accelerator=hardware,
            accelerators_per_node=8,
            intra_node_bw=Q_("900 GB/s"),
        ),
        count=max((gpus_needed + 7) // 8, 1),
        fabric=NetworkFabric(
            name="IB NDR",
            bandwidth=Q_("400 Gbps").to("GB/s"),
        )
    )

    tco = econ.solve(fleet=fleet, duration_days=365)
    fleet_results[K] = tco
    fleet_objects[K] = fleet

    rows.append([K, f"{qt_s * 1000:.1f}ms", f"{qps_per_gpu:.2f}", fleet.total_accelerators, f"${tco.tco_usd:,.0f}"])

table(["K", "Query (ms)", "QPS/GPU", "GPUs", "Annual TCO ($)"], rows)

K   Query (ms)  QPS/GPU   GPUs  Annual TCO ($)
──────────────────────────────────────────────
1    67482.7ms     0.01   6752     $70,271,265
4   225995.8ms     0.00  22600    $235,208,915
8   437346.6ms     0.00  43736    $455,181,289
16  860048.2ms     0.00  86008    $895,126,035

At 100 QPS, moving from $K=1$ to $K=8$ expands the required fleet from 6,752 GPUs ($70.3M annual TCO) to 43,736 GPUs ($455.2M annual TCO), and to 86,008 GPUs ($895.1M annual TCO) at $K=16$. That jump propagates through the entire infrastructure stack: more GPUs, more power, more cooling, more network fabric, and more capital expenditure.

Key Insight

A seemingly algorithmic decision (“add more reasoning steps”) is actually an infrastructure spending decision. $K=8$ chain-of-thought reasoning multiplies per-query latency by 6.5×, so maintaining 100 QPS scales the fleet from 6,752 to 43,736 GPUs and increases annual TCO from $70.3M to $455.2M. The decision to add CoT reasoning is not only a model architecture choice; it is a capital expenditure decision that belongs in infrastructure budget planning.

5. The Routing Argument

The $455.2M annual TCO at $K=8$ reframes the conversation from model architecture to capital planning. It also raises the question of whether every query must pay the full reasoning cost.

In production, not every query does, and the optimization that avoids it is routing. A router sends easy queries to a smaller, faster model (Llama-3.1-70B at $K=1$, 2,764.5 ms per query) and routes only hard queries to the full GPT-4 $K=8$ reasoning pipeline (437,346.6 ms per query). The cell below models a 70/30 split.

# Scenario: 70% of queries go to Llama-3 70B (no reasoning, K=1)
# 30% go to GPT-4 with K=8 reasoning
model_cheap = Models.Language.Llama3_70B

# Cheap path: Llama-3 70B, K=1
r_cheap = cot_solver.solve(
    model=model_cheap, hardware=hardware,
    reasoning_steps=1, context_length=2048, precision="fp16"
)

# Expensive path: GPT-4, K=8 (already computed)
r_expensive = results[8]

# Weighted average query time
qt_cheap = r_cheap.total_reasoning_time.to("s").magnitude
qt_expensive = r_expensive.total_reasoning_time.to("s").magnitude
qt_blended = 0.70 * qt_cheap + 0.30 * qt_expensive

# Compare: all queries to GPT-4 K=8 vs. routed
qps_blended = 1.0 / qt_blended if qt_blended > 0 else 0
gpus_blended = int(target_qps / qps_blended) + 1

fleet_routed = Fleet(
    name="Routed Serving",
    node=Node(name="H100", accelerator=hardware,
              accelerators_per_node=8, intra_node_bw=Q_("900 GB/s")),
    count=max((gpus_blended + 7) // 8, 1),
    fabric=NetworkFabric(name="IB", bandwidth=Q_("400 Gbps").to("GB/s")),
)

tco_routed = econ.solve(fleet=fleet_routed, duration_days=365)
tco_all_k8 = fleet_results[8]

savings = tco_all_k8.tco_usd - tco_routed.tco_usd
pct_savings = savings / tco_all_k8.tco_usd * 100 if tco_all_k8.tco_usd > 0 else 0

table(
    ["Strategy", "GPUs", "Annual TCO ($)"],
    [
        ["All queries -> GPT-4 K=8", fleet_objects[8].total_accelerators, f"${tco_all_k8.tco_usd:,.0f}"],
        ["70/30 routed", fleet_routed.total_accelerators, f"${tco_routed.tco_usd:,.0f}"],
    ]
)
info(Savings=f"${savings:,.0f} ({pct_savings:.0f}%)")

Strategy                   GPUs  Annual TCO ($)
───────────────────────────────────────────────
All queries -> GPT-4 K=8  43736    $455,181,289
70/30 routed              13320    $138,627,555
Savings:  $316,553,733 (70%)

Routing 70% of traffic to Llama-3.1-70B ($K=1$) reduces the fleet from 43,736 GPUs to 13,320 GPUs and cuts annual TCO from $455.2M to $138.6M, saving $316.6M (70%). The 30% of queries that still need full reasoning continue to require dedicated GPU capacity, and the routing classifier itself introduces latency and operational complexity. Routing contains the cost multiplier, while capital planning remains essential.

Next Steps

Geography is a Systems Variable: See how location changes the carbon cost of a serving fleet
Scaling to 1000 GPUs: Discover reliability costs when training the models that do the reasoning
Quantization: Not a Free Lunch: Learn when reducing precision helps with serving costs
Sensitivity Analysis: Sweep parameters to find which lever matters most for a deployment