Skip to main content

How Much Memory Does Llama-3 70B Actually Need?

serving
memory
llm
A 3-minute calculation with mlsysim that answers the question every ML engineer asks.
Published

April 1, 2026

Every ML engineer eventually confronts the same capacity problem: whether Llama-3.1-70B (70.6B parameters) fits on the available hardware.

The answer depends on three quantities: precision, KV cache, and batch size. The calculation takes 30 seconds.

The Weights

import mlsysim

llama70b = mlsysim.Models.Language.Llama3_70B

# FP16: 2 bytes per parameter
fp16_size = llama70b.size_in_bytes()
print(f"FP16 weights: {fp16_size.to('GB'):~.1f}")

# INT8: 1 byte per parameter
int8_size = llama70b.size_in_bytes(mlsysim.ureg("1 byte"))
print(f"INT8 weights: {int8_size.to('GB'):~.1f}")

# INT4: 0.5 bytes per parameter
int4_size = llama70b.size_in_bytes(mlsysim.ureg("0.5 byte"))
print(f"INT4 weights: {int4_size.to('GB'):~.1f}")
FP16 weights: 141.2 GB
INT8 weights: 70.6 GB
INT4 weights: 35.3 GB

Result: 141.2 GB in FP16, 70.6 GB in INT8, and 35.3 GB in INT4.

An H100 or A100 has 80 GiB (85.9 GB), so Llama-3.1-70B in FP16 does not fit on one GPU. Serving it requires either tensor parallelism (\(\text{TP}=2\)) or quantization to INT8/INT4.

The KV Cache

Weights are only part of the total. Each active request also needs a KV cache:

from mlsysim.physics import calc_kv_cache_size

head_dim = llama70b.hidden_dim // llama70b.heads
kv_per_request = calc_kv_cache_size(
    n_layers=llama70b.layers,
    n_heads=llama70b.kv_heads,
    head_dim=head_dim,
    seq_len=4096,
    batch_size=1,
    bytes_per_elem=2,
)
print(f"KV cache per request (4K context): {kv_per_request.to('GB'):~.2f} ({kv_per_request.to('MB'):~.0f})")
KV cache per request (4K context): 1.34 GB (1342 MB)

At 1.34 GB (1,342 MB) per request across 80 layers and 8 GQA KV heads (128-dim per head), an 80 GiB (85.9 GB) H100 serving INT4 Llama-3.1-70B has:

  • 35.3 GB for weights
  • 50.6 GB (50,599 MB) remaining for KV cache
  • 50.6 GB / 1.34 GB \(\approx\) 37 concurrent requests max

At 32K context, the KV cache grows to 10.74 GB per request, which leaves room for at most 4 concurrent requests.

The Full Picture

from mlsysim.solvers import ServingModel

result = ServingModel().solve(
    model=mlsysim.Models.Language.Llama3_70B,
    hardware=mlsysim.Hardware.Cloud.H100,
    seq_len=4096,
    batch_size=1,
    precision="fp16",
)
print(f"Feasible:    {result.feasible}")
print(f"Memory util: {result.memory_utilization:.1%}")
Feasible:    False
Memory util: 165.9%

The conclusion: A 70.6B-parameter model needs more than 141.2 GB. It needs 141.2 GB + (\(\text{KV cache} \times \text{concurrent requests}\)). At production batch sizes and long contexts, the KV cache can consume more memory than the weights.

What To Do About It

Strategy Memory Impact Trade-off
INT4 quantization \(4\times\) smaller weights (141.2 GB \(\to\) 35.3 GB) ~2–5% accuracy loss
GQA (built into Llama-3) \(8\times\) smaller KV cache (64 \(\to\) 8 KV heads) None (architectural)
KV cache INT8 \(2\times\) smaller KV cache (1.34 GB \(\to\) 0.67 GB at 4K) Negligible quality loss
Tensor parallelism (\(\text{TP}=2\)) Split across 2 GPUs (70.6 GB weights/GPU) Adds NVLink communication
PagedAttention (vLLM) Eliminates KV fragmentation ~20–40% more concurrent requests

Reproducing the Calculation

pip install mlsysim
mlsysim serve Llama3_70B H100 --seq-len 4096 --batch-size 1

This analysis was computed with mlsysim, a first-principles analytical modeling framework for ML systems. All specifications are traceable to hardware datasheets and model cards.

Back to top