The Literature Zoo
Cited Scalars for MFU, Chinchilla, Batch Size, and Communication
The Literature Zoo holds published anchors cited in the textbook appendices — MFU bands, Chinchilla ratios, critical-batch-size anchors, and communication overheads. Every entry is a Sourced scalar with structured Provenance (see Provenance).
ImportantNot hardware specs
Do not confuse Literature.Training.MfuHigh with a GPU datasheet field. Literature entries are cited teaching assumptions; silicon numbers live in Hardware. Operational run-overhead profiles live in Ops, and scenario scale-efficiency profiles live in Scenarios; provenance records the source, but the registry path records the category.
Training MFU bands
| Entry | Value | Description |
|---|---|---|
| MFU Training (Upper Bound) | 0.5 | Upper bound MFU for excellent large-model training runs. |
| MFU Inference (Batch 1) | 0.05 | MFU for single-request inference, heavily memory-bandwidth-bound. |
| MFU Inference (Batched) | 0.4 | Illustrative MFU upper bound for large-batch inference. |
| MFU Training (Lower Bound) | 0.3 | Lower bound MFU for well-optimized large-model training. |
Benchmark anchors
| Entry | Value | Description |
|---|---|---|
| EfficientNet-B4 ImageNet top-1 accuracy (fraction) | 0.829 | EfficientNet-B4 ImageNet top-1 (Tan & Le 2019, Table 2). |
| Llama 3 8B H100 ITL Lower Bound | 3.0 | Lower edge of the H100 Llama-family decode latency sanity envelope. |
| Llama 3 8B H100 ITL Upper Bound | 10.0 | Upper edge of the H100 Llama-family decode latency sanity envelope. |
| MobileNetV3-Large ImageNet top-1 accuracy (fraction) | 0.752 | MobileNetV3-Large 1.0 ImageNet top-1 (Howard et al. 2019, Table 3). |
| ResNet-50 A100 Training Throughput | 3200.0 | Well-tuned single-A100 ResNet-50 v1.5 training rate, mixed precision; NVIDIA measures about 2,100 to 3,400 img/s per GPU depending on framework. |
| ResNet-50 H100 Inference Offline Throughput per Accelerator | 89470.0 | Images per second per H100-SXM in MLPerf Inference v5.0 Offline (TensorRT, INT8 weights and inputs); 715,761 for eight accelerators divided by eight. |
| ResNet-50 H100 Inference Server p99 Latency (ms) | 10.86 | Measured 99th-percentile request latency in the same MLPerf Inference v5.0 Server run, under its 15 ms bound. |
| ResNet-50 H100 Inference Server Throughput per Accelerator | 81500.0 | Images per second per H100-SXM in MLPerf Inference v5.0 Server (p99 latency 10.86 ms under the 15 ms bound); 652,002 for eight accelerators divided by eight. |
| ResNet-50 H100 Training Throughput | 5000.0 | Single-accelerator ResNet-50/H100 throughput anchor for empirical sanity checks. |
| ViT-L/16 ImageNet top-1 accuracy (fraction) | 0.8776 | ViT-L/16 ImageNet top-1 after JFT-300M pre-training (Dosovitskiy et al. 2021, Table 2). |
Critical batch size anchors
| Entry | Value | Description |
|---|---|---|
| BERT critical batch size | 256.0 | Rounded critical batch-size anchor for BERT-scale training examples. |
| Default critical batch size | 1024.0 | Generic rounded critical batch-size anchor for first-pass training examples. |
| GPT-3 critical batch size | 4096.0 | Rounded critical batch-size anchor for GPT-3-scale training examples. |
Chinchilla anchors
| Entry | Value | Description |
|---|---|---|
| Chinchilla Approach 1 parameter-allocation exponent a | 0.5 | N_opt proportional to C^a from the minimum over training curves (Table 2; 10th-90th percentile 0.488-0.502). |
| Chinchilla Approach 1 token-allocation exponent b | 0.5 | D_opt proportional to C^b from the minimum over training curves (Table 2; 10th-90th percentile 0.501-0.512). |
| Chinchilla Approach 2 (IsoFLOP) parameter-allocation exponent a | 0.49 | N_opt proportional to C^a from IsoFLOP profiles (Table 2, Figure 3; 10th-90th percentile 0.462-0.534). |
| Chinchilla Approach 2 (IsoFLOP) token-allocation exponent b | 0.51 | D_opt proportional to C^b from IsoFLOP profiles (Table 2, Figure 3; 10th-90th percentile 0.483-0.529). |
| Chinchilla Approach 3 (parametric) parameter-allocation exponent a | 0.46 | N_opt proportional to C^a implied by the parametric loss fit (Table 2; a = beta/(alpha+beta), Eq. 4, from the unrounded fit; the rounded 0.34 and 0.28 give 0.45). |
| Chinchilla Approach 3 (parametric) token-allocation exponent b | 0.54 | D_opt proportional to C^b implied by the parametric loss fit (Table 2; b = alpha/(alpha+beta), Eq. 4, from the unrounded fit; the rounded 0.34 and 0.28 give 0.55). |
| Training Compute Constant (C ≈ 6PD) | 6.0 | Training FLOPs multiplier (6PD): 2 forward + 4 backward FLOPs per parameter per token. |
| Decode Compute Constant (2P) | 2.0 | Autoregressive decode FLOPs multiplier (2P): 2 forward FLOPs per parameter per token. |
| Gopher training compute budget (FLOPs) | 5.76e+23 | One “Gopher unit” of training compute, the budget at which Approach 1 places a 67B-parameter model (Table 3). |
| IsoFLOP-projected compute-optimal parameters at the Gopher budget | 63000000000.0 | Approach 2 projection of the loss-optimal model size for the Gopher budget, marked 63B on Figure 3 (center). |
| IsoFLOP-projected compute-optimal training tokens at the Gopher budget | 1400000000000.0 | Approach 2 projection of the loss-optimal training tokens for the Gopher budget, marked 1.4T on Figure 3 (right). |
| Chinchilla IsoFLOP study, largest budget (FLOPs) | 3e+21 | Largest of the nine fixed training-FLOP budgets in the IsoFLOP study (Section 3.2; budgets 6e18, 1e19, 3e19, 6e19, 1e20, 3e20, 6e20, 1e21, 3e21). |
| Chinchilla IsoFLOP study, smallest budget (FLOPs) | 6e+18 | Smallest of the nine fixed training-FLOP budgets in the IsoFLOP study (Section 3.2). |
| Chinchilla parametric loss, parameter coefficient A | 406.4 | Coefficient of the model-size term A/N^alpha in the Approach 3 fit (Appendix D.2, Eq. 10). |
| Chinchilla parametric loss, parameter exponent alpha | 0.34 | Exponent on parameter count N in the Approach 3 fit (Appendix D.2, Eq. 10). |
| Chinchilla parametric loss, data coefficient B | 410.7 | Coefficient of the data term B/D^beta in the Approach 3 fit (Appendix D.2, Eq. 10). |
| Chinchilla parametric loss, data exponent beta | 0.28 | Exponent on training tokens D in the Approach 3 fit (Appendix D.2, Eq. 10). |
| Chinchilla parametric loss, irreducible term E | 1.69 | Entropy-of-text term of the Approach 3 fit L = E + A/N^alpha + B/D^beta (Appendix D.2, Eq. 10). |
| Compute-Optimal Token Ratio | 20.0 | Optimal training tokens per parameter (D ≈ 20P). |
Communication
| Entry | Value | Description |
|---|---|---|
| AllReduce factor | 2.0 | Ring AllReduce communication multiplier (2×). |
Python Access
import mlsysim
chinchilla_ratio = mlsysim.Literature.Chinchilla.TokensPerParam
mfu_high = mlsysim.Literature.Training.MfuHigh
ring_factor = mlsysim.Literature.Communication.RingAllreduceFactorWhen calibrating η in accuracy.qmd, cross-check against these literature bands before picking solver efficiency kwargs.