Batch Size Balancer

Push throughput to the edge of OOM.

Use Up and Down to change batch size. Serve the queue quickly without exceeding GPU memory.

batch 1 · score 0 ↑ / ↓ adjust batch size · keep memory under 100% · R retry

How to play

Increase batch size to raise throughput, but watch the memory meter. Larger batches process the queue faster; if queue pressure plus batch memory exceeds 100%, the run OOMs immediately and your score stops there. Find the largest safe batch size as demand changes.

The Systems Concept

Spikes in activation memory are the penalty. Larger batches improve hardware utilization, but they also reserve more memory per step. The optimal batch size is not simply “as large as possible”; it is the largest batch that keeps throughput high without crossing the memory limit.

---

Part of MLSysBook Playground. Found a bug? Report an issue.