Learning Resources

TinyTorch teaches you to build ML systems. These resources help you understand the why behind what you’re building.

Companion Textbooks

TinyTorch: From Tensors to Transformers

Engineering Deep Learning Systems from Scratch (available at mlsysbook.ai/tinytorch/)

The complete architectural reference textbook. While this Lab Guide provides hands-on modules and exercise notebooks, the companion textbook delivers complete reference implementations, formal systems invariants, memory traces, napkin-math budgets, and production bridges.

What it teaches: Full framework architecture from first principles: CPython tensor internals, DAG reverse-mode tape autograd, AdamW moment buffers, BPE tokenization pipelines, multi-head attention math, KV caching state machines, INT8 quantization grids, and compilation bridges to PyTorch Inductor and OpenAI Triton.

When to use it: Consult whenever you want to inspect a complete reference solution, trace an exact execution path through memory, or understand how a subsystem connects to production GPU infrastructure.

Machine Learning Systems

mlsysbook.ai by Prof. Vijay Janapa Reddi (Harvard University)

The parent ML Systems curriculum. The ML Systems book provides the broad theoretical depth and production context behind modern machine learning infrastructure.

What it teaches: Systems engineering for production ML—memory hierarchies, hardware accelerators, distributed training algorithms, and deployment strategies.

When to use it: Read in parallel with TinyTorch to understand the high-level landscape of production ML infrastructure.

Related Academic Courses

• CS 329S: Machine Learning Systems Design (Stanford) Production ML systems and deployment

• TinyML and Efficient Deep Learning (MIT 6.5940) Edge computing, model compression, and efficient ML

• CS 249r: Tiny Machine Learning (Harvard) TinyML systems and resource-constrained ML

• CS 231n: Convolutional Neural Networks (Stanford) Computer vision - complements TinyTorch Module 09

• CS 224n: Natural Language Processing (Stanford) Transformers and NLP - complements TinyTorch Modules 10-13

Other Textbooks

• Deep Learning by Goodfellow, Bengio, Courville Mathematical foundations behind what you implement in TinyTorch

• Hands-On Machine Learning by Aurélien Géron Practical implementations using established frameworks

Minimal Frameworks

Alternative approaches to building ML from scratch:

• micrograd by Andrej Karpathy Autograd in 100 lines. Perfect 2-hour intro before TinyTorch.

• nanoGPT by Andrej Karpathy Minimalist GPT implementation. Complements TinyTorch Modules 12-13.

• tinygrad by George Hotz Performance-focused educational framework with GPU acceleration.

Production Framework Internals

• PyTorch Internals by Edward Yang How PyTorch actually works under the hood

• PyTorch: Extending PyTorch Custom operators and autograd functions

Ready to start? See the Quick Start to set up in about five minutes and begin Module 01.