Build it
Reading is not understanding. Understanding is being able to reproduce. Every project ships a runnable build.py you assemble yourself.
esc to close
A code companion · 36 projects · 36 chapters
Under the
Hood.
Build Every Layer of a Large Language Model from Scratch
Build it. Break it. Measure it.— not a tagline, the entire epistemology of the book.
The Method
Break it
A working system tells you what it does. A broken system tells you why it has to be that way. Each project sabotages itself on purpose.
Measure it
Intuition without numbers is the most expensive failure mode in machine learning. Every claim comes with captured outputs.
The Curriculum
36 projects across 9 parts. Each builds on the previous. Read in order, or jump to what you need.
- 01Foundations4 projectsThe Learning MachinePredicting the Next CharacterBuilding a Tokenizer+1
- 02Building a GPT4 projectsYour GPT from a Blank FileFrom Prototype to nanoGPTThe Details That Matter+1
- 03Training at Scale4 projectsPretraining on the Real WebData Curation and ContaminationTraining Debugging: Spikes, NaNs, and Profiling.+1
- 04Inference and Serving5 projectsFast Inference: The KV Cache.Speculative Decoding.Grouped Query Attention.+2
- 05Scaling and Experimentation3 projectsMixture of ExpertsScaling LawsAutonomous Experimentation
- 06Aligning and Evaluating4 projectsFine-Tuning and Instruction TuningEvaluation Methodology.Reward Models and RLHF+1
- 07Reasoning, Tools, and Retrieval4 projectsTest-Time Reasoning (CoT, Self-Consistency, Best-of-N).Tool Use and Function Calling.Quantization and Deployment+1
- 08Beyond the Transformer3 projectsMultimodal: A Tiny Vision-Language Model.Non-Transformer Architectures (Mamba, RWKV).Diffusion and Non-Autoregressive Decoding
- 09Modular Composition5 projectsLayer Freezing and TransferFusing Independently Trained SpecialistsThe Interface Specification+2
This is the code. The book is the why.
This site hosts every runnable project, every break_it experiment, and every captured output — free. The book itself (the long-form derivations, the prose, the full chapters) is distributed separately.