Build LLMs from scratch.
From backpropagation by hand to transformers, reasoning, agents and MCP. Each step is a small exercise in plain Python and NumPy, solved and tested right in your browser.
20 sections · 0/94 solved
Read the theory
Each section starts with a short chapter, with references to the papers and books it comes from.
Write the code
Implement each piece yourself in Python and NumPy. Your code runs in your browser, nothing to install.
Pass the tests
Tests check your answer instantly, with hints when something is off. Progress is saved as you go.
Roadmap
- 0
NumPy & Math Foundations
0/5Vectorized thinking, broadcasting and numerically stable primitives every later block relies on.
Read the theory → - 1
Backpropagation by Hand
0/4Derive and code gradients yourself, then prove them right with gradient checking.
Read the theory → - 2
Loss Functions
0/4MSE, binary and multi-class cross-entropy from logits with their gradients, and the masked, label-smoothed loss of a language model.
Read the theory → - 3
Optimizers & Schedules
0/4SGD, momentum, Adam/AdamW, warmup + cosine decay, gradient clipping.
Read the theory → - 4
Neural Network Layers
0/5Embedding lookup and its backward pass, LayerNorm and RMSNorm, inverted dropout, and the weight initializations that keep deep networks stable.
Read the theory → - 5
Language Modeling Basics
0/4Bigram and MLP language models, temperature, top-k and top-p sampling.
Read the theory → - 6
Tokenization
0/3Train a byte-pair encoding tokenizer, then encode and decode with it.
Read the theory → - 7
Attention & Transformers
0/5Scaled dot-product attention, causal masking, multi-head attention, rotary embeddings, grouped-query attention and a full LLaMA-style block.
Read the theory → - 8
Training LLMs
0/5The training loop, batches from a token stream, gradient accumulation, FP16/BF16 mixed precision with loss scaling, and the arithmetic of compute and scaling laws.
Read the theory → - 9
Distributed Training
0/5Data, tensor and pipeline parallelism, ZeRO/FSDP sharding and mixed precision, simulated in NumPy; the libraries that run them at scale.
Read the theory → - 10
Case Study: OLMo 3
0/5A fully open recent LLM end to end, from GPUs, libraries and data to every layer, loss and hyperparameter of its pre-training.
Read the theory → - 11
Efficient Inference & Compression
0/5KV cache, beam search, speculative decoding, int8/int4 quantization, pruning and LoRA merging, to run open-weight models on local hardware.
Read the theory → - 12
Post-Training & Alignment
0/5SFT loss masking, LoRA, reward models and the RLHF pipeline, DPO, PPO-clip and GRPO; adapting a base model to a domain such as law, medicine or code.
Read the theory → - 13
Diffusion & Multimodal Models
0/5DDPM noise schedules and the denoising loss, ViT patch embeddings and CLIP's contrastive loss, the pieces behind image generation and vision-language models.
Read the theory → - 14
Reasoning & Test-Time Compute
0/5Self-consistency, best-of-N, Tree-of-Thoughts and MCTS over reasoning steps, process vs outcome reward models, self-correction and inference scaling laws.
Read the theory → - 15
Retrieval & Information Extraction
0/5Embeddings, cosine similarity, BM25, chunking and retrieval, then structured extraction of entities and fields from text.
Read the theory → - 16
Agents & Multi-Agent Systems
0/5Tool-call parsing, the ReAct loop, agent memory and teams of agents with roles (planner, coder, reviewer, tester), driven by a scripted fake LLM.
Read the theory → - 17
Code Generation & Verification
0/5Checking generated code with tests and static analysis, running candidates in a sandbox, AST checks for unsafe code, test generation and repair loops.
Read the theory → - 18
Evaluation & Safety
0/5Calibration and uncertainty (ECE, temperature scaling), LLM-as-a-judge and its biases, hallucination checks and prompt-injection defenses.
Read the theory → - 19
Model Context Protocol
0/5JSON-RPC 2.0, stdio framing, MCP's stateless per-request metadata (revision 2026-07-28), discovery and a minimal tool server.
Read the theory →