Policy Gradients

How policy-gradient methods improve a policy directly: the policy gradient theorem, the log-derivative trick, REINFORCE, baselines, advantages and GAE.

1 · LLM Training Pipeline

Building Blocks
Transformer Architecture
Attention Variants
Model Families & Enc-Dec
Data: Curation & Synthetic
Pre-training
Optimization & Stability
Post-training
Finetuning & SFT
Mixture of Experts
State-Space Models (Mamba)
Long-Context Scaling

2 · RL Foundations

Reinforcement Learning
Markov Decision Processes
Exploration vs Exploitation
Q-Learning
Policy Gradients

3 · Alignment / RLHF

Reward Models
RLHF
PPO
DPO
GRPO
RLVR
Constitutional AI & RLAIF
Safety & Red-Teaming
RLCD & System One Models

4 · Reasoning & Inference

Decoding & Sampling
Reasoning Model Training
Speculative Decoding
Knowledge Distillation
Quantization & Serving

5 · Evaluation

Evaluation Methodology
LLM Evaluation Metrics

6 · Multimodal & Diffusion

Multimodal Models
Diffusion Models

7 · Applied LLMs

In-Context Learning
Embedding & Retrieval Models
Retrieval-Augmented Generation
Agents & Tool Use
Prompt Injection & Security