Chapter 18: Large Language Models

1. Autoregressive Language Modelling

A language model defines a probability distribution over sequences. Using the chain rule of probability:

Training minimises the cross-entropy loss (equivalently, maximises log-likelihood):

Perplexity = \(\exp(\mathcal{L})\) measures how “surprised” the model is on average. A perplexity of 20 means the model is roughly as uncertain as choosing uniformly among 20 tokens.

2. GPT Architecture (Decoder-Only Transformer)

GPT uses a decoder-only Transformer: the full Transformer decoder of Chapter 17, but with the cross-attention layer removed. A causal mask ensures each position only attends to previous tokens, maintaining the autoregressive property.

BERT's Masked Language Modelling: randomly mask 15% of tokens, predict the masked tokens using bidirectional context. This is a denoising objective: \(P(x_{\rm masked} \mid x_{\rm unmasked})\). BERT cannot generate text naturally (it sees future tokens); GPT cannot use future context for understanding.

3. Scaling Laws

Kaplan et al. (2020) and Hoffmann et al. (2022, “Chinchilla”) showed that language model loss follows a power law in the number of parameters \(N\), training tokens \(D\), and compute \(C\):

Chinchilla finding: for a fixed compute budget, it is better to train a smaller model on more data than a larger model on fewer tokens. Optimal allocation: \(D \approx 20 \times N\) tokens.

3.1 Emergent Abilities

Some capabilities (e.g., multi-step arithmetic, chain-of-thought reasoning, in-context learning) appear suddenly above a threshold model size — they are absent in small models and present in large ones without explicit training. This is called emergence and is an active area of research.

4. Training Pipeline: Pre-training, Fine-tuning, RLHF

1. Pre-training: Train on massive text corpus \(\mathcal{D}_{\rm pre}\) by minimising cross-entropy. Produces a model \(\pi_{\rm PT}\) that can complete text.
2. Supervised Fine-tuning (SFT): Fine-tune on high-quality demonstrations \(\{(prompt_i, response_i)\}\) of desired behaviour.
3. Reward Model Training: Collect human preference data: pairs of responses ranked by quality. Train a reward model \(r_\phi(x, y)\) using the Bradley-Terry preference model:
   \[ P(y_1 \succ y_2) = \sigma(r_\phi(x, y_1) - r_\phi(x, y_2)) \]
4. RLHF with PPO: Optimise the language model as a policy to maximise the reward model score, subject to a KL penalty from the SFT policy:
   \[ \mathcal{L}_{\rm RLHF}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}\!\left[r_\phi(x,y) - \beta\,\mathrm{KL}(\pi_\theta(\cdot|x) \| \pi_{\rm SFT}(\cdot|x))\right] \]
   The KL term prevents the model from exploiting the reward model with nonsensical outputs that score highly.

5. Tokenisation: Byte-Pair Encoding (BPE)

LLMs operate on tokens, not characters. BPE builds a vocabulary by iteratively merging the most frequent pair of adjacent tokens:

1. Start with character vocabulary.
2. Count all adjacent pairs in the corpus.
3. Merge the most frequent pair into a new token.
4. Repeat until vocabulary reaches target size (e.g., 50,257 for GPT-2).

BPE balances vocabulary size and sequence length. Common words become single tokens; rare words decompose into sub-word tokens, maintaining coverage without an infinite vocabulary.

6. In-Context Learning & Prompting

A surprising property of large LMs: they can learn new tasks from just a few examples in the prompt, without any gradient updates. Given \(k\) examples in context, the model predicts the next output:

7. GPT Architecture Diagram

8. Python: Tiny Character-Level Transformer

We implement a 2-layer character-level Transformer from scratch in NumPy, train it on a small text, and visualise: training loss, attention patterns, perplexity, scaling law curves, character frequency (BPE motivation), and temperature sampling effects.