image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/
Unit 03 · Chapter 6

Transformers

Transformers01

Generative Pre-trained Transformer (GPT)

ChatGPT writes its answer one word at a time: each word is a guess at what comes next, given everything before it

cheer0 cheer0 for1 old2 notre3 dame

Guessing the next word from the words before it is called next-token prediction, and it's all a GPT does: a Generative (it writes) Pre-trained (on lots of text) Transformer

Monday's self-attention mixes every word's context into its embedding, but it doesn't write anything. Today: from that layer to a GPT that writes the fight song

Transformers · GPT02

Next-Token Prediction

The first line of the chorus is already 5 guesses to learn from:

the words so farthe next word
cheer0cheer0
cheer0 cheer0for1
cheer0 cheer0 for1old2
cheer0 cheer0 for1 old2notre3
cheer0 cheer0 for1 old2 notre3dame4

The whole chorus is 50 training examples, one for every word after the first, and nobody had to label anything: the answer is just the next word

Using raw text like this, without needing to label ground-truth data, is called self-supervised learning

Transformers · GPT03

Self-Attention

\red{\mathrm{Attention}}(Q, K, V) = \gold{\mathrm{softmax}}\Bp{\lk{purple}{\frac{Q K\T}{\sqrt{d_k}}}} V \red{\mathrm{Attention}}(Q, K, V) = \gold{\mathrm{softmax}}\Bp{\lk{purple}{\frac{Q K\T}{\sqrt{d_k}}}} V image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ wake up the echoes cheering her name key (attending to) wake up the echoes cheering her name query (attending from) 0.07 0.27 0.12 0.18 0.11 0.16 0.11 0.26 0.09 0.15 0.12 0.15 0.09 0.15 0.32 0.05 0.12 0.12 0.17 0.08 0.14 0.16 0.27 0.07 0.37 0.05 0.02 0.05 0.01 0.18 0.10 0.03 0.07 0.51 0.10 0.07 0.08 0.16 0.05 0.15 0.31 0.18 0.02 0.09 0.12 0.02 0.10 0.51 0.14
Transformers · GPT04

The Whole Model

cheer cheer for old notre Tokenizer Embeddings + Positional Encoding Transformer Block 1 Transformer Block 2 ⋮ Transformer Block N × N Layer Normalization Linear Softmax dame next word: block input (one row per word) Layer Normalization Masked Self-Attention softmax(QKᵀ/√dₖ + M) V + Layer Normalization Feed-Forward (MLP) + block output residual residual
Transformers · GPT05

Self-Attention and Word Order

Shuffle our line: the7 cheering9 echoes8 name11 her10 wake5 up6. Every token gets the same output, just in a different row

Self-attention treats a sentence as a bag of words: wake up the echoes = the echoes wake up

wake up the echoes wake up the echoes wake 0.26 0.37 0.28 0.09 up 0.21 0.12 0.62 0.06 the 0.07 0.06 0.21 0.67 echoes 0.26 0.40 0.32 0.02 the echoes wake up the echoes wake up the 0.21 0.67 0.07 0.06 echoes 0.32 0.02 0.26 0.40 wake 0.28 0.09 0.26 0.37 up 0.62 0.06 0.21 0.12 shuffle wake → up is 0.37 in both: every weight is the same, just moved to another row and column one row per query, one column per key (no positions)
Transformers · GPT06

Positional Embeddings

Same problem in the chorus: it starts cheer0 cheer0, and both get exactly the same output. Attention can't tell the first cheer0 from the second

RNN: one word at a time cheer RNN cell step 1 cheer RNN cell step 2 for RNN cell step 3 c 1 c 2 c 3 Attention: every word at once cheer cheer for cheer 0.34 0.34 0.32 v₁ cheer 0.34 0.34 0.32 v₂ for 0.41 0.41 0.17 v₃ the two cheer rows are the same, so v₁ = v₂ one row per query, one column per key

An RNN got the order for free (it read the first cheer0 first). Attention is computed in parallel, so each word's position is lost

Transformers · Position07

Positional Embeddings

So we put the position back in ourselves: each token's position goes through sine and cosine functions, each column with a different frequency, and the result is added to its embedding

P_{pos,\,2i} = \sin\bp{pos \,/\, 10000^{2i/d}} P_{pos,\,2i+1} = \cos\bp{pos \,/\, 10000^{2i/d}} X = E_x + P P_{pos,\,2i} = \sin\bp{pos \,/\, 10000^{2i/d}}; P_{pos,\,2i+1} = \cos\bp{pos \,/\, 10000^{2i/d}}; X = E_x + P

This is referred to as Sinusoidal Position Encoding

E[x]: the word P: its position X = E[x] + P cheer position 0 0.04 −0.73 −1.33 −0.37 0.54 0.27 −1.40 0.06 + 0.00 1.00 0.00 1.00 0.00 1.00 0.00 1.00 = 0.04 0.27 −1.33 0.63 0.54 1.27 −1.40 1.06 cheer position 1 0.04 −0.73 −1.33 −0.37 0.54 0.27 −1.40 0.06 + 0.84 0.54 0.10 1.00 0.01 1.00 0.00 1.00 = 0.88 −0.19 −1.23 0.62 0.55 1.27 −1.39 1.06

NOTE: Modern LLMs use something called RoPE that doesn't actually modify the original embeddings

Transformers · Position08

Positional Embeddings

Every position gets its own pattern: the first columns flip every word (telling neighbors apart), the last ones barely move (telling the start of the chorus from the end), like the second, minute and hour hands of a clock

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0 5 10 15 20 25 30 35 40 45 50 position in the chorus sin 0 cos 0 sin 1 cos 1 sin 2 cos 2 sin 3 cos 3 positional encodings, 51 positions × 8 columns 0 10 20 30 40 50 position in the chorus −1 0 1 column 0: sin(pos) column 2: sin(pos / 10) column 4: sin(pos / 100)

So the two cheer0s now go in as different positions

Transformers · Position09

Training on Every Word at Once

We don't want to run the model 50 times. Instead we give it the whole chorus and ask every row to guess the word after it, 50 guesses in one pass:

incheer0 cheer0 for1 old2 notre3…
guesscheer0 for1 old2 notre3 dame4…

Unfortunately, attention lets every word look at every other word, so the row for notre3 could just look at dame4 and copy it: the model would learn to cheat

Transformers · Masking10

Masking

Before the softmax in self-attention, every score where a token looks at a later token is set to −∞

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ cheer cheer for old notre dame cheer cheer for old notre dame 0.05 −∞ −∞ −∞ −∞ −∞ 0.50 −0.42 −∞ −∞ −∞ −∞ −0.12 0.47 0.41 −∞ −∞ −∞ −0.40 −0.48 0.37 −0.12 −∞ −∞ −2.16 −2.13 0.66 0.06 −0.51 −∞ −2.06 −1.98 0.27 −0.61 −0.90 0.23 scores Q Kᵀ / √8, masked cheer cheer for old notre dame cheer cheer for old notre dame 1.00 0 0 0 0 0 0.71 0.29 0 0 0 0 0.22 0.40 0.38 0 0 0 0.19 0.17 0.40 0.25 0 0 0.03 0.03 0.50 0.28 0.16 0 0.03 0.04 0.35 0.14 0.11 0.33 attention weights (softmax of each row)

NOTE: e−∞ = 0, so notre3 weighs the words up to itself (0.03, 0.03, 0.50, 0.28, 0.16), and dame4 gets 0

Transformers · Masking11

Simple Decoder/Generative Transformer Diagram

The mask lives in the block's attention. One transformer block, at GPT-2 small's size:

Masked Self-Attention 12 heads, 64 columns each <T, 768> Join the 12 heads, × Ω O weights · V <T, 64> per head softmax (each row) attention weights <T, T> + M −∞ above the diagonal Q Kᵀ / √64 scores <T, T> <T, 64> Q = HΩ Q <T, 64> K = HΩ K <T, 64> V = HΩ V H = LN(X) <T, 768> to the residual + Feed-Forward each word on its own · Ω₂ <T, 768> GELU <T, 3072> (ours: ReLU) H Ω₁ <T, 3072> H = LN(X) <T, 768> to the residual + -2 2 2 ReLU GELU GELU dips just below 0 and bends smoothly at 0
Transformers · The Transformer Block12

Residual Connections, Again

Monday we added Z back onto X. A transformer block does that twice, once around each half:

X \leftarrow X + \lk{red}{\red{\mathrm{Attention}}\bp{\mathrm{LN}(X)}} X \leftarrow X + \lk{purple}{\purple{\mathrm{FFN}}\bp{\mathrm{LN}(X)}} X \leftarrow X + \lk{red}{\red{\mathrm{Attention}}\bp{\mathrm{LN}(X)}}; X \leftarrow X + \lk{purple}{\purple{\mathrm{FFN}}\bp{\mathrm{LN}(X)}} X Layer norm Masked Self-Attention + Layer norm Feed-Forward + X residual residual X ← X + Attention(LN(X)) X ← X + FFN(LN(X))

Each half only adds a change to every word's row, the word itself is still there: a residual connection

A block's output has exactly the same shape as its input, a row of d numbers per word

Transformers · The Transformer Block13

Layer Normalization

Before each half, every word's row is rescaled to mean 0 and standard deviation 1: layer normalization

\mathrm{LN}(x) = \frac{x - \mathrm{mean}(x)}{\mathrm{std}(x)} \mathrm{LN}(x) = \frac{x - \mathrm{mean}(x)}{\mathrm{std}(x)}
dame4x₁x₂x₃x₄x₅x₆x₇x₈meanstd
before−1.86−0.17−0.090.42−0.471.730.48−0.56−0.060.96
after−1.87−0.11−0.020.50−0.421.860.57−0.5201

Why: every block adds onto the rows, so their scale drifts as the blocks stack up, and big numbers make softmax give nearly all attention to one token

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ cheer cheer for old notre 0.0 0.5 1.0 attention from notre 0.03 0.03 0.50 0.28 0.16 rows as they are cheer cheer for old notre 0.00 0.00 0.91 0.08 0.01 rows 2× bigger: scores × 4 cheer cheer for old notre 0.00 0.00 1.00 0.00 0.00 rows 4× bigger: scores × 16
Transformers · The Transformer Block14

The Feed-Forward Sublayer

Attention mixes the words together. Then a small feed-forward network works on each word by itself:

\purple{\mathrm{FFN}}(H) = \gold{\mathrm{ReLU}}\bp{\lk{purple}{H \Omega_1}}\, \Omega_2 \purple{\mathrm{FFN}}(H) = \gold{\mathrm{ReLU}}\bp{\lk{purple}{H \Omega_1}}\, \Omega_2

It's the feed-forward network from a few weeks ago, run on every word's row on its own

cheer h₁ <8> · Ω₁ (8 → 32) ReLU · Ω₂ (32 → 8) out₁ <8> cheer h₂ <8> · Ω₁ (8 → 32) ReLU · Ω₂ (32 → 8) out₂ <8> for h₃ <8> · Ω₁ (8 → 32) ReLU · Ω₂ (32 → 8) out₃ <8> old h₄ <8> · Ω₁ (8 → 32) ReLU · Ω₂ (32 → 8) out₄ <8> the same Ω₁ and Ω₂ for every word, and no arrows between the rows: each word on its own
Transformers · The Transformer Block15

Stacking Blocks

A block gives back rows of the same shape it was given, so its output can go straight into another block, and another after that:

X <T, 768> Block 1 Attention Feed-Forward Block 2 Attention Feed-Forward · · · Block 12 Attention Feed-Forward Layer norm Scores <T, 768> <T, 768> every block gives back the same shape it was given, so they stack: GPT-2 small has 12, GPT-3 96

Each block mixes the words again, so later blocks work on rows that already know about their neighbors

Transformers · The Transformer Block16

Our Tiny GPT in PyTorch

Our tiny GPT: the chorus as 41 different words, 8 numbers per word, and the positional encoding from earlier, in the same colours

\lk{blue}{P_{pos,\,2i}} = \sin\bp{\lk{gold}{pos \,/\, 10000^{2i/d}}} \lk{purple}{P_{pos,\,2i+1}} = \cos\bp{\lk{gold}{pos \,/\, 10000^{2i/d}}} \lk{blue}{P_{pos,\,2i}} = \sin\bp{\lk{gold}{pos \,/\, 10000^{2i/d}}}; \lk{purple}{P_{pos,\,2i+1}} = \cos\bp{\lk{gold}{pos \,/\, 10000^{2i/d}}}
import torch
from ai2ai import load
song = load("fight-song")
print(song)
def pos_enc(T, d=8):
pos = torch.arange(T)[:, None]
i = torch.arange(0, d, 2)
angle = pos / 10000 ** (i / d)
P = torch.zeros(T, d)
P[:, 0::2] = torch.sin(angle)
P[:, 1::2] = torch.cos(angle)
return P
Output
51 words, 41 different ones
Transformers · The Transformer Block17

Layer Norm and the Weights

One transformer block's weights, loaded before training, coloured by the part of the model that uses them, plus the layer norm (grey)

x <5> Embeddings + position E[x] + P Layer norm Masked self-attention softmax(QKᵀ/√d + M) V + Layer norm Feed-forward ReLU(H Ω₁) Ω₂ + Layer norm Scores H Eᵀ ŷ <5, 41> softmax
W = load("tiny-gpt", trained=False)
def norm(X):
mean = X.mean(1, keepdim=True)
var = ((X - mean) ** 2).mean(1, keepdim=True)
return (X - mean) / var.sqrt()
Output
E   (41, 8)
Q   (8, 8)
K   (8, 8)
V   (8, 8)
O   (8, 8)
F1  (8, 32)
F2  (32, 8)
Transformers · The Transformer Block18

The Output Layer

x <5> Embeddings + position E[x] + P Layer norm Masked self-attention softmax(QKᵀ/√d + M) V + Layer norm Feed-forward ReLU(H Ω₁) Ω₂ + Layer norm Scores H Eᵀ ŷ <5, 41> softmax
def gpt(x, W, mask=True):
T = len(x)
X = W.E[x] + pos_enc(T)
H = norm(X)
Q, K, V = H @ W.Q, H @ W.K, H @ W.V
S = Q @ K.T / 8 ** 0.5
if mask:
S = S + torch.full((T, T), float("-inf")).triu(1)
attention_scores = torch.softmax(S, 1)
context_vectors = attention_scores @ V
X = X + context_vectors @ W.O
X = X + torch.relu(norm(X) @ W.F1) @ W.F2
return norm(X) @ W.E.T
x = song.ids[:5] # cheer cheer for old notre
probs = torch.softmax(gpt(x, W)[-1], 0)
guess = probs.argmax().item()
dame = song.vocab.index("dame")
p = round(probs[guess].item(), 2)
print(song.decode([guess]), p)
print(song.decode([dame]), round(probs[dame].item(), 4))
Output
[odds] 0.26
[dame] 0.0008

Untrained: 26% on odds24, 0.08% on dame4, a thirtieth of guessing evenly (2.4%)

Transformers · Writing the Fight Song19

Training Our Tiny GPT

The loss is cross-entropy, but with 41 classes (tokens in our vocabulary)

\lk{red}{ℒ = -\frac{1}{50} \sum_{t=1}^{50} \log \yhat_t[x_{t+1}]} \lk{blue}{Ω \leftarrow \mathrm{Adam}\bp{Ω,\; \frac{\partial ℒ}{\partial Ω}}} \lk{red}{ℒ = -\frac{1}{50} \sum_{t=1}^{50} \log \yhat_t[x_{t+1}]}; \lk{blue}{Ω \leftarrow \mathrm{Adam}\bp{Ω,\; \frac{\partial ℒ}{\partial Ω}}}
import torch.nn.functional as F
opt = torch.optim.Adam(W.params(), lr=0.03)
x, y = song.ids[:-1], song.ids[1:] # y: the next word
for step in range(301):
loss = F.cross_entropy(gpt(x, W), y)
opt.zero_grad()
loss.backward()
opt.step()
if step % 100 == 0:
print(f"step {step} loss {loss.item():.3f}")
Output
step 0  loss 7.513
step 100  loss 0.077
step 200  loss 0.024
step 300  loss 0.013

From 7.513 to 0.013 in 300 steps of Adam

Transformers · Writing the Fight Song20

Generating Text

Pick the most likely word, add it to the end, and repeat:
This is called auto-regressive generation with greedy decoding

\lk{gold}{\mathrm{scores} = \mathrm{GPT}(x_1, \dots, x_T)} \lk{blue}{x_{T+1} = \arg\max\; \mathrm{scores}_T} \lk{gold}{\mathrm{scores} = \mathrm{GPT}(x_1, \dots, x_T)}; \lk{blue}{x_{T+1} = \arg\max\; \mathrm{scores}_T}
def generate(W, start, n, mask=True):
x = song.encode(start)
for _ in range(n): # greedy decoding
scores = gpt(torch.tensor(x), W, mask)
x.append(scores[-1].argmax().item())
return song.decode(x)
print(generate(W, "cheer cheer for old notre", 46))
Output
[cheer, cheer, for, old, notre, dame, wake, up, the, echoes, cheering, her, name, send, a, volley, cheer, on, high, shake, down, the, thunder, from, the, sky, what, though, the, odds, be, great, or, small, old, notre, dame, will, win, over, all, while, her, loyal, sons, and, daughters, march, on, to, victory]

Our GPT writes the fight song. It memorized 51 words (GPT-3 learned from about 300 billion tokens), but it's the same loop that writes every ChatGPT answer

Transformers · Writing the Fight Song21

Without the Mask

Same model, same training, no mask: the red part is gone

\red{\mathrm{Attention}}(Q, K, V) = \gold{\mathrm{softmax}}\Bp{\lk{purple}{\frac{Q K\T}{\sqrt{d_k}}} \lk{red}{+ M}} V \red{\mathrm{Attention}}(Q, K, V) = \gold{\mathrm{softmax}}\Bp{\lk{purple}{\frac{Q K\T}{\sqrt{d_k}}} \lk{red}{+ M}} V
Wn = load("tiny-gpt-nomask") # trained without it
start = "cheer cheer for old notre"
print(generate(Wn, start, 46, mask=False))
Output
[cheer, cheer, for, old, notre, dame, shake, a, volley, up, the, wake, up, the, wake, up, the, wake, up, dame, wake, up, volley, on, be, great, a, volley, cheer, while, notre, dame, will, win, win, win, win, win, win, win, win, win, notre, dame, will, win, win, notre, dame, and, dame]

The loss gets just as low, because during training every row could peek at the answer.

When it writes, there's no next word to peek at, so it falls apart after dame4

Transformers · Writing the Fight Song22