CSE 30124 · Introduction to Artificial Intelligence William Theisen · University of Notre Dame
Transformers01
Generative Pre-trained Transformer (GPT)
ChatGPT writes its answer one word at a time: each word is a guess at what comes next, given everything before it
cheer0cheer0for1old2notre3
Guessing the next word from the words before it is called , and it's all a GPT does: a Generative (it writes) Pre-trained (on lots of text) Transformer
Monday's self-attention mixes every word's context into its embedding, but it doesn't write anything. Today: from that layer to a GPT that writes the fight song
Transformers · GPT02
Next-Token Prediction
The first line of the chorus is already 5 guesses to learn from:
the words so far
the next word
cheer0
cheer0
cheer0cheer0
for1
cheer0cheer0for1
old2
cheer0cheer0for1old2
notre3
cheer0cheer0for1old2notre3
dame4
The whole chorus is 50 training examples, one for every word after the first, and nobody had to label anything: the answer is just the next word
Using raw text like this, without needing to label ground-truth data, is called
Transformers · GPT03
Self-Attention
\red{\mathrm{Attention}}(Q, K, V) = \gold{\mathrm{softmax}}\Bp{\lk{purple}{\frac{Q K\T}{\sqrt{d_k}}}} V
Q = XωQ, the : what this token is looking for
K = XωK, the : what it offers to be matched against
V = XωV, the : what it passes along
Transformers · GPT04
The Whole Model
Transformers · GPT05
Self-Attention and Word Order
Shuffle our line: the7cheering9echoes8name11her10wake5up6. Every token gets the same output, just in a different row
Self-attention treats a sentence as a bag of words: wake up the echoes = the echoes wake up
Transformers · GPT06
Positional Embeddings
Same problem in the chorus: it starts cheer0cheer0, and both get exactly the same output. Attentioncan't tell the first cheer0 from the second
An RNN got the order for free (it read the first cheer0 first). Attention is computed , so each word's position is lost
Transformers · Position07
Positional Embeddings
So we put the position back in ourselves: each token's position goes through functions, each column with a different frequency, and the result is added to its embedding
This is referred to as Sinusoidal Position Encoding
NOTE: Modern LLMs use something called RoPE that doesn't actually modify the original embeddings
Transformers · Position08
Positional Embeddings
Every position gets its own pattern: the first columns flip every word (telling neighbors apart), the last ones barely move (telling the start of the chorus from the end), like the second, minute and hour hands of a clock
So the two cheer0s now go in as different positions
Transformers · Position09
Training on Every Word at Once
We don't want to run the model 50 times. Instead we give it the whole chorus and ask every row to guess the word after it, 50 guesses in one pass:
in
cheer0cheer0for1old2notre3
…
guess
cheer0for1old2notre3dame4
…
Unfortunately, attention lets every word look at every other word, so the row for notre3 could just look at dame4 and copy it: the model would learn to cheat
Transformers · Masking10
Masking
Before the softmax in self-attention, every score where a token looks at a later token is set to −∞
NOTE:e−∞ = 0, so notre3 weighs the words up to itself (0.03, 0.03, 0.50, 0.28, 0.16), and dame4 gets 0
Transformers · Masking11
Simple Decoder/Generative Transformer Diagram
The mask lives in the block's attention. One transformer block, at GPT-2 small's size:
Transformers · The Transformer Block12
Residual Connections, Again
Monday we added Z back onto X. A transformer block does that twice, once around each half:
X \leftarrow X + \lk{red}{\red{\mathrm{Attention}}\bp{\mathrm{LN}(X)}}X \leftarrow X + \lk{purple}{\purple{\mathrm{FFN}}\bp{\mathrm{LN}(X)}}
Each half only adds a change to every word's row, the word itself is still there: a
A block's output has exactly the same shape as its input, a row of d numbers per word
Transformers · The Transformer Block13
Layer Normalization
Before each half, every word's row is rescaled to mean 0 and standard deviation 1:
print(generate(W, "cheer cheer for old notre", 46))
Output
[cheer, cheer, for, old, notre, dame, wake, up, the, echoes, cheering, her, name, send, a, volley, cheer, on, high, shake, down, the, thunder, from, the, sky, what, though, the, odds, be, great, or, small, old, notre, dame, will, win, over, all, while, her, loyal, sons, and, daughters, march, on, to, victory]
Our GPT writes the fight song. It memorized 51 words (GPT-3 learned from about 300 billion tokens), but it's the same loop that writes every ChatGPT answer
Transformers · Writing the Fight Song21
Without the Mask
Same model, same training, no mask: the red part is gone
\red{\mathrm{Attention}}(Q, K, V) = \gold{\mathrm{softmax}}\Bp{\lk{purple}{\frac{Q K\T}{\sqrt{d_k}}} \lk{red}{+ M}} V