Self-Attention
Flash Survey Results
I thought "Good idea but do it better" was the best answer personally
Context
Often times when working with sequence data, we need to recall things that happened a long time ago.
Embeddings
Additionally, when we have tokens with multiple definitions like
This doesn't really make sense, when you see the word
Attention
In an ideal world, we could allow each token to decide how much attention they should pay to each token and which context from each token they should use.
We call this self-attention
The Self-Attention Equation
Q = X ωQ , the query: what this token is looking forK = X ωK , the key: what it offers to be matched againstV = X ωV , the value: what it passes along
Self-Attention is a way to update our embeddings using the context of the surrounding token's embeddings
Note:
An Analogy for Self-Attention
Another mental model for self-attention might be "using tinder", where everyone on the app is simultaneously looking for someone and being looked at.
- Q – Query: What each person is looking for in a person
- "I'm looking for a 10/10 terrachad ASU frat leader"
- K – Key: the first photo each person has on their profile
- "I'm a rocket"
- V – Value: The rest of the profile
softmax(Q KT / √dk )
Self-Attention in PyTorch
Our seven words with their real GloVe embeddings, 300 numbers each
torch.Size([7, 300]) torch.Size([7, 300]) tensor([0.0600, 0.1000, 0.1200, 0.0700, 0.0600, 0.1300, 0.4600])
How the Embeddings Change
Add
name and wake 0.09 -> 0.38 name and up 0.30 -> 0.54 name and the 0.45 -> 0.66 name and echoes 0.13 -> 0.38 name and cheering 0.02 -> 0.20 name and her 0.35 -> 0.54 name before and after: 0.98
Self-Attention (old exam question)
Values
All softmax(QK/d) does is compute how much attention each token should pay to the others.
To actually update our embeddings we then multiply these attention scores by the values matrix V.
We then add these back onto our original embeddings, updating each with the context provided by the surrounding tokens
Attention Heads
One set of
But a word relates to its sentence in many ways (
Multi-Head Attention
Several heads at once is called multi-head attention. Each head focuses on a different facet of the context, maybe one on the grammar and another on the meaning
Each head runs the attention equation on its own slice, then we concatenate their outputs and pass them through one linear layer,
Multi-Head Attention (note)
Done not with multiple weight matrices, but a logical partition of a single matrix
More heads don't add a single weight: Attention Is All You Need split 512 columns into 8 heads of 64, GPT-3 splits 12,288 into 96 heads of 128
Multi-Head Attention in PyTorch
torch.Size([1, 7, 4]) tensor([ 0.2600, -0.3000, 0.2300, 0.2900])
Both heads and
Visualization
Word Order
A GPT is mostly multi-head self-attention stacked many times (GPT-3: 96 layers), with no RNN anywhere
Shuffle our line:
Self-attention treats a sentence as a bag of words: wake up the echoes = the echoes wake up
Next lecture: positional encoding