image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ wake up the echoes cheering her name
Unit 03 · Chapter 5

Self-Attention

Self-Attention01

Flash Survey Results

I thought "Good idea but do it better" was the best answer personally

Self-Attention02

Context

Often times when working with sequence data, we need to recall things that happened a long time ago.

cheer cheer for old notre dame wake up the echoes cheering her name 6 words back

her10 is referring to notre3 dame4, 6 words back. We call this a long-term dependency. However due to the sequential nature of an RNN, notre3 dame4 can only effect her10 indirectly, through the tokens in between.

Self-Attention · Attention03

Embeddings

Additionally, when we have tokens with multiple definitions like mole41, they currently all share the same embedding. We call these static embeddings.

The mole dug a tunnel under the lawn the animal One mole of carbon weighs about 12 grams the unit The doctor checked the mole on her arm the spot on skin The mole passed secrets to the other side the spy

This doesn't really make sense, when you see the word mole41, you use context clues to determine which definition of mole41 the sentence is using.

Self-Attention · Attention04

Attention

In an ideal world, we could allow each token to decide how much attention they should pay to each token and which context from each token they should use.

each token on top pays some attention to every token below wake wake up up the the echoes echoes cheering cheering her her name name all in one step: red for the least attention, yellow for typical, green for the most, thicker for more

We call this self-attention

Self-Attention · Attention05

The Self-Attention Equation

\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\Bp{\frac{Q K\T}{\sqrt{d_k}}} V \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\Bp{\frac{Q K\T}{\sqrt{d_k}}} V

Self-Attention is a way to update our embeddings using the context of the surrounding token's embeddings

Note: X is our embeddings

Self-Attention · Attention06

An Analogy for Self-Attention

Another mental model for self-attention might be "using tinder", where everyone on the app is simultaneously looking for someone and being looked at.

Self-Attention · Attention07

softmax(Q KT / √dk)

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ wake up the echoes cheering her name key (attending to) wake up the echoes cheering her name query (attending from) 0.07 0.27 0.12 0.18 0.11 0.16 0.11 0.26 0.09 0.15 0.12 0.15 0.09 0.15 0.32 0.05 0.12 0.12 0.17 0.08 0.14 0.16 0.27 0.07 0.37 0.05 0.02 0.05 0.01 0.18 0.10 0.03 0.07 0.51 0.10 0.07 0.08 0.16 0.05 0.15 0.31 0.18 0.02 0.09 0.12 0.02 0.10 0.51 0.14
Self-Attention · Attention08

Self-Attention in PyTorch

Our seven words with their real GloVe embeddings, 300 numbers each

import torch
words = "wake up the echoes cheering her name".split()
glove = {}
for line in open("glove-victory-march.txt"):
w, *nums = line.split() # a word, 300 numbers
glove[w] = [float(n) for n in nums]
X = torch.tensor([glove[w] for w in words]) # 7 by 300
WQ = WK = WV = torch.eye(300) # untrained: q = k = v = x
def attention(Q, K, V):
S = Q @ K.T / K.shape[1] ** 0.5
A = torch.softmax(S, dim=1)
return A @ V, A
Z, A = attention(X @ WQ, X @ WK, X @ WV)
print(X.shape, Z.shape)
print(A[-1].round(decimals=2)) # name's weights
Output
torch.Size([7, 300]) torch.Size([7, 300])
tensor([0.0600, 0.1000, 0.1200, 0.0700, 0.0600, 0.1300, 0.4600])
X <7, 300> self-attention softmax(QKᵀ/√dₖ) V Z <7, 300> + X + Z <7, 300> residual new
Self-Attention · Attention09

How the Embeddings Change

Add Z back onto X and compare how similar name11 is to each word, before and after

new = X + Z # add the context back onto X
def cos(M): # cosine similarity of every pair
M = M / M.norm(dim=1, keepdim=True)
return M @ M.T
before, after = cos(X), cos(new)
i = words.index("name")
for j, w in enumerate(words[:-1]):
b, a = before[i, j], after[i, j]
print(f"'name' and {w!r:10} {b:.2f} -> {a:.2f}")
same = cos(torch.stack([X[i], new[i]]))[0, 1]
print(f"'name' before and after: {same:.2f}")
Output
name and wake     0.09 -> 0.38
name and up       0.30 -> 0.54
name and the      0.45 -> 0.66
name and echoes   0.13 -> 0.38
name and cheering 0.02 -> 0.20
name and her      0.35 -> 0.54
name before and after: 0.98

name11 moves toward its context: every similarity goes up (her10 0.35 → 0.54), yet it is still 0.98 like its old self

Self-Attention · Attention10

Self-Attention (old exam question)

Self-Attention · Attention11

Values

All softmax(QK/d) does is compute how much attention each token should pay to the others.

To actually update our embeddings we then multiply these attention scores by the values matrix V.

X <7, 4> self-attention softmax(QKᵀ/√dₖ) V Z <7, 4> + X + Z <7, 4> residual connection: X skips around and is added back embeddings with context

We then add these back onto our original embeddings, updating each with the context provided by the surrounding tokens

Self-Attention · Attention12

Attention Heads

One set of ωQ, ωK, ωV gives one grid of weights: one kind of relevance, context for a single "thing"

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ wake up the echoes cheering her name head 1 wake up the echoes cheering her name 0.02 0.40 0.10 0.10 0.06 0.24 0.08 0.28 0.07 0.13 0.13 0.16 0.09 0.14 0.56 0.02 0.08 0.09 0.13 0.03 0.10 0.00 0.73 0.04 0.10 0.02 0.09 0.02 0.01 0.18 0.11 0.04 0.09 0.45 0.12 0.32 0.03 0.13 0.07 0.19 0.10 0.17 0.11 0.05 0.15 0.04 0.19 0.25 0.21 wake up the echoes cheering her name head 2 0.29 0.09 0.10 0.20 0.14 0.06 0.11 0.17 0.14 0.16 0.12 0.13 0.12 0.15 0.08 0.17 0.14 0.12 0.14 0.22 0.14 0.82 0.01 0.02 0.09 0.04 0.00 0.02 0.03 0.20 0.14 0.07 0.10 0.34 0.13 0.01 0.20 0.13 0.05 0.08 0.42 0.11 0.01 0.20 0.11 0.03 0.07 0.49 0.10

But a word relates to its sentence in many ways (her10 owns name11, echoes8 get woken up6). So we use several, each an attention head

Self-Attention · Multi-Head Attention13

Multi-Head Attention

Several heads at once is called multi-head attention. Each head focuses on a different facet of the context, maybe one on the grammar and another on the meaning

\mathrm{MultiHead}(X) = \bb{\,\mathrm{head}_1 \;\; \mathrm{head}_2 \;\; \cdots \;\; \mathrm{head}_h\,} \Omega_O \mathrm{MultiHead}(X) = \bb{\,\mathrm{head}_1 \;\; \mathrm{head}_2 \;\; \cdots \;\; \mathrm{head}_h\,} \Omega_O

Each head runs the attention equation on its own slice, then we concatenate their outputs and pass them through one linear layer, ωO:

Self-Attention · Multi-Head Attention14

Multi-Head Attention (note)

Done not with multiple weight matrices, but a logical partition of a single matrix

0.9 1.0 0.8 0.9 −0.8 −0.3 −0.7 −0.1 0.9 0.1 0.9 0.3 −0.7 0.9 0.3 0.1 head 1 head 2 Ω Q = Q 1 = X Ω Q [:, 0:2] Q 2 = X Ω Q [:, 2:4]

More heads don't add a single weight: Attention Is All You Need split 512 columns into 8 heads of 64, GPT-3 splits 12,288 into 96 heads of 128

Self-Attention · Multi-Head Attention15

Multi-Head Attention in PyTorch

import numpy as np
import torch
from torch import nn
rng = np.random.default_rng(28)
X = rng.uniform(-1, 1, (7, 4)).round(1) + 0.0
Ws = rng.uniform(-1, 1, (4, 4, 4)).round(1)
X = torch.tensor(X, dtype=torch.float32)
WQ, WK, WV, WO = torch.tensor(Ws, dtype=torch.float32)
mha = nn.MultiheadAttention(embed_dim=4, num_heads=2,
batch_first=True)
W = torch.cat([WQ.T, WK.T, WV.T]) # 12 by 4
with torch.no_grad(): # our weights, no biases
mha.in_proj_weight.copy_(W)
mha.out_proj.weight.copy_(WO.T)
mha.in_proj_bias.zero_()
mha.out_proj.bias.zero_()
Z, A = mha(X[None], X[None], X[None]) # a batch of 1
print(Z.shape, Z[0, -1].detach().round(decimals=2))
Output
torch.Size([1, 7, 4]) tensor([ 0.2600, -0.3000,  0.2300,  0.2900])

Both heads and ωO in one call (nn.MultiheadAttention, our weights loaded)

Self-Attention · Multi-Head Attention16

Visualization

The attention visualizer on wake up the echoes cheering her name (synthetic data, 4-number embeddings, head 0 of 4): the tokens, their embeddings, E times W_q, W_k and W_v, a 7 by 7 grid of attention weights between 0.08 and 0.22, the 4 heads concatenated, times W_o, plus E, giving the output embedding
Self-Attention · Multi-Head Attention17

Word Order

A GPT is mostly multi-head self-attention stacked many times (GPT-3: 96 layers), with no RNN anywhere

Shuffle our line: the7 cheering9 echoes8 name11 her10 wake5 up6. Every token gets the same output, just in a different row

Self-attention treats a sentence as a bag of words: wake up the echoes = the echoes wake up

Next lecture: positional encoding

Self-Attention · Multi-Head Attention18