Converting tokens to embeddings allows us to interpret them as points in high dimensional space, which allows us to compare them against each other!
NOTE: In this plot we use PCA to view our embeddings in 2 dimensions but GloVe embeddings are actually <1, 300> for each token
Backpropagation Through Time · Sequence Data06
Recurrent Neural Networks
Backpropagation Through Time · Sequence Data07
Output from an RNN
For our task of predicting the next token in a sequence, the output from our RNNsoutput layer is a probability of every token in our vocabulary being the next token
This is identical to how an LLM like ChatGPT or Claude creates output too!
Backpropagation Through Time · Sequence Data08
Output from an RNN
To train a text generation model, we can re-use cross entropy loss and think of text generation almost as a classification task.
Backpropagation Through Time · Sequence Data09
RNNs (Old Exam Questions)
Backpropagation Through Time · Sequence Data10
Backpropagation Through Time
Backpropagation through time (BPTT) is just backpropagation on the unrolled network, from the last word back to the first
Backpropagation Through Time · Sequence Data11
Chain Rule - Multiplication and Addition
Along one path the chain rule multiplies the derivatives. When something is used more than once, like ωc at both steps of "Good Luck", we add its paths:
Backpropagation Through Time · Sequence Data12
ℒ =
\frac{\partial ℒ}{\partial \Omega_x} =
\frac{\partial ℒ}{\partial \Omega_x} =
Backpropagation Through Time · Sequence Data13
BPTT by Hand - Embedding Layer
Last lecture's classes only went forward. Now each layer gets a backward that takes the gradient of its output and returns the gradient of its input.
Backpropagation Through Time · BPTT from Scratch22
Vanishing/Exploding Gradients
RNNs are particularly vulnerable to the vanishing gradient problem because we have to compute the gradients of the lossback through every time step
This is why RNNs use tanh, not the sigmoid. The sigmoid's slope is at most 0.25, so 50 steps back shrink the gradient at least 0.2550 ≈ 8 × 10⁻³¹ times. Tanh's slope reaches 1, so it only shrinks the gradient where the cell is saturated.
Backpropagation Through Time · BPTT from Scratch23
Vanishing Gradient
Since you are multiplying increasingly small values with each other during backprop, the gradient eventually trends to 0, meaning updates will stop happening.
Backpropagation Through Time · BPTT from Scratch24
Exploding Gradient
Almost the opposite of the vanishing gradient, the exploding gradient is when the gradients become too large, so the weights take huge steps
Anexploding gradient has a simple fix: clip, or put a ceiling on, the size of the gradient we'll accept.
Vanishing gradients are harder: they need a new kind of cell.
Backpropagation Through Time · BPTT from Scratch25
Another Issue with RNNs: Long-Term Dependencies
In last chapter's RNN, change the first word from cheer0 to sky21 and measure how far apart the hidden states are after each word. A word that depends on one far back is a long-term dependency.
By the ninth word it's 0.001: the RNNforgets how the song started. One fix: a gate that lets each step choose what to keep
Backpropagation Through Time · Vanishing and Exploding Gradients26
Gates and Computation Graphs
Add a new component called a gate that allows each step choose how much of the context (hidden state) to keep
Backpropagation Through Time · Vanishing and Exploding Gradients27
Effect of Gates
Backpropagation Through Time · LSTMs and GRUs28
Long Short-Term Memory units - LSTMs (1997)
LSTMs introduce the idea of gates to help solve the vanishing gradient problem.
Backpropagation Through Time · LSTMs and GRUs29
LSTMs and GRUs
A real gate is its own sigmoid layer, so the network learns when to hold on and when to let go
LSTM (1997): three gates and a separate cell state
GRU (2014): an update gate and a reset gate
1
2
3
4
from torch import nn
for cell in [nn.RNN(2, 2), nn.GRU(2, 2), nn.LSTM(2, 2)]:
GRUs are a simpler version of LSTMs that simplify the gate structure
Backpropagation Through Time · LSTMs and GRUs32
Static Embeddings
So far, the inputs to our model are what are called static embeddings, embeddings that are the same embedding for a token no matter its context
Backpropagation Through Time · Toward Attention33
Attention
Bahdanau, Cho and Bengio (2014): before each token it writes, the decoder looks back at every encoder hidden state
Every word written gets its own context vector
Backpropagation Through Time · Toward Attention34
Attention Weights
Last chapter's network at the last the7, looking back at the 24 words before it:
Most weight on up6 and the other the7s: their hidden states have the biggest dot product with this one. Next chapter: attention with learned weights
Backpropagation Through Time · Toward Attention35
Attention
Some parts of our context are more important than others! Attention is a mechanism through which we can focus on the most relevant parts of the context