CSE 30124 · Introduction to Artificial Intelligence William Theisen · University of Notre Dame
Backpropagation Through Time01
Training an RNN
Last chapter's recurrent neural network read the Victory March one word at a time, mixing each word with a hidden state carried over from the word before:
Unfortunately its weights were made up, so it forgot how the song started within a few words. To learn real ones, this chapter gives the network more room: the whole chorus, 51 words and 41 different ones, tokenized the same way, and a hidden state of 16 numbers. Each word comes in one-hot, so xtωxT picks out the word's column of ωx, which makes ωx the embedding table, learned with everything else. At every word the job is to predict the next one:
Backpropagation Through Time · Backpropagation Through Time02
Loss Over a Sequence
Pictured on the last page is the network unrolled over the chorus, one copy of the cell for each of the 50 predictions. They aren't separate networks, every copy uses the same three weight matrices. Drawn this way an RNN is just a very deep feed-forward network, 50 layers deep, that happens to reuse its weights in every layer. So we should be able to train it the way we train any other network, with a loss and backpropagation.
Each copy makes a prediction, so each copy gets a loss. Copy t gives a probability to each of the 41 words, and as with logistic regression we charge it the negative log of the probability it gave the word that actually came next, yt. The loss for the whole chorus is the average over all 50:
A model that knows nothing at all gives every word the same chance, 1 in 41, and pays −log(1/41) = log 41 = 3.714 on every word. That makes a handy yardstick for a model that hasn't learned anything yet, such as ours.
Backpropagation Through Time · Backpropagation Through Time03
Computing the Loss
Here's the loss in PyTorch, for our RNN with random starting weights (numpy draws them, so they come out the same everywhere):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import numpy as np
import torch
song = """cheer cheer for old notre dame wake up the
echoes cheering her name send a volley cheer on
high shake down the thunder from the sky what
though the odds be great or small old notre dame
will win over all while her loyal sons and
daughters march on to victory"""
words = song.split()
vocab = list(dict.fromkeys(words)) # first appearance
seq = [vocab.index(w) for w in words]
x, y = seq[:-1], seq[1:] # each word, the next
V, H, T = len(vocab), 16, len(x)
rng = np.random.default_rng(0) # same start as before
f = lambda d, n: torch.tensor(rng.normal(0, d, n)).float()
Wx = f(V ** -0.5, (H, V))
Wc = f(H ** -0.5, (H, H))
Wy = f(H ** -0.5, (V, H))
bc, by = torch.zeros(H), torch.zeros(V)
c, loss = torch.zeros(H), 0
for t inrange(T):
c = torch.tanh(Wx[:, x[t]] + c @ Wc.T + bc)
o = c @ Wy.T + by
p = torch.softmax(o, dim=0)
loss -= torch.log(p[y[t]])
print(V, T, round(loss.item() / T, 3),
round(np.log(V), 3))
Output
41503.7193.714
The untrained RNN scores 3.719, a hair worse than knowing nothing, which is about what random weights should do.
Backpropagation Through Time · Backpropagation Through Time04
Backpropagation Through Time
To train it we need the gradient of ℒ for ωx, ωc and ωy. We get it by running the chain rule backward through the unrolled network, from the last word to the first, which is called backpropagation through time (BPTT). Really it's just backpropagation on a very long network.
The output end is familiar. Through a softmax, the cross-entropy's derivative is ŷt − yt (with yt one-hot), the same "prediction minus truth" as logistic regression. What's new is that each ct (pictured below) feeds its own prediction and the next copy, so its gradient comes from both:
Backpropagation Through Time · Backpropagation Through Time05
Gradients of Shared Weights
In those two equations, zt is what goes into the tanh, 1 − (ct)2 is the tanh's derivative, and we've left out the 1/50 to keep things readable. The last copy has no next copy, so the gradient starts there with just its own loss and gets handed back one word at a time, the same way the forward pass handed the hidden state forward.
Once we have ∂ℒ/∂zt at every word, the weights are nearly done, with one twist. ωc is used by all 50 copies, so it gets a gradient from all 50, and the chain rule says to add them up. The same goes for ωx and ωy:
Each term is a one-layer gradient of the kind we've seen since logistic regression, an error times an input, and there's one per copy. Adding them is what makes it one network rather than 50. ωc moves in whichever direction helps all the words, on average.
Over the page is all of it in PyTorch, by hand, repeated a thousand times with α = 0.5:
Backpropagation Through Time · Backpropagation Through Time06
Training on the Victory March
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import numpy as np
import torch
song = """cheer cheer for old notre dame wake up the
echoes cheering her name send a volley cheer on high
shake down the thunder from the sky what though the
odds be great or small old notre dame will win over all
while her loyal sons and daughters march on to victory"""
vocab = list(dict.fromkeys(song.split()))
s = [vocab.index(w) for w in song.split()]
V, H, T = len(vocab), 16, len(s) - 1
X, Y = torch.eye(V)[s[:-1]], torch.eye(V)[s[1:]]
rng = np.random.default_rng(0)
f = lambda d, n: torch.tensor(rng.normal(0, d, n)).float()
Backpropagation Through Time · Backpropagation Through Time07
Gradient Checking
After 1,000 steps the loss is down from 3.719 to 0.015 and the RNN gets every one of the 50 next words right. That includes both "dame"s, "wake" after the first and "will" after the second, which it can only do by remembering what came before. (Memorizing one chorus with 1,625 weights isn't much of an achievement, but it's a good sign the gradients are right.)
Only a sign, though. A network can learn a fair amount with slightly wrong gradients, and with this many indices it's easy to get one wrong. The way to be sure is to compare against brute force, called gradient checking. Nudging a single entry of ωc up and down by 0.000001 in the untrained RNN and measuring how much the loss changes gives a slope of −0.01508333. BPTT gives −0.01508333 too.
Writing the backward loop by hand is a good way to see what BPTT does and a bad way to get it right. As we saw in Backpropagation, PyTorch records every operation of the forward pass and can run the chain rule back through them for us, and an unrolled RNN is just a lot of operations. Over the page is the same loss with ωc marked as needing a gradient, and the same nudge as a check:
Backpropagation Through Time · Backpropagation Through Time08
BPTT in PyTorch
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import numpy as np
import torch
song = """cheer cheer for old notre dame wake up the
echoes cheering her name send a volley cheer on high
shake down the thunder from the sky what though the
odds be great or small old notre dame will win over all
while her loyal sons and daughters march on to victory"""
vocab = list(dict.fromkeys(song.split()))
s = [vocab.index(w) for w in song.split()]
V, H, T = len(vocab), 16, len(s) - 1
rng = np.random.default_rng(0) # the same start
f = lambda d, n: torch.tensor(rng.normal(0, d, n))
Both agree with our hand-written BPTT to eight decimal places, so from here on loss.backward() does BPTT for us.
Backpropagation Through Time · Backpropagation Through Time09
Truncated BPTT
Keeping every step of the forward pass around for the backward pass takes memory, and for a long text (a book, say) far too much. So in practice the text is cut into chunks of a few dozen words, the hidden state is carried from chunk to chunk as usual, and the gradient is stopped at the start of each chunk with c.detach(). This is called truncated BPTT.
We tried it on the chorus, stopping the gradient every k words and otherwise training exactly as before:
k
1
2
3
5
10
50 (all)
loss
0.018
0.022
0.016
0.018
0.015
0.015
right
50
50
50
50
50
50
Even k = 1, where the gradient never crosses from one word to the one before, gets all 50. The hidden state still carries the song forward (even random weights give the two "dame"s different hidden states), and our chorus never needs the network to learn to hold on to something far back, so the gradient never has to go far.
Unfortunately, most text does. Last chapter our made-up RNN had forgotten how the song started by the ninth word, and teaching a network to hold on longer means sending the gradient back at least that far.
Backpropagation Through Time · Backpropagation Through Time10
Vanishing Gradients
Each word we walk back multiplies the gradient by the tanh's derivative, never more than 1, and by ωc. Following the last word's loss back to word k repeats that pair of factors (Dt is a diagonal matrix of the 1 − (ct)2):
With the usual starting weights (navy) the gradient is 1.00 at the last word. Ten words back it's 0.17, twenty words back 0.023, and at the first "cheer" it's 0.000092, about 11,000 times smaller. This is called the vanishing gradient. Any deep network can have it, but an RNN is as deep as its sequence is long, so an RNN struggles to learn anything that depends on words far back.
Backpropagation Through Time · Vanishing and Exploding Gradients11
Exploding Gradients
The red curve on the last page is the same network with ωc three times bigger. Now the factors tend to grow the gradient rather than shrink it: about 10 at ten words back, 134 at twenty, and about 48,000 at the first "cheer". That's an exploding gradient, and it's no better. Multiply 48,000 by any sensible learning rate and the weights land somewhere wild.
Which one we get depends on whether each word's factor tends to be below or above 1, and either way it gets raised to a big power. A factor of 0.8 per word leaves 0.00002 after 49 words, and 1.2 per word gives 7,584. Unfortunately, no setting is safe for long sequences, since anything a little away from exactly 1 ends up enormous or negligible. (This is extremely approximate, since the factors are matrices that change from word to word, but it's the right idea.)
The two problems also have the same cause. The gradient has to pass through ωc and a tanh at every single word, with no way around them.
Backpropagation Through Time · Vanishing and Exploding Gradients12
Gradients Through a Gate
Last chapter's fix for forgetting was a gate, a vector zt of numbers between 0 and 1 deciding how much of the old hidden state to keep. It turns out to be a way around for the gradient too:
Walking back one word, ct−1 reaches ct two ways, through the candidate c̃t (with its ωc and tanh, as before) and straight across, multiplied only by zt. The straight path has no weight matrix and no tanh on it. A gate of 0.9 at every word leaves 0.949 = 0.0057 of the gradient after 49 words, about 60 times what reached the first "cheer" in our RNN, and a gate of 0.99 leaves 0.61.
The catch, as we saw, is that a gate stuck near 1 remembers everything just as hard. So the gate has to be learned as well, from the word and the hidden state, and that's what LSTMs and GRUs do.
Backpropagation Through Time · LSTMs and GRUs13
Long Short-Term Memory
The Long Short-Term Memory unit (LSTM, Hochreiter and Schmidhuber, 1997) was built around that straight path. Alongside the hidden state it keeps a second vector, the cell statemt, which is only ever changed by a multiplication and an addition, along the top of the picture:
ft is how much of the old cell state to keep, and it ⊙ m̃t is the new information being written in. Following the gradient back along that top line, ∂mt/∂mt−1 is just ft. There's no ωc and no tanh in the way, so when ft is close to 1, the gradient passes back through the cell state almost untouched.
Backpropagation Through Time · LSTMs and GRUs14
The LSTM's Gates
The forget gate is one of three. Each gate is a sigmoid layer looking at the current word and the previous hidden state, so each is a vector of numbers between 0 and 1 that multiplies something elementwise:
The hidden state is now a filtered view of the cell state, squashed by a tanh and scaled by the output gate. Predictions only ever see ct, so the cell state is free to hold on to things the current prediction doesn't need.
That's four layers where the RNN had one, so four times the recurrent weights, 3,712 for our 16 units against the RNN's 928.
Backpropagation Through Time · LSTMs and GRUs15
An LSTM on the Victory March
We trained an LSTM with 16 units on the chorus the same way as the RNN (1,000 steps, α = 0.5), and it gets all 50 next words right as well. Here's one of its units the two times the chorus sings "old notre dame":
word
f
i
m̃
o
m
c
4
old
0.91
0.94
−0.97
1.00
−2.25
−0.97
5
notre
0.93
0.94
0.53
1.00
−1.59
−0.92
6
dame
0.79
0.95
0.92
1.00
−0.38
−0.36
35
old
0.97
0.89
−0.96
1.00
−8.18
−1.00
36
notre
0.97
0.89
0.31
1.00
−7.66
−1.00
37
dame
0.91
0.88
0.77
1.00
−6.33
−1.00
The two "dame"s get exactly the same input, so the only way to tell them apart is what the cell remembers. In this unit the cell state is −0.38 at the first "dame" and −6.33 at the second, and the hidden state passes that on as −0.36 and −1.00. Notice the forget gate, 0.97 on the second "old" and "notre", keeping almost all of the old cell state each word.
Backpropagation Through Time · LSTMs and GRUs16
Memory in the Cell State
Pictured below is the same unit across the whole chorus. The cell state wanders well outside −1 to 1, while the hidden state, squashed by its tanh, never does:
From the third "cheer" (word 17) to the second "old" (word 35) the cell state sinks by a little under half a step every word, so this unit seems to be keeping track of how far into the song we are. Its forget gates from word 27 to word 36 multiply to 0.66, so a gradient sent back along its cell state over those 10 words keeps two thirds of its size. The untrained RNN's shrank by a factor of about 6 over the same distance.
(On a chorus this short the plain RNN gets every word right too, so this shows how the LSTM remembers, not that it needs to.)
Backpropagation Through Time · LSTMs and GRUs17
Gated Recurrent Units
All those weights make the LSTM slow. The Gated Recurrent Unit (GRU, Cho and others, 2014) is last chapter's gate made learnable, with no separate cell state. An update gate zt does the job of the forget and input gates, and a reset gate rt decides how much of the old state the candidate gets to see:
When zt is near 1, ct is just ct−1 and the gradient passes straight across. That's 2,784 recurrent weights for our 16 units, three times the RNN's, and on many tasks GRUs do about as well as LSTMs.
Backpropagation Through Time · LSTMs and GRUs18
Gradient Clipping
Unfortunately, gates only fix half the problem. The cell state's path keeps the gradient from vanishing, but every other path still goes through weight matrices, and LSTMs are still prone to exploding gradients. While ours trained, the gradient's size was usually around 0.08, and at step 199 it jumped to 3.09, nearly forty times that.
The usual fix is blunt. If the gradient's size (its norm, over all the weights at once) is bigger than some ceiling τ, we scale it down to τ before taking the step:
g \leftarrow \tau\,\frac{g}{\norm{g}} \qquad \text{if } \norm{g} > \tau
This is called gradient clipping. It keeps the gradient's direction and only limits how far one step can go. Our plain RNN shows the difference best. With α = 1, twice the learning rate that worked, its loss is down to 0.26 by step 278, then lurches back up to 6. It recovers, and around step 507 the loss shoots from 0.3 to over 4, with gradients of 17, 45, 81 and 194 in the next 40 steps. After 1,000 steps it gets 6 of the 50 words right.
Backpropagation Through Time · Gradient Clipping19
Gradient Clipping in Practice
Clipped at 1, the same run gets all 50, and only 7 of its 1,000 steps ever needed clipping. Clipping does nothing for vanishing gradients though, since a tiny gradient is already under the ceiling.
Backpropagation Through Time · Gradient Clipping20
Gradient Clipping in PyTorch
In practice clipping is a single line, between computing the gradients and taking the step. Here is a PyTorch RNN learning to add up 50 numbers, clipped at 1:
clip_grad_norm_ returns the norm before clipping, about 8 to 10 here (desktop PyTorch draws different random numbers, so its numbers differ), so every step is cut to a length of 1. loss.backward() runs BPTT for us, which after the last few pages is probably for the best.
Backpropagation Through Time · Gradient Clipping21
Static Embeddings
Whatever the cell, the input for a word is the same. In our RNN it's a column of ωx, the word's embedding. "cheer" appears three times in the chorus, and each time it gets exactly the same 16 numbers, whether it's the first word of the song or the one in "send a volley cheer on high". An embedding that doesn't depend on the words around it is called a static embedding.
The only context the RNN has is its hidden state. After each of the three "cheer"s the trained RNN's hidden states are quite different from each other (cosine similarities of only 0.34 to 0.40), and they need to be, since the next words are "cheer", "for" and "on". So all the context lives in ct, 16 numbers that have to carry everything the network knows about the song so far.
Backpropagation Through Time · Toward Attention22
Attention
That's a lot to ask of 16 numbers. To predict "victory", anything the RNN needs from "cheer, cheer for old notre dame" has to survive 49 updates of the hidden state, and the gradient that would teach it to keep it has to make the same 49 steps back.
Last chapter's attention was a way around that. Instead of trusting one hidden state to have remembered everything, each prediction looks back at every earlier hidden state directly and decides how much each one matters. Pictured below is what that does to the distance between words:
With attention every word is a single step from every other, for the forward pass and for the gradient. The price is that every word gets compared with every word before it, 1,275 comparisons for our 50 words where the RNN takes 50 steps, and about 5 billion for a 100,000-word book. The work grows with the square of the length, an RNN's only with the length.
Backpropagation Through Time · Toward Attention23
Memory Caching
That price is why RNNs haven't quite gone away. In February 2026 a group at Google Research (Behrouz and others) proposed Memory Caching, an RNN that reads its text in segments and saves a copy of its hidden state at the end of each one, so later words can look back at the saved copies as well as their own hidden state:
Its memory grows by one hidden state per segment rather than one per word, a middle ground between an RNN's fixed memory and attention's growing one. It generally improved the recurrent models they added it to, and closed much of the gap to attention-based models on recall tasks.
The models behind today's chatbots drop the recurrence altogether, though, and build the whole network out of attention, with no hidden state carried from word to word at all.