CSE 30124 · Introduction to Artificial Intelligence William Theisen · University of Notre Dame
Recurrent Neural Networks01
So Far
So far we've seen networks that take in single samples that lack context
When our CNN labels a photo, it doesn't matter what photo came before it or comes after. This is not the case for sequence data like language
Recurrent Neural Networks · Sequence Data02
Sequence Data
Examples of sequence data:
Text (sentences, documents)
Speech and audio
Time series (stock prices, weather)
Video (a sequence of images)
How do we handle data where the order of what we're seeing matters? How do we maintain context over time?
Recurrent Neural Networks · Sequence Data03
Language as Input
Computers understand only numbers so we need to somehow convert text into numbers in such a way as to enable computer understanding. We call this subfield of AI: Natural Language Processing(NLP)
NLP starts with two preprocessing steps: tokenization and embedding
Recurrent Neural Networks · Tokenization04
Tokenization
Tokenization is the process of splitting a piece of text into tokens.
1
2
3
4
5
6
7
8
9
10
11
12
import re
text = """Cheer, cheer for old Notre Dame,
Wake up the echoes cheering her name,
Send a volley cheer on high,
Shake down the thunder from the sky."""
clean = re.sub(r"[^a-z ]", " ", text.lower())
tokens = clean.split()
vocab = list(dict.fromkeys(tokens)) # first appearance
GPT-4o has about 200,000tokens, so almost every word is a token of its own, with its space. "Cheer" at the start of a line is 2 pieces, Che21294er259, but cheer38313 after a space is one.
This implies that cheer0 at the start of a sentence is pretty rare, but it appears enough within sentences to deserve its own token.
The numbers say nothing about meaning of the words
thunder19 isn't more like sky21 than cheer0
Recurrent Neural Networks · Word Embeddings12
Word Embeddings
After tokenization, our next step is to "embed" each token. The embedding table has one row per token ID: for our 22 words, GloVe's is <22, 300>. Embedding a token is simply looking up its row.
token
ID
0
1
2
3
…
299
cheer0
0
0.41
−0.15
−0.04
−0.37
…
0.19
for1
1
−0.24
0.12
0.19
−0.27
…
0.16
old2
2
−0.38
−0.07
−0.37
−0.34
…
1.12
notre3
3
0.01
0.34
0.02
0.32
…
0.42
dame4
4
−0.09
0.34
0.11
0.26
…
0.48
⋮
⋮
⋮
⋮
⋮
⋮
⋮
⋮
sky21
21
−0.03
−0.22
0.72
0.20
…
−0.08
This gives us a feature vector for each token, similar to the 4,096 features our CNN made for each photo. The embedding is like the "definition" of a token
Recurrent Neural Networks · Word Embeddings13
GloVe Embeddings
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import torch
vocab, rows = [], []
for line inopen("glove-victory-march.txt"):
word, *nums = line.split() # a word, 300 numbers
vocab.append(word)
rows.append([float(n) for n in nums])
E = torch.tensor(rows) # our 22 words, in song order
Converting tokens to embeddings allows us to interpret them as points in high dimensional space, which allows us to compare them against each other!
NOTE: In this plot we use PCA to view our embeddings in 2 dimensions but GloVe embeddings are actually <1, 300> for each token
Recurrent Neural Networks · Word Embeddings15
Embeddings
GloVeembeddings are based on co-occurrences, even though up6 and down18 mean opposite things, they appear next to the same words, so they sit close together (0.75).
Embeddings learn whatever patterns the text has, biases included.
Recurrent Neural Networks · Word Embeddings16
Exploration of Biases in Embeddings
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import numpy as np
import torch
import matplotlib.pyplot as plt
# type any words here, then Run,
# NOTE: it will only work if GloVe knows the word
words = ["nurse", "engineer", "teacher", "football"]
E = torch.tensor(np.load("glove-20k.npy")).float()
defvec(w):
returnE[vocab.index(w)]
axis = vec("woman") - vec("man")
axis = axis / axis.norm() # the man-to-woman direction
words = [w for w in words if w in vocab] # 20,002 words
pos = [float(vec(w) @ axis) for w in words]
for w, p insorted(zip(words, pos), key=lambda t: t[1]):
print(f"{w:12} {p:6.2f}")
cols = ["#002B5B"if p < 0else"#DCB439"for p in pos]
plt.figure(figsize=(7, 0.4 * len(words) + 1))
plt.barh(words, pos, color=cols)
plt.axvline(0, color="gray")
plt.xlabel('man woman')
plt.show()
Output
football-0.61engineer-0.56teacher1.20nurse1.86
Recurrent Neural Networks · Word Embeddings17
Python RNN by Hand - Embedding Layer
An embedding layer is just a table lookup
1
2
3
4
5
6
7
8
9
import torch
classEmbedding:
def__init__(self, n_tokens, d):
self.E = torch.randn(n_tokens, d) * 0.1
defforward(self, ids): # (B, T) token IDs
self.ids = ids # keep for backward
return self.E[ids] # (B, T, d): one row per ID
Basically all NLP models, including LLMs like Claude and ChatGPT share the same two preprocessing steps: tokenization and embedding
Recurrent Neural Networks · RNN from Scratch18
NLP Tasks
Tokenization and embedding are only the preprocessing steps we need before we use an NLP model, they only give us the input for our model.
Some tasks we've seen so far are regression and classification, but NLP has some new tasks of its own.
Recurrent Neural Networks · RNN from Scratch19
NLP Task: Generation
All four of the NLP tasks require context. If we are predicting the next token after the current one, after notre3 always comes dame4, but picking the next token after the7 is harder
We need a model that reads one token at a time but also maintains the context of what we've seen so far over time
Recurrent Neural Networks · RNNs20
Sequence Networks - RNNs (1986)
We can begin to solve the problem of context by using a new model, called a Recurrent Neural Network(RNN)
Recurrent Neural Networks · RNNs21
Exploring the RNN
Recurrent Neural Networks · RNNs22
Fully Connected Layers vs. RNNs (Old Exam Question)
Recurrent Neural Networks · RNNs23
Activation Functions - tanh
While we've seen the sigmoid and RELU, in RNNs the tanh function is used.
Why we use the tanh function will become clear when we starting training RNNs next lecture
Recurrent Neural Networks · RNNs24
Time After Time
Most RNN diagrams depict them in their "unrolled" state, which can be confusing
Unlike an FFN, this isn't four different layers: every RNN cell and output layer sharesthe same weights.
Recurrent Neural Networks · RNNs25
RNNs (Old Exam Questions)
Recurrent Neural Networks · RNNs26
Python RNN by Hand - Recurrent Layer
One step is the RNN equation: ct = tanh(xtωxT + ct−1ωcT + bc)
For our task of predicting the next token in a sequence, the output from our RNNsoutput layer is a probability of every token in our vocabulary being the next token