Unit 03 · Chapter 2

Convolutional Neural Networks

Convolutional Neural Networks01

Dataset - MNIST

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ N O T R E D A M E 7 2 a b d ? ? ? ? ? ? ? ? ? ? ? ? police archive (EMNIST): labelled, 28 × 28 Professor Dingler's note: no labels
Convolutional Neural Networks · Digits02

Reviewing Neural Networks

An MNIST 5 drawn as a grid of pixel values
digit0123456789
ŷ0.110.140.130.210.100.090.080.060.050.03
y0.000.000.000.000.001.000.000.000.000.00
Convolutional Neural Networks · Digits03

Why do we care about CNNs?

We get 99% on MNIST without using anything fancy, why do we care?

Training accuracy over five epochs on MNIST: the CNN goes from about 94.7% to 99.4%, the feed-forward network from about 93.2% to 98.8%
Convolutional Neural Networks · Digits04

CIFAR-10

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ airplane automobile bird cat deer dog frog horse ship truck

CIFAR-10 is 60,000 tiny color photos, 32 × 32 pixels, of 10 kinds of thing. We train on 50,000 and keep 10,000 aside as unseen photos. A lot harder than handwritten digits.

Convolutional Neural Networks · Photos05

A Feed-Forward Network on Photos

To hand a photo to a feed-forward network we flatten it: 32 × 32 pixels with 3 colors each is 3,072 inputs.

import torch
from torch import nn
ffn = nn.Sequential(
nn.Flatten(), # 3 x 32 x 32 -> 3,072
nn.Linear(3072, 512), nn.ReLU(),
nn.Linear(512, 256), nn.ReLU(),
nn.Linear(256, 10),
)
x = torch.zeros(5, 3, 32, 32) # 5 blank photos
print(ffn(x).shape)
print(sum(p.numel() for p in ffn.parameters()))
Output
torch.Size([5, 10])
1707274
Convolutional Neural Networks · Photos06

CNN vs. FFN on CIFAR-10

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 1 2 3 4 5 6 7 8 9 10 epoch 0 10 20 30 40 50 60 70 80 unseen photos right (%) 71.1% 49.5% guessing: 10% CNN feed-forward network

Same photos, same training: the FFN gets 49.5% of unseen photos right, a CNN gets 71.1%.

Convolutional Neural Networks · Photos07

Shuffling the Pixels

Shuffle every photo's pixels into the same random order (nothing is lost, it's just rearranged) and train both networks again:

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ a photo its pixels shuffled
unseen photos rightfeed-forwardCNN
photos as they are49.5%71.1%
pixels shuffled48.8%50.4%

The FFN doesn't use the layout. The CNN's whole advantage is making use of the overall structure of the image

Convolutional Neural Networks · Photos08

Filters

A photo is 3,072 numbers, too many to work by hand, so we'll build our first one on an 8 × 8 handwritten zero (pixels from 0 to 16). We want a neuron that looks for specific structures in an image

0 0 5 13 9 1 0 0 0 0 13 5 0 0 3 15 8 0 0 4 12 8 0 0 5 8 0 0 9 8 0 0 4 11 0 1 12 7 0 0 2 14 5 10 12 0 0 0 0 6 13 10 0 0 0 15 10 15 2 0 11 0 0 8 1 1 1 0 0 0 -1 -1 -1 32 15 10 15 2 0 11 0 0 8 × = patch filter output

Lay a 3 × 3 grid of weights over a 3 × 3 patch, multiply each pixel by the weight on top of it, and add up the nine products: 15 + 10 + 15 − 8 = 32.

A small matrix of weights encoding some pattern we're looking for in an image is called a filter or a kernel.

Convolutional Neural Networks · Convolution09

A Horizontal Edge Filter

1s along the top, 0s through the middle, −1s along the bottom: it adds up the top row and takes away the bottom row.

-1 -1 -1 0 0 0 1 1 1 image kernel output
Convolutional Neural Networks · Convolution10

Convolution

Slide the filter across the image one pixel at a time, doing the same multiply and add at every stop. A 3 × 3 filter fits in 6 places across an 8 pixel row:

import torch
from sklearn.datasets import load_digits
zero = torch.tensor(load_digits().images[0],
dtype=torch.float32)
K = torch.tensor([[ 1., 1., 1.],
[ 0., 0., 0.],
[-1., -1., -1.]])
out = torch.zeros(6, 6)
for i in range(6):
for j in range(6):
patch = zero[i:i+3, j:j+3]
out[i, j] = (patch * K).sum()
print(out.long())
Output
tensor([[-13,  -2,  10,  10,  -9, -18],
        [ -3,  12,  26,  32,  14,   4],
        [  5,   7,   9,   4,   2,   2],
        [  1,   1,   0,  -5,  -4,  -3],
        [ -3,  -8, -21, -18,  -5,   5],
        [  9,  -4, -17, -10,  10,  19]])
Convolutional Neural Networks · Convolution11

Cross-Correlation

CNNs are named after convolution, which combines two functions to create a third by flipping the kernel before sliding it:

(f * g)(x, y) = \sum_{i}\sum_{j} f(i, j)\, g(x - i,\, y - j) (f * g)(x, y) = \sum_{i}\sum_{j} f(i, j)\, g(x - i,\, y - j)

Without the flip it's called cross-correlation, and that's what we just did:

S(i, j) = (I \star K)(i, j) = \sum_{m}\sum_{n} I(i + m,\, j + n)\, K(m, n) S(i, j) = (I \star K)(i, j) = \sum_{m}\sum_{n} I(i + m,\, j + n)\, K(m, n)

Nearly every library computes cross-correlation and calls it convolution. The kernel is learned anyway, so a network that needed it flipped would learn it flipped.

Convolutional Neural Networks · Convolution12

Feature Maps

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ the zero (8 × 8) -13 -2 10 10 -9 -18 -3 12 26 32 14 4 5 7 9 4 2 2 1 1 0 -5 -4 -3 -3 -8 -21 -18 -5 5 9 -4 -17 -10 10 19 horizontal-edge filter -33 -27 14 3 6 27 -40 -10 30 -17 -11 34 -35 10 35 -26 -24 28 -31 13 30 -29 -22 29 -33 6 22 -28 -4 33 -31 -12 10 -6 14 24 vertical-edge filter

The grid a filter makes is a feature map, one number per position saying how strongly that patch matches the filter.

Turned on its side, the filter picks out the zero's sides instead (−40 and 35). Different filters find different patterns, so a layer applies several filters and keeps every map.

Convolutional Neural Networks · Convolution13

Edges in a Photo

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ the photo (32 × 32) horizontal-edge filter vertical-edge filter

The same two filters on a CIFAR photo (turned grey by averaging its three colors): the horizontal-edge filter traces the top and bottom of the plane, the vertical-edge filter finds the building's columns.

A real CNN's first filters are 3 × 3 × 3, so they can look for colors, not just brightness too.

Convolutional Neural Networks · Convolution14

Output Size

4 8 12 13 14 15 16 0 -1 0 -1 5 -1 0 -1 0 5 12 15 16 11 1 2 3 5 6 7 9 10 11 ★ + = input kernel bias output

A k × k filter on an n × n image fits in n − k + 1 places across, so a 4 × 4 image gives a 2 × 2 output.

Each output also gets a bias, the same job it did in our neurons: 5·6 − 2 − 5 − 7 − 10 + 5 = 11.

Convolutional Neural Networks · Convolution15

Padding

Shrinking is a nuisance: two 3 × 3 layers take our digit down to 4 × 4, and border pixels get looked at less.

Padding is a ring of zeros around the image. One ring makes our zero 10 × 10, the filter fits in 8 places, and the map is the same size as the image.

-1 -1 -1 0 0 0 1 1 1 image kernel output
Convolutional Neural Networks · Convolution16

Stride

stride 2 input (7 × 7) output (3 × 3)

The stride is how many pixels the filter moves each step. Stride 2 over a 7 × 7 input makes 3 stops across, so the output is 3 × 3.

A cheap way to shrink a map, at the price of skipping positions where a pattern might have matched.

Convolutional Neural Networks · Convolution17

Output Size, in General

An n × n input, a k × k filter, padding p and stride s give an output of size

o = \left\lfloor \frac{n + 2p - k}{s} \right\rfloor + 1 o = \left\lfloor \frac{n + 2p - k}{s} \right\rfloor + 1
Convolutional Neural Networks · Convolution18

Learned Filters

Instead of hard-coding what the filters look for, the numbers in each filter are weights, learned during training like every other weight.

A filter is a neuron with a few weights and a bias, used at every position. The first layer of our CIFAR CNN against a fully connected layer with the same 32,768 outputs (32 maps of 32 × 32):

layerweights
convolutional, 32 filters of 3 × 3 × 3896
fully connected, 3,072 → 32,768100,696,064

This is called weight sharing.

Convolutional Neural Networks · Convolution19

Pooling

Pooling, or downsampling, is a method through which we can summarize small neighborhoods of a feature map to allow filters to match even if what we're looking for isn't perfectly aligned

The most common version keeps the largest number in each 2 × 2 window, halving the map both ways.

Convolutional Neural Networks · Pooling20

Pooling the Zero

0 0 0 0 0 0 0 0 0 0 0 10 10 0 0 0 0 0 12 26 32 14 4 0 0 5 7 9 4 2 2 0 0 1 1 0 0 0 0 1 3 0 0 0 0 0 5 8 4 9 0 0 0 10 19 7 2 16 21 29 27 22 12 0 0 10 10 0 5 26 32 4 3 1 0 8 16 29 27 19 0 0 0 0 0 0 0 0 0 0 0 0 10 10 0 0 0 0 0 12 26 32 14 4 0 0 5 7 9 4 2 2 0 0 1 1 0 0 0 0 0 3 0 0 0 0 0 5 0 4 9 0 0 0 10 19 0 2 16 21 29 27 22 12 0 0 10 0 0 12 32 14 3 1 0 5 4 21 29 22 max pool the zero max pool moved right feature map (8 × 8) pooled (4 × 4)

The top edge's 32 moved in the map but not in the pooled map. The bottom edge's 29 crossed into the next window, so pooling only forgives a shift that stays inside a window.

Convolutional Neural Networks · Pooling21

Pooling a Moved Photo

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ the photo photo feature map (32 × 32) after max pool (16 × 16) moved 1 pixel right changed 26% changed 18%

Over all 32 first-layer filters, a one-pixel move changes the maps by 37% and the pooled maps by 25%. Pooling softens a small shift, it doesn't remove it.

Convolutional Neural Networks · Pooling22

Max and Average Pooling

The two common types are max pooling and average pooling. The window of the zero's map that holds the 32:

\max(32, 14, 4, 2) = 32 \tfrac{1}{4}(32 + 14 + 4 + 2) = 13 \max(32, 14, 4, 2) = 32; \tfrac{1}{4}(32 + 14 + 4 + 2) = 13

Networks that use max pooling tend to perform better than those using average pooling, so when people speak of pooling they usually mean max pooling

Pooling has no weights, and cuts every map to a quarter of its size.

Convolutional Neural Networks · Pooling23

Pooling Layers (exercise)

Compute what the average and max pooling output would be.

3 2 −3 2 1 6 20 5 4 −13 2 6 −2 3 9 3 Average Max
Convolutional Neural Networks · Pooling24

Receptive Fields

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ the zero, padded (input) pooled first-layer map (4 × 4)

One 3 × 3 filter sees 9 pixels, enough for an edge but not for a zero. After a 2 × 2 pool each cell summarizes a 4 × 4 patch, so a 3 × 3 filter on top sees 8 × 8 pixels, the whole digit.

The patch one output can see is its receptive field, and it grows with every layer.

Convolutional Neural Networks · CNNs25

Hierarchical Filters

The intuition is that earlier filters learn simple, low-level features like edges which then get passed into later (deeper) filters that learn to recognize more complex things like corners, stripes and parts of objects from the lower level filters output

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ layer 1: 3 × 3 patches layer 2: 8 × 8 patches

What excites our CIFAR CNN most: a layer-1 unit sees 3 × 3 pixels and finds edges and single colors, a layer-2 unit sees 8 × 8 and finds corners, strokes and stripes.

Convolutional Neural Networks · CNNs26

Where the Idea Came From

In 1959 Lettvin, Maturana, McCulloch and Pitts found cells in a frog's eye that fire for small dark moving spots (bug detectors, more or less).

It was somewhat replicated in humans: Quiroga et al. (2005) found a single neuron that fired for photos of Jennifer Aniston and very little else.

Meme: a monkey with a cartoon brain labelled Neuron activation, captioned Monkey sees action, next to a photo of Jennifer Aniston
Convolutional Neural Networks · CNNs27

Simple and Complex Cells

In 1959 Hubel and Wiesel recorded single neurons in a cat's visual cortex while showing it bars of light. Each simple cell fired for an edge or bar at one angle, in one small patch of what the cat saw (its receptive field).

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0° 45° 90° 135°

Complex cells fired for the same angle anywhere in a larger patch (a bit like pooling). Hubel and Wiesel shared the 1981 Nobel Prize for this.

Convolutional Neural Networks · CNNs28

The Neocognitron (1980)

Kunihiko Fukushima took Hubel and Wiesel's two kinds of cells and stacked them into a network: layers of S-cells (one pattern, one small patch) alternating with layers of C-cells (the same pattern, anywhere nearby).

Neocognitron (1980) input S-cells C-cells S-cells C-cells output CNN image conv pool conv pool fully conn. simple cells: one pattern, one patch simple cells: one pattern, one patch complex cells: same pattern, nearby complex cells: same pattern, nearby S for simple, C for complex, after Hubel and Wiesel

Swap S for convolution and C for pooling and it's a CNN. It learned its filters without labels. LeCun et al. (1989) trained one with backprop instead, to read handwritten zip codes.

Convolutional Neural Networks · CNNs29

CNN Dimensions (exercise)

Convolutional Neural Networks · CNNs30

CNN Architecture

Blocks of convolution, ReLU and pooling learn a learned representation, then a feed-forward network classifies it. Our CIFAR CNN:

layeroutput shapeweights
photo3 × 32 × 32
conv, 32 filters of 3 × 3 × 3, padding 1, ReLU32 × 32 × 32896
max pool 2 × 232 × 16 × 160
conv, 64 filters of 3 × 3 × 32, padding 1, ReLU64 × 16 × 1618,496
max pool 2 × 264 × 8 × 80
flatten4,0960
fully connected, ReLU5122,097,664
fully connected, ReLU256131,328
fully connected, softmax102,570
total2,250,954
Convolutional Neural Networks · CNNs31

CNNs in PyTorch

The convolutional block: two rounds of convolution, ReLU and pooling.

import torch
from torch import nn
cnn = nn.Sequential(
# the convolutional block
nn.Conv2d(3, 32, kernel_size=3, padding=1),
nn.ReLU(),
nn.MaxPool2d(2),
nn.Conv2d(32, 64, kernel_size=3, padding=1),
nn.ReLU(),
nn.MaxPool2d(2),
Convolutional Neural Networks · CNNs32

CNNs in PyTorch

Then the fully connected block, which flattens the maps and classifies:

# the fully connected block
nn.Flatten(),
nn.Linear(64 * 8 * 8, 512), nn.ReLU(),
nn.Linear(512, 256), nn.ReLU(),
nn.Linear(256, 10),
)
x = torch.zeros(5, 3, 32, 32) # 5 blank photos
print(cnn(x).shape)
print(sum(p.numel() for p in cnn.parameters()))
Output
torch.Size([5, 10])
2250954
Convolutional Neural Networks · CNNs33

Where the Weights Are

networkweightsunseen photos right
feed-forward1,707,27449.5%
CNN2,250,95471.1%

The CNN has more, not fewer, weights than the feed-forward network, and 2,231,562 of them (99%) are in its fully connected layers. Its two convolutional layers have only 19,392.

So it isn't winning by size, it's winning by how its few filter weights are used.

Convolutional Neural Networks · CNNs34

Training a CNN

Softmax, cross-entropy and backprop, same as before. The one new piece: each filter weight was used at every position, so its gradient adds up all of them:

\frac{\partial ℒ}{\partial K(m, n)} = \sum_{i}\sum_{j} \frac{\partial ℒ}{\partial S(i, j)}\; I(i + m,\, j + n) \frac{\partial ℒ}{\partial K(m, n)} = \sum_{i}\sum_{j} \frac{\partial ℒ}{\partial S(i, j)}\; I(i + m,\, j + n)

Which is itself a cross-correlation, of the image with the feature map's gradient.

Convolutional Neural Networks · Training35

What Our CNN Learned

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 32 learned first-layer filters (3 × 3) what 7 of them find in the photo (after the ReLU)

Trained on 50,000 photos it gets 71.1% of unseen photos right. Nobody told its filters what to look for, but most of them find an edge at some angle.

Convolutional Neural Networks · Training36

Learned Filters (MNIST)

A large grid of small learned filters from a CNN trained on MNIST, with ten of them enlarged: stripes and curved strokes in yellow and blue, and a small meme in the corner
Convolutional Neural Networks · Training37

CNNs and the Visual Cortex

Trained on 1.2 million photos, AlexNet's 96 first-layer filters came out looking a lot like V1's simple cells: edges at every angle, plus a few blobs of color (Krizhevsky, Sutskever and Hinton, 2012). Nobody designed them.

Further up, a CNN trained only to recognize objects had middle layers that predicted neurons in V4 and top layers that predicted neurons in IT, the brain's object area (Yamins et al., 2014).

Built to copy the visual cortex, then trained to do its job, CNNs ended up a lot like it.

Convolutional Neural Networks · Training38

Why the CNN Wins

  1. Each filter looks at a small patch, because neighboring pixels belong together.
  2. The same filter is used everywhere in the photo, because an edge is an edge wherever it is.
  3. Layers build bigger patterns out of smaller ones.

All three depend on where each pixel is, which is exactly what flattening threw away. Shuffle the pixels and the CNN falls from 71.1% to 50.4%, right back to the feed-forward network.

Convolutional Neural Networks · Training39

Efficiency in CNN Implementations (homework)

These nested loops are really really really slow! It took me ~9 hours to run my CNN this way.

If we're particularly clever, we can collapse it all into a single matrix multiplication!

(this took me from ~9 hours to ~5 minutes)

Convolutional Neural Networks · Implementation40

Python CNN by Hand - Convolutional Layer

Same three methods as last lecture. Each filter is one row of W, holding its c_in * k * k weights.

import torch
import torch.nn.functional as F
class ConvLayer:
def __init__(self, c_in, c_out, k):
self.k = k
n = c_in * k * k # weights in one filter
self.W = torch.randn(c_out, n) * (2 / n)**0.5
self.b = torch.zeros(c_out)
Convolutional Neural Networks · CNN from Scratch41

Python CNN by Hand - Convolutional Layer, Forward

The trick is im2col: cut every patch out as one row of a matrix, and the whole layer is a single @.

def forward(self, X):
N, C, H, W = X.shape
k, Ho, Wo = self.k, H - self.k + 1, W - self.k + 1
self.shape = X.shape # keep for backward
# im2col: every k x k patch becomes one row
cols = F.unfold(X, kernel_size=k) # (N, C*k*k, L)
self.cols = cols.transpose(1, 2).reshape(
N * Ho * Wo, C * k * k)
Z = self.cols @ self.W.T + self.b # every patch
Z = Z.reshape(N, Ho, Wo, -1)
return Z.permute(0, 3, 1, 2) # (N, C_out, Ho, Wo)
Convolutional Neural Networks · CNN from Scratch42

Python CNN by Hand - Convolutional Layer, Backward

F.fold is F.unfold in reverse: it adds every patch's gradient back where that patch came from, summing where patches overlap.

def backward(self, dZ):
N, C, H, W = self.shape
k, Ho, Wo = self.k, H - self.k + 1, W - self.k + 1
dZ = dZ.permute(0, 2, 3, 1).reshape(N * Ho * Wo,
-1)
self.dW = dZ.T @ self.cols # dL/dW
self.db = dZ.sum(dim=0) # dL/db
dcols = (dZ @ self.W).reshape(N, Ho * Wo, -1)
# col2im: each patch's gradient back in place
dX = F.fold(dcols.transpose(1, 2), (H, W),
kernel_size=k)
return dX # dL/dX, sent back
def update(self, lr):
self.W -= lr * self.dW
self.b -= lr * self.db
Convolutional Neural Networks · CNN from Scratch43

Python CNN by Hand - Max Pooling, Forward

class MaxPool:
def forward(self, X):
N, C, H, W = X.shape
self.shape = X.shape # keep for backward
# each 2 x 2 window in its own last dimension
win = X.reshape(N, C, H // 2, 2, W // 2, 2)
win = win.permute(0, 1, 2, 4, 3, 5).reshape(
N, C, H // 2, W // 2, 4)
out, self.idx = win.max(dim=-1) # keep the winner
return out
Convolutional Neural Networks · CNN from Scratch44

Python CNN by Hand - Max Pooling, Backward

Only the pixel that won the max gets a gradient, every other pixel in its window gets 0.

def backward(self, dA):
N, C, H, W = self.shape
g = torch.zeros(N, C, H // 2, W // 2, 4)
# each gradient goes to the pixel that won
g.scatter_(-1, self.idx.unsqueeze(-1),
dA.unsqueeze(-1))
g = g.reshape(N, C, H // 2, W // 2, 2, 2)
g = g.permute(0, 1, 2, 4, 3, 5)
return g.reshape(N, C, H, W)
def update(self, lr):
pass # no weights to learn
Convolutional Neural Networks · CNN from Scratch45

Python CNN by Hand - Linear Layer

The same LinearLayer as last lecture:

class LinearLayer:
def __init__(self, n_in, n_out):
self.W = torch.randn(n_out, n_in) * (2/n_in)**0.5
self.b = torch.zeros(n_out)
def forward(self, X):
self.X = X # keep for backward
return X @ self.W.T + self.b
def backward(self, dY):
self.dW = dY.T @ self.X # dL/dW
self.db = dY.sum(dim=0) # dL/db
return dY @ self.W # dL/dX, sent back
def update(self, lr):
self.W -= lr * self.dW
self.b -= lr * self.db
Convolutional Neural Networks · CNN from Scratch46

Python CNN by Hand - ReLU and Flatten

class ReLU:
def forward(self, Z):
self.Z = Z # saved
return Z.clamp(min=0)
def backward(self, dA):
return dA * (self.Z > 0).float() # dL/dZ
def update(self, lr):
pass # no weights to learn
class Flatten:
def forward(self, X):
self.shape = X.shape # keep for backward
return X.reshape(X.shape[0], -1)
def backward(self, dY):
return dY.reshape(self.shape) # un-flatten
def update(self, lr):
pass # no weights to learn
Convolutional Neural Networks · CNN from Scratch47

Python CNN by Hand - Network

The same Network as last lecture, it doesn't care what kind of layers it holds.

class Network:
def __init__(self, layers):
self.layers = layers
def forward(self, X):
for layer in self.layers:
X = layer.forward(X)
return X # logits
def backward(self, y_hat, y):
grad = y_hat - y # softmax + cross-entropy
for layer in reversed(self.layers):
grad = layer.backward(grad)
def update(self, lr):
for layer in self.layers:
layer.update(lr)
Convolutional Neural Networks · CNN from Scratch48

Training a CNN by Hand - The Digits

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
d = load_digits()
X_tr, X_te, y_tr, y_te = train_test_split(
d.images / 16, d.target, test_size=0.25,
random_state=0)
# (N, 1, 8, 8): one channel per digit
X_tr = torch.tensor(X_tr).float().unsqueeze(1)
X_te = torch.tensor(X_te).float().unsqueeze(1)
Y_tr = torch.eye(10)[torch.tensor(y_tr)] # one-hot
Convolutional Neural Networks · CNN from Scratch49

Training a CNN by Hand

torch.manual_seed(0)
model = Network([ConvLayer(1, 8, 3), MaxPool(), ReLU(),
Flatten(), LinearLayer(8 * 3 * 3, 10)])
for epoch in range(30):
for i in range(0, len(X_tr), 32):
logits = model.forward(X_tr[i:i + 32])
y_hat = torch.softmax(logits, dim=1)
model.backward(y_hat, Y_tr[i:i + 32])
model.update(0.01)
pred = model.forward(X_te).argmax(dim=1)
print('unseen digits right:',
(pred == torch.tensor(y_te)).float().mean())
Output
unseen digits right: tensor(0.9778)

That's the browser's run. Desktop PyTorch starts from different random weights and gets 98.7%. A feed-forward network on the same digits gets 97.3%.

Convolutional Neural Networks · CNN from Scratch50

Training CNNs (homework bonus)

We could ask the same four questions I asked at the start of class for a CNN too!

Convolutional Neural Networks · CNN from Scratch51

Training CNNs (homework bonus)

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
class CNN(nn.Module):
def __init__(self):
super(CNN, self).__init__()
self.conv_layers = nn.Sequential(
nn.Conv2d(3, 32, kernel_size=3, padding=1),
nn.ReLU(),
nn.MaxPool2d(2),
nn.Conv2d(32, 64, kernel_size=3, padding=1),
nn.ReLU(),
nn.MaxPool2d(2)
)
Convolutional Neural Networks · CNN from Scratch52

Training CNNs (homework bonus)

self.fc_layers = nn.Sequential(
nn.Flatten(),
nn.Linear(64 * 8 * 8, 512),
nn.Sigmoid(),
nn.Linear(512, 256),
nn.Sigmoid(),
nn.Linear(256, 10)
)
def forward(self, x):
x = self.conv_layers(x)
return self.fc_layers(x)
Convolutional Neural Networks · CNN from Scratch53

Training CNNs: A and B

Convolutional Neural Networks · CNN from Scratch54

Training CNNs: C and D

Convolutional Neural Networks · CNN from Scratch55

Computer Vision

Prior to 2020 and the paper "An image is worth 16x16 words" basically all of computer vision was done using:

Convolutional Neural Networks (CNNs)

CNNs are still widely used today and were partly an inspiration for transformers

Convolutional Neural Networks · Computer vision56

Vision Tasks

There are a ton of new tasks in computer vision we haven't seen so far!

These include: Image Classification, Object Detection, Semantic Segmentation, Instance Segmentation, Panoptic Segmentation, Keypoint Detection, Optical Character Recognition (OCR), Image Captioning, Visual Question Answering (VQA), Image-to-Image Translation, Super-Resolution, Image Restoration, 3D Reconstruction, Depth Estimation

Convolutional Neural Networks · Computer vision57

Next: Sequences

When our CNN labels a photo, it doesn't matter what photo came before it or comes after. Shuffle the photos and every one gets the same label. Shuffle the words of a sentence and it says something else.

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ airplane ship cat airplane ship truck truck cat photos in order shuffled: same labels words in order cheer cheer for old notre dame shuffled: a different sentence notre cheer for cheer dame old
Convolutional Neural Networks · Computer vision58