Unit 03 · Chapter 2

Convolutional Neural Networks

Convolutional Neural Networks01

Dataset - MNIST

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ N O T R E D A M E 7 2 a b d ? ? ? ? ? ? ? ? ? ? ? ? police archive (EMNIST): labelled, 28 × 28 Professor Dingler's note: no labels
Convolutional Neural Networks · Digits02

Reviewing Neural Networks

An MNIST 5 drawn as a grid of pixel values
digit0123456789
ŷ0.110.140.130.210.100.090.080.060.050.03
y0.000.000.000.000.001.000.000.000.000.00
Convolutional Neural Networks · Digits03

Why do we care about CNNs?

We get 99% on MNIST without using anything fancy, why do we care?

Training accuracy over five epochs on MNIST: the CNN goes from about 94.7% to 99.4%, the feed-forward network from about 93.2% to 98.8%
Convolutional Neural Networks · Digits04

CIFAR-10

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ airplane automobile bird cat deer dog frog horse ship truck

CIFAR-10 is 60,000 tiny color photos, 32 × 32 pixels, of 10 kinds of thing. We train on 50,000 and keep 10,000 aside as  . A lot harder than handwritten digits.

Convolutional Neural Networks · Photos05

A Feed-Forward Network on Photos

To hand a photo to a feed-forward network we   it: 32 × 32 pixels with 3 colors each is   inputs.

import torch
from torch import nn
ffn = nn.Sequential(
nn.Flatten(), # 3 x 32 x 32 -> 3,072
nn.Linear(3072, 512), nn.ReLU(),
nn.Linear(512, 256), nn.ReLU(),
nn.Linear(256, 10),
)
x = torch.zeros(5, 3, 32, 32) # 5 blank photos
print(ffn(x).shape)
print(sum(p.numel() for p in ffn.parameters()))
Output
torch.Size([5, 10])
1707274
Convolutional Neural Networks · Photos06

CNN vs. FFN on CIFAR-10

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 1 2 3 4 5 6 7 8 9 10 epoch 0 10 20 30 40 50 60 70 80 unseen photos right (%) 71.1% 49.5% guessing: 10% CNN feed-forward network

Same photos, same training: the FFN gets 49.5% of unseen photos right, a CNN gets 71.1%.

Convolutional Neural Networks · Photos07

Shuffling the Pixels

Shuffle every photo's pixels into the same random order (nothing is lost, it's just rearranged) and train both networks again:

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ a photo its pixels shuffled
unseen photos rightfeed-forwardCNN
photos as they are49.5%71.1%
pixels shuffled48.8%50.4%

The FFN doesn't use the layout. The CNN's whole advantage is making use of the overall structure of the image

Convolutional Neural Networks · Photos08

Filters

A photo is 3,072 numbers, too many to work by hand, so we'll build our first one on an 8 × 8 handwritten zero (pixels from 0 to 16). We want a neuron that looks for  in an image

0 0 5 13 9 1 0 0 0 0 13 5 0 0 3 15 8 0 0 4 12 8 0 0 5 8 0 0 9 8 0 0 4 11 0 1 12 7 0 0 2 14 5 10 12 0 0 0 0 6 13 10 0 0 0 15 10 15 2 0 11 0 0 8 1 1 1 0 0 0 -1 -1 -1 32 15 10 15 2 0 11 0 0 8 × = patch filter output

Lay a 3 × 3 grid of weights over a 3 × 3 patch, multiply each pixel by the weight on top of it, and  : 15 + 10 + 15 − 8 = 32.

A small matrix of weights encoding some pattern we're looking for in an image is called a filter or a  .

Convolutional Neural Networks · Convolution09

A Horizontal Edge Filter

1s along the top, 0s through the middle, −1s along the bottom: it adds up the top row and  .

-1 -1 -1 0 0 0 1 1 1 image kernel output
Convolutional Neural Networks · Convolution10

Convolution

Slide the filter across the image  , doing the same multiply and add at every stop. A 3 × 3 filter fits in 6 places across an 8 pixel row:

import torch
from sklearn.datasets import load_digits
zero = torch.tensor(load_digits().images[0],
dtype=torch.float32)
K = torch.tensor([[ 1., 1., 1.],
[ 0., 0., 0.],
[-1., -1., -1.]])
out = torch.zeros(6, 6)
for i in range(6):
for j in range(6):
patch = zero[i:i+3, j:j+3]
out[i, j] = (patch * K).sum()
print(out.long())
Output
tensor([[-13,  -2,  10,  10,  -9, -18],
        [ -3,  12,  26,  32,  14,   4],
        [  5,   7,   9,   4,   2,   2],
        [  1,   1,   0,  -5,  -4,  -3],
        [ -3,  -8, -21, -18,  -5,   5],
        [  9,  -4, -17, -10,  10,  19]])
Convolutional Neural Networks · Convolution11

Cross-Correlation

CNNs are named after convolution, which combines two functions to create a third by   before sliding it:

Without the flip it's called cross-correlation, and that's what we just did:

S(i, j) = (I \star K)(i, j) = \sum_{m}\sum_{n} I(i + m,\, j + n)\, K(m, n) S(i, j) = (I \star K)(i, j) = \sum_{m}\sum_{n} I(i + m,\, j + n)\, K(m, n)

Nearly every library computes cross-correlation and calls it convolution. The kernel is learned anyway, so a network that needed it flipped would  .

Convolutional Neural Networks · Convolution12

Feature Maps

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ the zero (8 × 8) -13 -2 10 10 -9 -18 -3 12 26 32 14 4 5 7 9 4 2 2 1 1 0 -5 -4 -3 -3 -8 -21 -18 -5 5 9 -4 -17 -10 10 19 horizontal-edge filter -33 -27 14 3 6 27 -40 -10 30 -17 -11 34 -35 10 35 -26 -24 28 -31 13 30 -29 -22 29 -33 6 22 -28 -4 33 -31 -12 10 -6 14 24 vertical-edge filter

The grid a filter makes is a feature map, one number per position saying  .

Turned on its side, the filter picks out the zero's sides instead (−40 and 35). Different filters find different patterns, so a layer applies   and keeps every map.

Convolutional Neural Networks · Convolution13

Edges in a Photo

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ the photo (32 × 32) horizontal-edge filter vertical-edge filter

The same two filters on a CIFAR photo (turned grey by averaging its three colors): the horizontal-edge filter traces  , the vertical-edge filter finds  .

A real CNN's first filters are 3 × 3 × 3, so they can look for   too.

Convolutional Neural Networks · Convolution14

Output Size

4 8 12 13 14 15 16 0 -1 0 -1 5 -1 0 -1 0 5 12 15 16 11 1 2 3 5 6 7 9 10 11 ★ + = input kernel bias output

A k × k filter on an n × n image fits in   places across, so a 4 × 4 image gives a 2 × 2 output.

Each output also gets a bias, the same job it did in our neurons: 5·6 − 2 − 5 − 7 − 10 + 5 = 11.

Convolutional Neural Networks · Convolution15

Padding

Shrinking is a nuisance: two 3 × 3 layers take our digit down to 4 × 4, and   get looked at less.

Padding is  . One ring makes our zero 10 × 10, the filter fits in 8 places, and the map is  .

-1 -1 -1 0 0 0 1 1 1 image kernel output
Convolutional Neural Networks · Convolution16

Stride

stride 2 input (7 × 7) output (3 × 3)

The stride is  . Stride 2 over a 7 × 7 input makes 3 stops across, so the output is 3 × 3.

A cheap way to shrink a map, at the price of skipping positions where a pattern might have matched.

Convolutional Neural Networks · Convolution17

Output Size, in General

An n × n input, a k × k filter, padding p and stride s give an output of size

o = \left\lfloor \frac{n + 2p - k}{s} \right\rfloor + 1 o = \left\lfloor \frac{n + 2p - k}{s} \right\rfloor + 1
Convolutional Neural Networks · Convolution18

Learned Filters

Instead of hard-coding what the filters look for, the numbers in each filter are weights,   like every other weight.

A filter is a neuron with a few weights and a bias, used at every position. The first layer of our CIFAR CNN against a fully connected layer with the same 32,768 outputs (32 maps of 32 × 32):

layerweights
convolutional, 32 filters of 3 × 3 × 3896
fully connected, 3,072 → 32,768100,696,064

This is called weight sharing.

Convolutional Neural Networks · Convolution19

Pooling

Pooling, or  , is a method through which we can   to allow filters to match even if what we're looking for isn't perfectly aligned

The most common version keeps   in each 2 × 2 window, halving the map both ways.

Convolutional Neural Networks · Pooling20

Pooling the Zero

0 0 0 0 0 0 0 0 0 0 0 10 10 0 0 0 0 0 12 26 32 14 4 0 0 5 7 9 4 2 2 0 0 1 1 0 0 0 0 1 3 0 0 0 0 0 5 8 4 9 0 0 0 10 19 7 2 16 21 29 27 22 12 0 0 10 10 0 5 26 32 4 3 1 0 8 16 29 27 19 0 0 0 0 0 0 0 0 0 0 0 0 10 10 0 0 0 0 0 12 26 32 14 4 0 0 5 7 9 4 2 2 0 0 1 1 0 0 0 0 0 3 0 0 0 0 0 5 0 4 9 0 0 0 10 19 0 2 16 21 29 27 22 12 0 0 10 0 0 12 32 14 3 1 0 5 4 21 29 22 max pool the zero max pool moved right feature map (8 × 8) pooled (4 × 4)

The top edge's 32 moved in the map but  . The bottom edge's 29 crossed into the next window, so pooling only forgives a shift that stays inside a window.

Convolutional Neural Networks · Pooling21

Pooling a Moved Photo

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ the photo photo feature map (32 × 32) after max pool (16 × 16) moved 1 pixel right changed 26% changed 18%

Over all 32 first-layer filters, a one-pixel move changes the maps by 37% and the pooled maps by 25%. Pooling softens a small shift, it doesn't  .

Convolutional Neural Networks · Pooling22

Max and Average Pooling

The two common types are max pooling and average pooling. The window of the zero's map that holds the 32:

\max(32, 14, 4, 2) = 32 \tfrac{1}{4}(32 + 14 + 4 + 2) = 13 \max(32, 14, 4, 2) = 32; \tfrac{1}{4}(32 + 14 + 4 + 2) = 13

Networks that use max pooling   than those using average pooling, so when people speak of pooling they usually mean max pooling

Pooling has  , and cuts every map to   of its size.

Convolutional Neural Networks · Pooling23

Pooling Layers (exercise)

Compute what the average and max pooling output would be.

3 2 −3 2 1 6 20 5 4 −13 2 6 −2 3 9 3 Average Max
Convolutional Neural Networks · Pooling24

Receptive Fields

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ the zero, padded (input) pooled first-layer map (4 × 4)

One 3 × 3 filter sees 9 pixels, enough for   but not for a zero. After a 2 × 2 pool each cell summarizes a 4 × 4 patch, so a 3 × 3 filter on top sees  , the whole digit.

The patch one output can see is its receptive field, and it  .

Convolutional Neural Networks · CNNs25

Hierarchical Filters

The intuition is that earlier filters learn   like edges which then get passed into   filters that learn to recognize more complex things like corners, stripes and parts of objects from the lower level filters output

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ layer 1: 3 × 3 patches layer 2: 8 × 8 patches

What excites our CIFAR CNN most: a layer-1 unit sees 3 × 3 pixels and finds  , a layer-2 unit sees 8 × 8 and finds  .

Convolutional Neural Networks · CNNs26

Where the Idea Came From

In 1959 Lettvin, Maturana, McCulloch and Pitts found cells in a frog's eye that fire for   (bug detectors, more or less).

It was somewhat replicated in humans: Quiroga et al. (2005) found a single neuron that fired for   and very little else.

Meme: a monkey with a cartoon brain labelled Neuron activation, captioned Monkey sees action, next to a photo of Jennifer Aniston
Convolutional Neural Networks · CNNs27

Simple and Complex Cells

In 1959 Hubel and Wiesel recorded single neurons in a cat's visual cortex while showing it bars of light. Each simple cell fired for an edge or bar at one  , in one small patch of what the cat saw (its receptive field).

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0° 45° 90° 135°

Complex cells fired for the same angle anywhere in a larger patch (a bit like pooling). Hubel and Wiesel shared the 1981 Nobel Prize for this.

Convolutional Neural Networks · CNNs28

The Neocognitron (1980)

Kunihiko Fukushima took Hubel and Wiesel's two kinds of cells and stacked them into a network: layers of S-cells (one pattern, one small patch) alternating with layers of C-cells (the same pattern, anywhere nearby).

Neocognitron (1980) input S-cells C-cells S-cells C-cells output CNN image conv pool conv pool fully conn. simple cells: one pattern, one patch simple cells: one pattern, one patch complex cells: same pattern, nearby complex cells: same pattern, nearby S for simple, C for complex, after Hubel and Wiesel

Swap S for   and C for   and it's a CNN. It learned its filters without labels. LeCun et al. (1989) trained one with backprop instead, to read handwritten zip codes.

Convolutional Neural Networks · CNNs29

CNN Dimensions (exercise)

Convolutional Neural Networks · CNNs30

CNN Architecture

Blocks of   learn a learned representation, then a   classifies it. Our CIFAR CNN:

layeroutput shapeweights
photo3 × 32 × 32
conv, 32 filters of 3 × 3 × 3, padding 1, ReLU32 × 32 × 32896
max pool 2 × 232 × 16 × 160
conv, 64 filters of 3 × 3 × 32, padding 1, ReLU64 × 16 × 1618,496
max pool 2 × 264 × 8 × 80
flatten4,0960
fully connected, ReLU5122,097,664
fully connected, ReLU256131,328
fully connected, softmax102,570
total2,250,954
Convolutional Neural Networks · CNNs31

CNNs in PyTorch

The convolutional block: two rounds of  .

import torch
from torch import nn
cnn = nn.Sequential(
# the convolutional block
nn.Conv2d(3, 32, kernel_size=3, padding=1),
nn.ReLU(),
nn.MaxPool2d(2),
nn.Conv2d(32, 64, kernel_size=3, padding=1),
nn.ReLU(),
nn.MaxPool2d(2),

Then the fully connected block, which  :

# the fully connected block
nn.Flatten(),
nn.Linear(64 * 8 * 8, 512), nn.ReLU(),
nn.Linear(512, 256), nn.ReLU(),
nn.Linear(256, 10),
)
x = torch.zeros(5, 3, 32, 32) # 5 blank photos
print(cnn(x).shape)
print(sum(p.numel() for p in cnn.parameters()))
Output
torch.Size([5, 10])
2250954
Convolutional Neural Networks · CNNs32

Where the Weights Are

networkweightsunseen photos right
feed-forward1,707,27449.5%
CNN2,250,95471.1%

The CNN has   weights than the feed-forward network, and 2,231,562 of them (99%) are in its fully connected layers. Its two convolutional layers have only 19,392.

So it isn't winning by size, it's winning by  .

Convolutional Neural Networks · CNNs33

Training a CNN

Softmax, cross-entropy and backprop, same as before. The one new piece: each filter weight was used  , so its gradient  :

\frac{\partial ℒ}{\partial K(m, n)} = \sum_{i}\sum_{j} \frac{\partial ℒ}{\partial S(i, j)}\; I(i + m,\, j + n) \frac{\partial ℒ}{\partial K(m, n)} = \sum_{i}\sum_{j} \frac{\partial ℒ}{\partial S(i, j)}\; I(i + m,\, j + n)

Which is itself a cross-correlation, of the image with the feature map's gradient.

Convolutional Neural Networks · Training34

What Our CNN Learned

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 32 learned first-layer filters (3 × 3) what 7 of them find in the photo (after the ReLU)

Trained on 50,000 photos it gets 71.1% of unseen photos right. Nobody told its filters what to look for, but most of them  .

Convolutional Neural Networks · Training35

Learned Filters (MNIST)

A large grid of small learned filters from a CNN trained on MNIST, with ten of them enlarged: stripes and curved strokes in yellow and blue, and a small meme in the corner
Convolutional Neural Networks · Training36

CNNs and the Visual Cortex

Trained on 1.2 million photos, AlexNet's 96 first-layer filters came out looking a lot like V1's simple cells: edges at every  , plus a few blobs of color (Krizhevsky, Sutskever and Hinton, 2012). Nobody designed them.

Further up, a CNN trained only to recognize objects had middle layers that predicted neurons in V4 and top layers that predicted neurons in IT, the brain's object area (Yamins et al., 2014).

Built to copy the visual cortex, then trained to do its job, CNNs ended up a lot like it.

Convolutional Neural Networks · Training37

Why the CNN Wins

  1. Each filter looks at  , because neighboring pixels belong together.
  2. The same filter is used  , because an edge is an edge wherever it is.
  3. Layers build  .

All three depend on where each pixel is, which is exactly what flattening threw away. Shuffle the pixels and the CNN falls from 71.1% to 50.4%,  .

Convolutional Neural Networks · Training38

Efficiency in CNN Implementations (homework)

These nested loops are really really really slow! It took me ~9 hours to run my CNN this way.

If we're particularly clever, we can collapse it all into a single matrix multiplication!

(this took me from ~9 hours to ~5 minutes)

Convolutional Neural Networks · Implementation39

Python CNN by Hand - Convolutional Layer

Same three methods as last lecture. Each filter is  , holding its c_in * k * k weights.

import torch
import torch.nn.functional as F
class ConvLayer:
def __init__(self, c_in, c_out, k):
self.k = k
n = c_in * k * k # weights in one filter
self.W = torch.randn(c_out, n) * (2 / n)**0.5
self.b = torch.zeros(c_out)
Convolutional Neural Networks · CNN from Scratch40

Python CNN by Hand - Convolutional Layer, Forward

The trick is im2col: cut every patch out as  , and the whole layer is  .

def forward(self, X):
N, C, H, W = X.shape
k, Ho, Wo = self.k, H - self.k + 1, W - self.k + 1
self.shape = X.shape # keep for backward
# im2col: every k x k patch becomes one row
cols = F.unfold(X, kernel_size=k) # (N, C*k*k, L)
self.cols = cols.transpose(1, 2).reshape(
N * Ho * Wo, C * k * k)
Z = self.cols @ self.W.T + self.b # every patch
Z = Z.reshape(N, Ho, Wo, -1)
return Z.permute(0, 3, 1, 2) # (N, C_out, Ho, Wo)
Convolutional Neural Networks · CNN from Scratch41

Python CNN by Hand - Convolutional Layer, Backward

F.fold is F.unfold in reverse: it adds every patch's gradient back  , summing where patches overlap.

def backward(self, dZ):
N, C, H, W = self.shape
k, Ho, Wo = self.k, H - self.k + 1, W - self.k + 1
dZ = dZ.permute(0, 2, 3, 1).reshape(N * Ho * Wo,
-1)
self.dW = dZ.T @ self.cols # dL/dW
self.db = dZ.sum(dim=0) # dL/db
dcols = (dZ @ self.W).reshape(N, Ho * Wo, -1)
# col2im: each patch's gradient back in place
dX = F.fold(dcols.transpose(1, 2), (H, W),
kernel_size=k)
return dX # dL/dX, sent back
def update(self, lr):
self.W -= lr * self.dW
self.b -= lr * self.db
Convolutional Neural Networks · CNN from Scratch42

Python CNN by Hand - Max Pooling, Forward

class MaxPool:
def forward(self, X):
N, C, H, W = X.shape
self.shape = X.shape # keep for backward
# each 2 x 2 window in its own last dimension
win = X.reshape(N, C, H // 2, 2, W // 2, 2)
win = win.permute(0, 1, 2, 4, 3, 5).reshape(
N, C, H // 2, W // 2, 4)
out, self.idx = win.max(dim=-1) # keep the winner
return out
Convolutional Neural Networks · CNN from Scratch43

Python CNN by Hand - Max Pooling, Backward

Only the pixel that won the max  , every other pixel in its window gets 0.

def backward(self, dA):
N, C, H, W = self.shape
g = torch.zeros(N, C, H // 2, W // 2, 4)
# each gradient goes to the pixel that won
g.scatter_(-1, self.idx.unsqueeze(-1),
dA.unsqueeze(-1))
g = g.reshape(N, C, H // 2, W // 2, 2, 2)
g = g.permute(0, 1, 2, 4, 3, 5)
return g.reshape(N, C, H, W)
def update(self, lr):
pass # no weights to learn
Convolutional Neural Networks · CNN from Scratch44

Python CNN by Hand - Linear Layer

The same LinearLayer as last lecture:

class LinearLayer:
def __init__(self, n_in, n_out):
self.W = torch.randn(n_out, n_in) * (2/n_in)**0.5
self.b = torch.zeros(n_out)
def forward(self, X):
self.X = X # keep for backward
return X @ self.W.T + self.b
def backward(self, dY):
self.dW = dY.T @ self.X # dL/dW
self.db = dY.sum(dim=0) # dL/db
return dY @ self.W # dL/dX, sent back
def update(self, lr):
self.W -= lr * self.dW
self.b -= lr * self.db
Convolutional Neural Networks · CNN from Scratch45

Python CNN by Hand - ReLU and Flatten

class ReLU:
def forward(self, Z):
self.Z = Z # saved
return Z.clamp(min=0)
def backward(self, dA):
return dA * (self.Z > 0).float() # dL/dZ
def update(self, lr):
pass # no weights to learn
class Flatten:
def forward(self, X):
self.shape = X.shape # keep for backward
return X.reshape(X.shape[0], -1)
def backward(self, dY):
return dY.reshape(self.shape) # un-flatten
def update(self, lr):
pass # no weights to learn
Convolutional Neural Networks · CNN from Scratch46

Python CNN by Hand - Network

The same Network as last lecture, it doesn't care  .

class Network:
def __init__(self, layers):
self.layers = layers
def forward(self, X):
for layer in self.layers:
X = layer.forward(X)
return X # logits
def backward(self, y_hat, y):
grad = y_hat - y # softmax + cross-entropy
for layer in reversed(self.layers):
grad = layer.backward(grad)
def update(self, lr):
for layer in self.layers:
layer.update(lr)
Convolutional Neural Networks · CNN from Scratch47

Training a CNN by Hand - The Digits

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
d = load_digits()
X_tr, X_te, y_tr, y_te = train_test_split(
d.images / 16, d.target, test_size=0.25,
random_state=0)
# (N, 1, 8, 8): one channel per digit
X_tr = torch.tensor(X_tr).float().unsqueeze(1)
X_te = torch.tensor(X_te).float().unsqueeze(1)
Y_tr = torch.eye(10)[torch.tensor(y_tr)] # one-hot
Convolutional Neural Networks · CNN from Scratch48

Training a CNN by Hand

torch.manual_seed(0)
model = Network([ConvLayer(1, 8, 3), MaxPool(), ReLU(),
Flatten(), LinearLayer(8 * 3 * 3, 10)])
for epoch in range(30):
for i in range(0, len(X_tr), 32):
logits = model.forward(X_tr[i:i + 32])
y_hat = torch.softmax(logits, dim=1)
model.backward(y_hat, Y_tr[i:i + 32])
model.update(0.01)
pred = model.forward(X_te).argmax(dim=1)
print('unseen digits right:',
(pred == torch.tensor(y_te)).float().mean())
Output
unseen digits right: tensor(0.9778)

That's the browser's run. Desktop PyTorch starts from different random weights and gets 98.7%. A feed-forward network on the same digits gets 97.3%.

Convolutional Neural Networks · CNN from Scratch49

Training CNNs (homework bonus)

We could ask the same four questions I asked at the start of class for a CNN too!

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
class CNN(nn.Module):
def __init__(self):
super(CNN, self).__init__()
self.conv_layers = nn.Sequential(
nn.Conv2d(3, 32, kernel_size=3, padding=1),
nn.ReLU(),
nn.MaxPool2d(2),
nn.Conv2d(32, 64, kernel_size=3, padding=1),
nn.ReLU(),
nn.MaxPool2d(2)
)
self.fc_layers = nn.Sequential(
nn.Flatten(),
nn.Linear(64 * 8 * 8, 512),
nn.Sigmoid(),
nn.Linear(512, 256),
nn.Sigmoid(),
nn.Linear(256, 10)
)
def forward(self, x):
x = self.conv_layers(x)
return self.fc_layers(x)
Convolutional Neural Networks · CNN from Scratch50

Training CNNs: A and B

Convolutional Neural Networks · CNN from Scratch51

Training CNNs: C and D

Convolutional Neural Networks · CNN from Scratch52

Computer Vision

Prior to 2020 and the paper "An image is worth 16x16 words" basically all of computer vision was done using:

 

CNNs are still widely used today and were partly an inspiration for transformers

Convolutional Neural Networks · Computer vision53

Vision Tasks

There are a ton of new tasks in computer vision we haven't seen so far!

These include: Image Classification, Object Detection, Semantic Segmentation, Instance Segmentation, Panoptic Segmentation, Keypoint Detection, Optical Character Recognition (OCR), Image Captioning, Visual Question Answering (VQA), Image-to-Image Translation, Super-Resolution, Image Restoration, 3D Reconstruction, Depth Estimation

Convolutional Neural Networks · Computer vision54

Next: Sequences

When our CNN labels a photo, it doesn't matter what photo came before it or comes after. Shuffle the photos and every one gets the same label. Shuffle the words of a sentence and  .

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ airplane ship cat airplane ship truck truck cat photos in order shuffled: same labels words in order cheer cheer for old notre dame shuffled: a different sentence notre cheer for cheer dame old
Convolutional Neural Networks · Computer vision55