Convolutional Neural Networks
Dataset - MNIST
Reviewing Neural Networks
| digit | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|---|
| 0.11 | 0.14 | 0.13 | 0.21 | 0.10 | 0.09 | 0.08 | 0.06 | 0.05 | 0.03 | |
| 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 |
Why do we care about CNNs?
We get 99% on MNIST without using anything fancy, why do we care?
CIFAR-10
CIFAR-10 is 60,000 tiny color photos, 32 × 32 pixels, of 10 kinds of thing. We train on 50,000 and keep 10,000 aside as . A lot harder than handwritten digits.
A Feed-Forward Network on Photos
To hand a photo to a feed-forward network we it: 32 × 32 pixels with 3 colors each is inputs.
torch.Size([5, 10]) 1707274
CNN vs. FFN on CIFAR-10
Same photos, same training: the FFN gets 49.5% of unseen photos right, a CNN gets 71.1%.
Shuffling the Pixels
Shuffle every photo's pixels into the same random order (nothing is lost, it's just rearranged) and train both networks again:
| unseen photos right | feed-forward | CNN |
|---|---|---|
| photos as they are | 49.5% | 71.1% |
| pixels shuffled | 48.8% | 50.4% |
The FFN doesn't use the layout. The CNN's whole advantage is making use of the overall structure of the image
Filters
A photo is 3,072 numbers, too many to work by hand, so we'll build our first one on an 8 × 8 handwritten zero (pixels from 0 to 16). We want a neuron that looks for in an image
Lay a 3 × 3 grid of weights over a 3 × 3 patch, multiply each pixel by the weight on top of it, and :
A small matrix of weights encoding some pattern we're looking for in an image is called a filter or a .
A Horizontal Edge Filter
1s along the top, 0s through the middle, −1s along the bottom: it adds up the top row and .
- ink on top, blank below:
- a blank patch:
- ink only at the bottom:
Convolution
Slide the filter across the image , doing the same multiply and add at every stop. A 3 × 3 filter fits in 6 places across an 8 pixel row:
tensor([[-13, -2, 10, 10, -9, -18], [ -3, 12, 26, 32, 14, 4], [ 5, 7, 9, 4, 2, 2], [ 1, 1, 0, -5, -4, -3], [ -3, -8, -21, -18, -5, 5], [ 9, -4, -17, -10, 10, 19]])
Cross-Correlation
CNNs are named after convolution, which combines two functions to create a third by before sliding it:
Without the flip it's called cross-correlation, and that's what we just did:
Nearly every library computes cross-correlation and calls it convolution. The kernel is learned anyway, so a network that needed it flipped would .
Feature Maps
The grid a filter makes is a feature map, one number per position saying .
Turned on its side, the filter picks out the zero's sides instead (−40 and 35). Different filters find different patterns, so a layer applies and keeps every map.
Edges in a Photo
The same two filters on a CIFAR photo (turned grey by averaging its three colors): the horizontal-edge filter traces , the vertical-edge filter finds .
A real CNN's first filters are 3 × 3 × 3, so they can look for too.
Output Size
A
Each output also gets a bias, the same job it did in our neurons:
Padding
Shrinking is a nuisance: two 3 × 3 layers take our digit down to 4 × 4, and get looked at less.
Padding is . One ring makes our zero 10 × 10, the filter fits in 8 places, and the map is .
Stride
The stride is . Stride 2 over a 7 × 7 input makes 3 stops across, so the output is 3 × 3.
A cheap way to shrink a map, at the price of skipping positions where a pattern might have matched.
Output Size, in General
An
- our zero, padding 1, stride 1:
(8 + 2 − 3)/1 + 1 = - stride 2:
(8 + 2 − 3)/2 = 3.5 , rounded down, plus 1 = - an MNIST digit, padding 1, stride 1:
(28 + 2 − 3)/1 + 1 =
Learned Filters
Instead of hard-coding what the filters look for, the numbers in each filter are weights, like every other weight.
A filter is a neuron with a few weights and a bias, used at every position. The first layer of our CIFAR CNN against a fully connected layer with the same 32,768 outputs (32 maps of 32 × 32):
| layer | weights |
|---|---|
| convolutional, 32 filters of 3 × 3 × 3 | 896 |
| fully connected, 3,072 → 32,768 | 100,696,064 |
This is called weight sharing.
Pooling
Pooling, or , is a method through which we can to allow filters to match even if what we're looking for isn't perfectly aligned
The most common version keeps in each 2 × 2 window, halving the map both ways.
Pooling the Zero
The top edge's 32 moved in the map but . The bottom edge's 29 crossed into the next window, so pooling only forgives a shift that stays inside a window.
Pooling a Moved Photo
Over all 32 first-layer filters, a one-pixel move changes the maps by 37% and the pooled maps by 25%. Pooling softens a small shift, it doesn't .
Max and Average Pooling
The two common types are max pooling and average pooling. The window of the zero's map that holds the 32:
Networks that use max pooling than those using average pooling, so when people speak of pooling they usually mean max pooling
Pooling has , and cuts every map to of its size.
Pooling Layers (exercise)
Compute what the average and max pooling output would be.
Receptive Fields
One 3 × 3 filter sees 9 pixels, enough for but not for a zero. After a 2 × 2 pool each cell summarizes a 4 × 4 patch, so a 3 × 3 filter on top sees , the whole digit.
The patch one output can see is its receptive field, and it .
Hierarchical Filters
The intuition is that earlier filters learn like edges which then get passed into filters that learn to recognize more complex things like corners, stripes and parts of objects from the lower level filters output
What excites our CIFAR CNN most: a layer-1 unit sees 3 × 3 pixels and finds , a layer-2 unit sees 8 × 8 and finds .
Where the Idea Came From
In 1959 Lettvin, Maturana, McCulloch and Pitts found cells in a frog's eye that fire for (bug detectors, more or less).
It was somewhat replicated in humans: Quiroga et al. (2005) found a single neuron that fired for and very little else.
Simple and Complex Cells
In 1959 Hubel and Wiesel recorded single neurons in a cat's visual cortex while showing it bars of light. Each simple cell fired for an edge or bar at one , in one small patch of what the cat saw (its receptive field).
Complex cells fired for the same angle anywhere in a larger patch (a bit like pooling). Hubel and Wiesel shared the 1981 Nobel Prize for this.
The Neocognitron (1980)
Kunihiko Fukushima took Hubel and Wiesel's two kinds of cells and stacked them into a network: layers of S-cells (one pattern, one small patch) alternating with layers of C-cells (the same pattern, anywhere nearby).
Swap S for and C for and it's a CNN. It learned its filters without labels. LeCun et al. (1989) trained one with backprop instead, to read handwritten zip codes.
CNN Dimensions (exercise)
CNN Architecture
Blocks of learn a learned representation, then a classifies it. Our CIFAR CNN:
| layer | output shape | weights |
|---|---|---|
| photo | 3 × 32 × 32 | |
| conv, 32 filters of 3 × 3 × 3, padding 1, ReLU | 32 × 32 × 32 | 896 |
| max pool 2 × 2 | 32 × 16 × 16 | 0 |
| conv, 64 filters of 3 × 3 × 32, padding 1, ReLU | 64 × 16 × 16 | 18,496 |
| max pool 2 × 2 | 64 × 8 × 8 | 0 |
| flatten | 4,096 | 0 |
| fully connected, ReLU | 512 | 2,097,664 |
| fully connected, ReLU | 256 | 131,328 |
| fully connected, softmax | 10 | 2,570 |
| total | 2,250,954 |
CNNs in PyTorch
The convolutional block: two rounds of .
Then the fully connected block, which :
torch.Size([5, 10]) 2250954
Where the Weights Are
| network | weights | unseen photos right |
|---|---|---|
| feed-forward | 1,707,274 | 49.5% |
| CNN | 2,250,954 | 71.1% |
The CNN has weights than the feed-forward network, and 2,231,562 of them (99%) are in its fully connected layers. Its two convolutional layers have only 19,392.
So it isn't winning by size, it's winning by .
Training a CNN
Softmax, cross-entropy and backprop, same as before. The one new piece: each filter weight was used , so its gradient :
Which is itself a cross-correlation, of the image with the feature map's gradient.
What Our CNN Learned
Trained on 50,000 photos it gets 71.1% of unseen photos right. Nobody told its filters what to look for, but most of them .
Learned Filters (MNIST)
CNNs and the Visual Cortex
Trained on 1.2 million photos, AlexNet's 96 first-layer filters came out looking a lot like V1's simple cells: edges at every , plus a few blobs of color (Krizhevsky, Sutskever and Hinton, 2012). Nobody designed them.
Further up, a CNN trained only to recognize objects had middle layers that predicted neurons in V4 and top layers that predicted neurons in IT, the brain's object area (Yamins et al., 2014).
Built to copy the visual cortex, then trained to do its job, CNNs ended up a lot like it.
Why the CNN Wins
- Each filter looks at , because neighboring pixels belong together.
- The same filter is used , because an edge is an edge wherever it is.
- Layers build .
All three depend on where each pixel is, which is exactly what flattening threw away. Shuffle the pixels and the CNN falls from 71.1% to 50.4%, .
Efficiency in CNN Implementations (homework)
These nested loops are really really really slow! It took me ~9 hours to run my CNN this way.
If we're particularly clever, we can collapse it all into a single matrix multiplication!
(this took me from ~9 hours to ~5 minutes)
Python CNN by Hand - Convolutional Layer
Same three methods as last lecture. Each filter is , holding its c_in * k * k weights.
Python CNN by Hand - Convolutional Layer, Forward
The trick is im2col: cut every patch out as , and the whole layer is .
Python CNN by Hand - Convolutional Layer, Backward
F.fold is F.unfold in reverse: it adds every patch's gradient back , summing where patches overlap.
Python CNN by Hand - Max Pooling, Forward
Python CNN by Hand - Max Pooling, Backward
Only the pixel that won the max , every other pixel in its window gets 0.
Python CNN by Hand - Linear Layer
The same LinearLayer as last lecture:
Python CNN by Hand - ReLU and Flatten
Python CNN by Hand - Network
The same Network as last lecture, it doesn't care .
Training a CNN by Hand - The Digits
Training a CNN by Hand
unseen digits right: tensor(0.9778)
That's the browser's run. Desktop PyTorch starts from different random weights and gets 98.7%. A feed-forward network on the same digits gets 97.3%.
Training CNNs (homework bonus)
We could ask the same four questions I asked at the start of class for a CNN too!
Training CNNs: A and B
Training CNNs: C and D
Computer Vision
Prior to 2020 and the paper "An image is worth 16x16 words" basically all of computer vision was done using:
CNNs are still widely used today and were partly an inspiration for transformers
Vision Tasks
There are a ton of new tasks in computer vision we haven't seen so far!
These include: Image Classification, Object Detection, Semantic Segmentation, Instance Segmentation, Panoptic Segmentation, Keypoint Detection, Optical Character Recognition (OCR), Image Captioning, Visual Question Answering (VQA), Image-to-Image Translation, Super-Resolution, Image Restoration, 3D Reconstruction, Depth Estimation
Next: Sequences
When our CNN labels a photo, it doesn't matter what photo came before it or comes after. Shuffle the photos and every one gets the same label. Shuffle the words of a sentence and .