Unit 02 · Chapter 1

Feed-Forward Networks

Feed-Forward Networks01

Predicting Game results

In Model Evaluation we asked "Did we win?" and a logistic regression answered with a straight line through the two teams' yards, getting 83.7% of unseen games right.

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 200 400 600 Notre Dame yards 200 400 600 opponent yards logistic regression p = 0.1 p = 0.9 p = 0.5 win loss
Feed-Forward Networks02

Review: Perceptrons

Really, a logistic regression is the same thing as a perceptron, just with a   activation function. Generally, we can call this unit an  

x₁ ω₁ x₂ ω₂ x₃ ω₃ x₄ ω₄ Σ φ inputs weights bias b summation activation z pre-activation ŷ output

Deep learning is all about how we can use  

Feed-Forward Networks03

Artificial Neurons

The summation transforms multiple inputs into one number that summarizes the information that the neuron cares about.

Feed-Forward Networks · Neurons04

Real Neurons

spike dendrites synapses cell body threshold axon x ω z φ ŷ

The artificial neuron is a (very) loose copy of a real one. A real neuron collects signals from other neurons on its dendrites, each synapse makes a signal count for more or less, and the neuron fires a spike down its   only when the total crosses a threshold.

Our brains have about 86 billion neurons and something like 100 trillion synapses. Our football network has 3 neurons.

Feed-Forward Networks · Neurons05

The XOR Problem

What if instead of asking win or loss, we wanted to decide whether a game was close or not (within 8 points)

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0 10 20 30 40 50 60 70 Notre Dame points (x₁) 0 10 20 30 40 50 opponent points (x₂) dashed: logistic regression, 67.4% right 0 10 20 30 40 50 60 70 Notre Dame points (x₁) 0 10 20 30 40 50 opponent points (x₂) h₁: won by 9+ h₂: lost by 9+ neither: one-score game

Asking if a game was close or not results in data that is very similar to the XOR problem we saw with perceptrons, where our data is not  

Feed-Forward Networks · Neurons06

The XOR Problem

A neuron is capable of drawing a single line, but our data requires two lines to separate. We can have two neurons each create one of the two lines and use a   to combine their output!

x₁ x₂ ω₁ ω₂ Σ σ h₁ ω₃ ω₄ Σ σ h₂ ω₅ ω₆ Σ σ ŷ b₁ z b₂ z b₃ z input layer hidden layer: 2 neurons output layer: 1 neuron ND points opp. points P(close)

Stacking neurons like this, so some take the outputs of others as their inputs, gives a multi-layer perceptron (MLP), or a feed-forward neural network (FFN)

Feed-Forward Networks · Neurons07

Neural Networks: Diagram

input layer the features, no neurons hidden layer 2 neurons output layer 1 neuron Σ σ b₁ + Σ σ b₂ + Σ σ b₃ + ×ω₁ ×ω₂ ×ω₃ ×ω₄ ×ω₅ ×ω₆ x₁ x₂ z₁ h₁ z₂ h₂ z₃ ŷ ND points opp. points P(close) data flows forward, left to right

Combining our three artificial neurons gives us a simple FFN
The organization of neurons in a neural network is called its  .

Feed-Forward Networks · Neurons08

The Cortical Column

In 1957 Vernon Mountcastle found that neurons stacked straight down through a cat's cortex all responded to the same patch of skin. The cortex is built out of these  , each a fraction of a millimeter wide.

I II/III IV V VI a cortical column (a fraction of a mm wide) input from the thalamus to the next area out of the cortex as a network x one layer weighted sums h

Douglas and Martin (1989) proposed that columns all over the cortex run the  , a bit like a network built by stacking the same kind of layer.

Feed-Forward Networks · Neurons09

Depth in the Brain

Our network has an input layer, an output layer and one layer of neurons in between. Vision in the brain has the same shape, only  : a chain of areas, each finding more complex patterns than the one before.

input layer hidden layers output layer eye retina spots of light V1 visual area 1 edges at an angle V2 visual area 2 contours, textures V4 visual area 4 curves, shapes IT inferior temporal objects, faces PFC prefrontal cortex decides what it is brain network x h₁ h₂ h₃ h₄ ŷ

Felleman and Van Essen (1991) counted 32 visual areas in the monkey brain, arranged in about 10 levels.

Feed-Forward Networks · Neurons10

Network Diagram with Weights

input layer the features, no neurons hidden layer 2 neurons output layer 1 neuron Σ σ −85 + Σ σ −85 + Σ σ 2 + ×10 ×−10 ×−10 ×10 ×−4 ×−4 x₁ x₂ z₁ h₁ z₂ h₂ z₃ ŷ ND points opp. points P(close) data flows forward, left to right h_1 = \sigma(\wnum{10}x_1 - \wnum{10}x_2 - \bnum{85}) \qquad h_2 = \sigma(\wnum{10}x_2 - \wnum{10}x_1 - \bnum{85}) \yhat = \sigma(\bnum{2} - \wnum{4} h_1 - \wnum{4} h_2) h_1 = \sigma(\wnum{10}x_1 - \wnum{10}x_2 - \bnum{85}) \qquad h_2 = \sigma(\wnum{10}x_2 - \wnum{10}x_1 - \bnum{85}); \yhat = \sigma(\bnum{2} - \wnum{4} h_1 - \wnum{4} h_2)
Feed-Forward Networks · Neurons11

Producing Output from a Neural Network

Producing output with a neural network is called "performing the  "

Step 1 of 52014 North Carolina, 50–43
The network with North Carolina 50–43 fed in: z₁ = −15 and z₂ = −155, so h₁ = h₂ = 0, z₃ = 2 and ŷ = 0.88, close.
gamescoreh1h2ŷclose?
2014 North Carolina50–43000.88yes
2015 Clemson22–24000.88yes
2012 Wake Forest38–0100.12no
2012 Alabama (title game)14–42010.12no
Feed-Forward Networks · Neurons12

Decision Regions

Our first two neurons each draw a  , our third neuron combines the two boundaries to create a decision region

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ x₁ −10 x₂ 10 −85 h₂ h₂ = 1 above this line: lost by more than 8 0 10 20 30 40 50 60 70 Notre Dame points (x₁) 0 10 20 30 40 50 opponent points (x₂) h₁ h₂ 2014 North Carolina 2015 Clemson 2012 Wake Forest 2012 Alabama x₁ 10 x₂ −10 −85 h₁ h₁ = 1 below this line: won by more than 8 ŷ = 0.88 between the lines ("close"), 0.12 outside them

On all 141 games it gets every one right (100%), while no straight line could do better than 88.7%.

Feed-Forward Networks · Neurons13

Hidden Layer Features

Layers that don't produce the final output are called .

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0 20 40 60 Notre Dame points (x₁) 0 10 20 30 40 50 opponent points (x₂) the games as points 0 1 h₁ (won by more than 8) 0 1 h₂ (lost by more than 8) the same games as hidden features 55 close 70 16 hidden layer

These are what transform our data into   points! They create new features for our data that our third neuron can separate.

Feed-Forward Networks · Neurons14

Hidden Features in the Brain

Our hidden layer turned each game into new features, won by more than 8 and lost by more than 8. Do the hidden layers of a bigger network find features like the brain's? Yamins et al. (2014) trained a network to recognize objects in photos, then compared its hidden layers with neurons recorded in monkeys looking at the same photos.

The networks were never shown any brain data. Learning to do the same job, they ended up with hidden features a lot like the brain's.

Feed-Forward Networks · Neurons15

Condensing the Hidden Layer

input layer the features, no neurons hidden layer 2 neurons output layer 1 neuron Σ σ −85 + Σ σ −85 + Σ σ 2 + ×10 ×−10 ×−10 ×10 ×−4 ×−4 x₁ x₂ z₁ h₁ z₂ h₂ z₃ ŷ ND points opp. points P(close) data flows forward, left to right

We can compactly represent the operations of the hidden layer as a single matrix multiplication.

X inputs, one row per game <4, 2> x₁ x₂ 2014 North Carolina 50 43 2015 Clemson 22 24 2012 Wake Forest 38 0 2012 Alabama 14 42 Ω₁ weights, one row per neuron <2, 2> x₁ x₂ neuron 1 10 −10 neuron 2 −10 10 −85 −85 b₁ biases <2>
Feed-Forward Networks · Neurons16

Weights as Matrices

Z_1 = X \Omega_1\T + b_1 = \Bb{\red{\begin{matrix} 50 & 43 \\ 22 & 24 \\ 38 & 0 \\ 14 & 42 \end{matrix}}} \Bb{\blue{\begin{matrix} 10 & -10 \\ -10 & 10 \end{matrix}}} + \Bb{\begin{matrix} \bnum{-85} & \bnum{-85} \end{matrix}} \phantom{Z_1} = \Bb{\begin{matrix} 70 & -70 \\ -20 & 20 \\ 380 & -380 \\ -280 & 280 \end{matrix}} + \Bb{\begin{matrix} \bnum{-85} & \bnum{-85} \end{matrix}} = \Bb{\purple{\begin{matrix} -15 & -155 \\ -105 & -65 \\ 295 & -465 \\ -365 & 195 \end{matrix}}} A_1 = \sigma(Z_1) = \Bb{\gold{\begin{matrix} 0 & 0 \\ 0 & 0 \\ 1 & 0 \\ 0 & 1 \end{matrix}}} Z_1 = X \Omega_1\T + b_1 = \Bb{\red{\begin{matrix} 50 & 43 \\ 22 & 24 \\ 38 & 0 \\ 14 & 42 \end{matrix}}} \Bb{\blue{\begin{matrix} 10 & -10 \\ -10 & 10 \end{matrix}}} + \Bb{\begin{matrix} \bnum{-85} & \bnum{-85} \end{matrix}}; \phantom{Z_1} = \Bb{\begin{matrix} 70 & -70 \\ -20 & 20 \\ 380 & -380 \\ -280 & 280 \end{matrix}} + \Bb{\begin{matrix} \bnum{-85} & \bnum{-85} \end{matrix}} = \Bb{\purple{\begin{matrix} -15 & -155 \\ -105 & -65 \\ 295 & -465 \\ -365 & 195 \end{matrix}}}; A_1 = \sigma(Z_1) = \Bb{\gold{\begin{matrix} 0 & 0 \\ 0 & 0 \\ 1 & 0 \\ 0 & 1 \end{matrix}}}

NOTE: This example is a little goofy because the result of the transpose is the same matrix

Feed-Forward Networks · Neurons17

Forward Pass of a Neural Network

import torch
X = torch.tensor([[50., 43.], [22., 24.], [38., 0.], [14., 42.]])
W1 = torch.tensor([[10., -10.], [-10., 10.]])
b1 = torch.tensor([-85., -85.])
A1 = torch.sigmoid(X @ W1.T + b1) # b1 is added to every row
print(X.shape, W1.shape, b1.shape)
print(A1.round(decimals=3))
Output
torch.Size([4, 2]) torch.Size([2, 2]) torch.Size([2])
tensor([[0., 0.],
        [0., 0.],
        [1., 0.],
        [0., 1.]])

similar to sklearn for machine learning, PyTorch is the library we will use to implement all of our neural networks.

The @ symbol is how we multiply matrices and .T gives us the transpose of a matrix

Feed-Forward Networks · Neurons18

A Forward Pass in the Brain

Thorpe, Fize and Marlot (1996) flashed photos for just 20 ms and asked people "is there an animal in it?"

Their brain signals for "animal" and "no animal" had already split apart about 150 ms after the photo appeared.

Neurons are slow (milliseconds per step), so 150 ms is only enough time for the signal to go up the visual hierarchy about  . A first look at a scene seems to work much like a  : no loops, no second guesses.

Feed-Forward Networks · Neurons19

Activation Functions

Critically, activation functions introduce   into the model. This is what allows neural networks to learn complex patterns (beyond just linear relationships).

Z_2 = \bp{X \Omega_1\T + b_1} \Omega_2\T + b_2 \phantom{Z_2} = X \bp{\Omega_2 \Omega_1}\T + \bp{b_1 \Omega_2\T + b_2} \phantom{Z_2} = X \Omega\T + b Z_2 = \bp{X \Omega_1\T + b_1} \Omega_2\T + b_2; \phantom{Z_2} = X \bp{\Omega_2 \Omega_1}\T + \bp{b_1 \Omega_2\T + b_2}; \phantom{Z_2} = X \Omega\T + b

If we didn't have activation functions, our neural network would be no more powerful than just a single neuron. However, this power comes at the cost of  .

For our simple neural network, we are using the sigmoid function as the activation function, but it's not the only choice.

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ −5 0 5 z 0.00 0.25 0.50 0.75 1.00 sigmoid −5 0 5 z −1.0 −0.5 0.0 0.5 1.0 tanh −5 0 5 z 0 2 4 6 8 ReLU

Other common options are the hyperbolic tangent and the ReLU function

Feed-Forward Networks · Activation Functions20

Activation Functions in Real Neurons

A real neuron stays silent below its threshold, then fires faster the harder it's pushed, until it levels off.

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0 2 4 6 8 10 input current (× threshold) 0 50 100 150 200 250 300 spikes per second silent below threshold levels off (a neuron needs 2 ms to recover after a spike) ReLU real neuron (model)

Near the threshold that's a lot closer to a   than to a sigmoid. Glorot et al. (2011) used this to argue for ReLU, along with the fact that only about 1–4% of the brain's neurons are active at once (sparse, like a ReLU layer's zeros).

Feed-Forward Networks · Activation Functions21

Universal Approximation Theorem

AAAWgmp1bWIAAAAeanVtZGMycGEAEQAQgAAAqgA4m3EDYzJwYQAAABZcanVtYgAAAEdqdW1kYzJtYQARABCAAACqADibcQN1cm46YzJwYToyZmUyNmU5YS0wYjM4LTQ5YTItODAwNC1mODI2NjU4MTdiMDgAAAADl2p1bWIAAAApanVtZGMyYXMAEQAQgAAAqgA4m3EDYzJwYS5hc3NlcnRpb25zAAAAALxqdW1iAAAARGp1bWRjYm9yABEAEIAAAKoAOJtxE2MycGEuaW5ncmVkaWVudC52MwAAAAAYYzJzaPCwevKbCxCFYDpxjwV2MHIAAABwY2JvcqNpZGM6Zm9ybWF0bWltYWdlL3N2Zyt4bWxqaW5zdGFuY2VJRHgseG1wOmlpZDpiNDJjMDc4OS0zZmM3LTRlZTUtYjE5NS1iZDFjMDZiZWI1NWVscmVsYXRpb25zaGlwaHBhcmVudE9mAAAB4mp1bWIAAABBanVtZGNib3IAEQAQgAAAqgA4m3ETYzJwYS5hY3Rpb25zLnYyAAAAABhjMnNognxtiw41lMQM45Sfl1eELQAAAZljYm9yomdhY3Rpb25zgqJmYWN0aW9ua2MycGEub3BlbmVkanBhcmFtZXRlcnOha2luZ3JlZGllbnRzgaJjdXJseC1zZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmluZ3JlZGllbnQudjNkaGFzaFggIom/+6SNh8nzYnrytohXeQ97WZR1OfzJgiJJWhg3HrCkZmFjdGlvbngdY29tLmFudGhyb3BpYy5jbGF1ZGUucHJvdmlkZWRqcGFyYW1ldGVyc6F4H2NvbS5hbnRocm9waWMub3JpZ2luLWNvbmZpZGVuY2VndW5rbm93bmtkZXNjcmlwdGlvbnhmQ2xhdWRlIHByb3ZpZGVkIHRoaXMgZmlsZSBhdCB0aGUgcmVxdWVzdCBvZiBhIHVzZXIgYW5kIG1heSBoYXZlIGNyZWF0ZWQgb3IgbW9kaWZpZWQgdGhlIGZpbGUgY29udGVudHMubXNvZnR3YXJlQWdlbnShZG5hbWVmQ2xhdWRlcmFsbEFjdGlvbnNJbmNsdWRlZPUAAADIanVtYgAAAEBqdW1kY2JvcgARABCAAACqADibcRNjMnBhLmhhc2guZGF0YQAAAAAYYzJzaDxbE3nTm/sycJ09BzSf1dUAAACAY2JvcqVjYWxnZnNoYTI1NmNwYWRMAAAAAAAAAAAAAAAAZGhhc2hYIPGy3R4ZXOprj3ox0aHNfjBgX6oWarbKXZ5oySOXAvsYZG5hbWVuanVtYmYgbWFuaWZlc3RqZXhjbHVzaW9uc4GiZXN0YXJ0GQFxZmxlbmd0aBkeBAAAAj5qdW1iAAAAJ2p1bWRjMmNsABEAEIAAAKoAOJtxA2MycGEuY2xhaW0udjIAAAACD2Nib3KlY2FsZ2ZzaGEyNTZpc2lnbmF0dXJleE1zZWxmI2p1bWJmPS9jMnBhL3VybjpjMnBhOjJmZTI2ZTlhLTBiMzgtNDlhMi04MDA0LWY4MjY2NTgxN2IwOC9jMnBhLnNpZ25hdHVyZWppbnN0YW5jZUlEeCx4bXA6aWlkOmZhYmQ2ZWQ5LTAzNWItNGJjZC04MTQ1LWU1NDFjY2EyYzcxMXJjcmVhdGVkX2Fzc2VydGlvbnODomN1cmx4LXNlbGYjanVtYmY9YzJwYS5hc3NlcnRpb25zL2MycGEuaW5ncmVkaWVudC52M2RoYXNoWCAiib/7pI2HyfNievK2iFd5D3tZlHU5/MmCIklaGDcesKJjdXJseCpzZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmFjdGlvbnMudjJkaGFzaFgge+6uSgRWa3bmBizPdS2Ddwga+CIm8p2IkpuX6oMYiT+iY3VybHgpc2VsZiNqdW1iZj1jMnBhLmFzc2VydGlvbnMvYzJwYS5oYXNoLmRhdGFkaGFzaFggEQjirUJH6zKIgCeLOMcJDmp+8xdJy7HaOIDs5rfTEtl0Y2xhaW1fZ2VuZXJhdG9yX2luZm+jZG5hbWVvQW50aHJvcGljIEZpbGVzZ3ZlcnNpb25lMS4wLjBrc3BlY1ZlcnNpb25lMi40LjAAABA4anVtYgAAAChqdW1kYzJjcwARABCAAACqADibcQNjMnBhLnNpZ25hdHVyZQAAABAIY2JvctKEWQISogEmGCFZAgowggIGMIIBjaADAgECAhRA5aAK7sI50L64g/oGQgU9Z1UTADAKBggqhkjOPQQDAzBJMRcwFQYDVQQKEw5BbnRocm9waWMsIFBCQzEuMCwGA1UEAxMlQW50aHJvcGljIENvbnRlbnQgQ3JlZGVudGlhbHMgUm9vdCBDQTAeFw0yNjA4MDcxODQzNTZaFw0yODA4MDYxOTQzNTZaMEQxFzAVBgNVBAoTDkFudGhyb3BpYywgUEJDMSkwJwYDVQQDEyBBbnRocm9waWMgQ2xhdWRlIENvbnRlbnQgU2lnbmluZzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJh6CmvLUBgFFNU0vUKlOVtE6djd17L5SuwX0LemFisBM3dkd/3cyjxFA3Qo5S46fX0/ihY0VZ7mfb9KF703t5OjWDBWMA4GA1UdDwEB/wQEAwIHgDAVBgNVHSUEDjAMBgorBgEEAYPoXgIBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUzlHiBIFOZFsj+OPEz5o+nMHXXMIwCgYIKoZIzj0EAwMDZwAwZAIwMXMdFJ4BetLLVY7ORuE9noqbbAZOZn/aArXyTwFAZfKrPzxF2vPoJNf1+UCdg1XGAjBwX1zd9WGqYkqmL5SFqw1QySjr1zJfpJM9+1rdDwSPLMOPOjKuiXjoU/pUUeG9RwmhY3BhZFkNngAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPZYQMf3SgC5ikFtfaq21kX5PDNnTS9GZenPgtSFu7EC0msgK2whzBpnh88H3X5gO//qi4lnw68rHMdCSDp/xMOrdAU= image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 30 40 polynomial, degree 1 polynomial, degree 3 polynomial, degree 10 (exact) 30 40 ND points per game tree, depth 1 tree, depth 3 tree, depth 6 (exact) 2012 2017 2022 30 40 network, 2 neurons 2012 2017 2022 season network, 4 neurons 2012 2017 2022 network, 10 neurons (exact)

The universal approximation theorem says a neural network with a single hidden layer, given enough hidden neurons, can approximate any continuous function

Feed-Forward Networks · Activation Functions22

One Neuron, One Network?

Our artificial neuron adds up its inputs once. A real neuron's dendrites branch, and each branch does its own non-linear sum before the cell body adds them up.

Poirazi, Brannon and Mel (2003) modeled a pyramidal neuron this way and found it behaves like a   neural network, every branch a hidden neuron.

Beniaguev, Segev and London (2021) trained networks to copy a detailed model of one cortical neuron, spike for spike. It took a network 7 layers deep.

So one real neuron is already a small network, and our 3-neuron FFN is a long way from even one of them.

Feed-Forward Networks · Activation Functions23

Multiclass Classification

Another powerful new ability we gain with neural networks, is we are no longer limited to binary outputs, we can use them naturally for multi-class classification

x₁ x₂ h₁ h₂ z₁ z₂ z₃ 2h₁ 2 − 2h₁ − 2h₂ 2h₂ 2014 North Carolina, 50 to 43 0 2 0 soft max 0.11 won by 9+ 0.79 one-score 0.11 lost by 9+
Feed-Forward Networks · Activation Functions24

Multi-class Classification

Our three output scores aren't probabilities. To turn them into probabilities that add up to 1, we use the   function:

\text{softmax}(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}} \text{softmax}(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}

Softmax is the usual activation function for the   of a classifier. With only two classes it's the same as the  .

Feed-Forward Networks · Activation Functions25

Representing Neural Networks

Another common way to represent the architecture of a neural network is as a  .

X <141, 2> Z₁ = X Ω₁ᵀ + b₁ <141, 2> Ω₁ <2, 2> b₁ <1, 2> A₁ = σ( Z₁ ) <141, 2> Z₂ = A₁ Ω₂ᵀ + b₂ <141, 1> Ω₂ <1, 2> b₂ <1, 1> ŷ = σ( Z₂ ) <141, 1> X <141, 2> Z₁ = X Ω₁ᵀ + b₁ <141, 2> Ω₁ <2, 2> b₁ <1, 2> A₁ = σ( Z₁ ) <141, 2> Z₂ = A₁ Ω₂ᵀ + b₂ <141, 1> Ω₂ <1, 2> b₂ <1, 1> ŷ = σ( Z₂ ) <141, 1> ℒ = BCE( ŷ , y ) <1> y <141, 1>

From the computation graph, we can also represent our neural network as a single large  .

X <141, 2> Z₁ = X Ω₁ᵀ + b₁ <141, 2> Ω₁ <2, 2> b₁ <1, 2> A₁ = σ( Z₁ ) <141, 2> Z₂ = A₁ Ω₂ᵀ + b₂ <141, 1> Ω₂ <1, 2> b₂ <1, 1> ŷ = σ( Z₂ ) <141, 1> ℒ = BCE( ŷ , y ) <1> y <141, 1>

This is critical for training our neural networks (next class)

Feed-Forward Networks · Activation Functions26

What FFNs Leave Out

eye spots of light LGN spots of light V1 edges at an angle V2 contours, textures V4 curves, shapes IT objects, faces circles: the patch of the scene each area responds to (not to scale) dashed: feedback (most areas are connected both ways) a deep network x h₁ h₂ h₃ h₄ ŷ early layers: small, simple features → later layers: big, complex ones

An FFN is a model of the brain's first, fast sweep, not the whole thing.

Feed-Forward Networks · Activation Functions27

An Old Exam Question

import torch
import torch.nn as nn
import torch.nn.functional as F
class SimpleNetwork(nn.Module):
def __init__(self):
super(SimpleNetwork, self).__init__()
self.fc1 = nn.Linear(2, 32)
self.fc2 = nn.Linear(32, 16)
self.fc3 = nn.Linear(16, 1)
def forward(self, x):
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = torch.sigmoid(self.fc3(x))
return x
criterion = nn.BCELoss()
Feed-Forward Networks · Activation Functions28