CSE 30124 · Introduction to Artificial Intelligence William Theisen · University of Notre Dame
Feed-Forward Networks01
Predicting Game results
In Model Evaluation we asked "Did we win?" and a logistic regression answered with a straight line through the two teams' yards, getting 83.7% of unseen games right.
Feed-Forward Networks02
Review: Perceptrons
Really, a logistic regression is the same thing as a perceptron, just with a activation function. Generally, we can call this unit an
Deep learning is all about how we can use
Feed-Forward Networks03
Artificial Neurons
(X): The information the neuron is receiving
(ω): defines how important a given input is for the neuron
(b or ω0): A small adjustment to the weighted sum of the inputs
Think of this as shifting the neurons threshold to make it harder or easier to activate
Summation (Σ): The weighted sum of the inputs and bias
represents the neurons "total input", also known as the pre-activation
The summation transforms multiple inputs into one number that summarizes the information that the neuron cares about.
(ϕ): A function applied to the pre-activation of the neuron
(ŷ): The final resulting value from the activation function
Feed-Forward Networks · Neurons04
Real Neurons
The artificial neuron is a (very) loose copy of a real one. A real neuron collects signals from other neurons on its dendrites, each synapse makes a signal count for more or less, and the neuron fires a spike down its only when the total crosses a threshold.
Our brains have about 86 billion neurons and something like 100 trillion synapses. Our football network has 3 neurons.
Feed-Forward Networks · Neurons05
The XOR Problem
What if instead of asking win or loss, we wanted to decide whether a game was close or not (within 8 points)
Asking if a game was close or not results in data that is very similar to the XOR problem we saw with perceptrons, where our data is not
Feed-Forward Networks · Neurons06
The XOR Problem
A neuron is capable of drawing a single line, but our data requires two lines to separate. We can have two neurons each create one of the two lines and use a to combine their output!
Stacking neurons like this, so some take the outputs of others as their inputs, gives a multi-layer perceptron (MLP), or a feed-forwardneural network (FFN)
Feed-Forward Networks · Neurons07
Neural Networks: Diagram
Combining our three artificial neurons gives us a simple FFN The organization of neurons in a neural network is called its.
Feed-Forward Networks · Neurons08
The Cortical Column
In 1957 Vernon Mountcastle found that neurons stacked straight down through a cat's cortex all responded to the same patch of skin. The cortex is built out of these , each a fraction of a millimeter wide.
Douglas and Martin (1989) proposed that columns all over the cortex run the , a bit like a network built by stacking the same kind of layer.
Feed-Forward Networks · Neurons09
Depth in the Brain
Our network has an input layer, an output layer and one layer of neurons in between. Vision in the brain has the same shape, only : a chain of areas, each finding more complex patterns than the one before.
Felleman and Van Essen (1991) counted 32 visual areas in the monkey brain, arranged in about 10 levels.
Producing output with a neural network is called "performing the "
Step 1 of 52014 North Carolina, 50–43
game
score
h1
h2
ŷ
close?
2014 North Carolina
50–43
0
0
0.88
yes
2015 Clemson
22–24
0
0
0.88
yes
2012 Wake Forest
38–0
1
0
0.12
no
2012 Alabama (title game)
14–42
0
1
0.12
no
Feed-Forward Networks · Neurons12
Decision Regions
Our first two neurons each draw a , our third neuron combines the two boundaries to create a decision region
On all 141 games it gets every one right (100%), while no straight line could do better than 88.7%.
Feed-Forward Networks · Neurons13
Hidden Layer Features
Layers that don't produce the final output are called.
These are what transform our data into points! They create new features for our data that our third neuron can separate.
Feed-Forward Networks · Neurons14
Hidden Features in the Brain
Our hidden layer turned each game into new features, won by more than 8 and lost by more than 8. Do the hidden layers of a bigger network find features like the brain's? Yamins et al. (2014) trained a network to recognize objects in photos, then compared its hidden layers with neurons recorded in monkeys looking at the same photos.
Its middle hidden layers matched , the area that responds to curves and shapes
Its last hidden layer matched , the area that responds to whole objects and faces
Networks that were better at recognizing objects matched the brain better
The networks were never shown any brain data. Learning to do the same job, they ended up with hidden features a lot like the brain's.
Feed-Forward Networks · Neurons15
Condensing the Hidden Layer
We can compactly represent the operations of the hidden layer as a single matrix multiplication.
similar to sklearn for machine learning, PyTorch is the library we will use to implement all of our neural networks.
The @ symbol is how we multiply matrices and .T gives us the transpose of a matrix
Feed-Forward Networks · Neurons18
A Forward Pass in the Brain
Thorpe, Fize and Marlot (1996) flashed photos for just 20 ms and asked people "is there an animal in it?"
Their brain signals for "animal" and "no animal" had already split apart about 150 ms after the photo appeared.
Neurons are slow (milliseconds per step), so 150 ms is only enough time for the signal to go up the visual hierarchy about . A first look at a scene seems to work much like a : no loops, no second guesses.
Feed-Forward Networks · Neurons19
Activation Functions
Critically, activation functions introduce into the model. This is what allows neural networks to learn complex patterns (beyond just linear relationships).
Z_2 = \bp{X \Omega_1\T + b_1} \Omega_2\T + b_2\phantom{Z_2} = X \bp{\Omega_2 \Omega_1}\T + \bp{b_1 \Omega_2\T + b_2}\phantom{Z_2} = X \Omega\T + b
If we didn't have activation functions, our neural network would be no more powerful than just a single neuron. However, this power comes at the cost of .
For our simple neural network, we are using the sigmoid function as the activation function, but it's not the only choice.
Other common options are the hyperbolic tangent and the ReLU function
Feed-Forward Networks · Activation Functions20
Activation Functions in Real Neurons
A real neuron stays silent below its threshold, then fires faster the harder it's pushed, until it levels off.
Near the threshold that's a lot closer to a than to a sigmoid. Glorot et al. (2011) used this to argue for ReLU, along with the fact that only about 1–4% of the brain's neurons are active at once (sparse, like a ReLU layer's zeros).
Feed-Forward Networks · Activation Functions21
Universal Approximation Theorem
The universal approximation theorem says a neural network with a single hidden layer, given enough hidden neurons, can approximate any continuous function
Feed-Forward Networks · Activation Functions22
One Neuron, One Network?
Our artificial neuron adds up its inputs once. A real neuron's dendrites branch, and each branch does its own non-linear sum before the cell body adds them up.
Poirazi, Brannon and Mel (2003) modeled a pyramidal neuron this way and found it behaves like a neural network, every branch a hidden neuron.
Beniaguev, Segev and London (2021) trained networks to copy a detailed model of one cortical neuron, spike for spike. It took a network 7 layers deep.
So one real neuron is already a small network, and our 3-neuron FFN is a long way from even one of them.
Feed-Forward Networks · Activation Functions23
Multiclass Classification
Another powerful new ability we gain with neural networks, is we are no longer limited to binary outputs, we can use them naturally for multi-class classification
Feed-Forward Networks · Activation Functions24
Multi-class Classification
Our three output scores aren't probabilities. To turn them into probabilities that add up to 1, we use the function:
Softmax is the usual activation function for the of a classifier. With only two classes it's the same as the .
Feed-Forward Networks · Activation Functions25
Representing Neural Networks
Another common way to represent the architecture of a neural network is as a .
From the computation graph, we can also represent our neural network as a single large .
This is critical for training our neural networks (next class)
Feed-Forward Networks · Activation Functions26
What FFNs Leave Out
: most areas are connected both ways, and only about 5–10% of the input to the LGN comes from the eye (most of the rest comes back down from the cortex)
Spikes: real neurons send all-or-nothing spikes, and their timing matters, not just one number
Learning: nobody has found the brain doing what we'll do next lecture (backprop)
An FFN is a model of the brain's first, fast sweep, not the whole thing.