ℒ
Unit 03 · Chapter 1

Backpropagation

Backpropagation01

The XOR Problem

What if instead of asking win or loss, we wanted to decide whether a game was close or not (within 8 points)

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0 10 20 30 40 50 60 70 Notre Dame points (x₁) 0 10 20 30 40 50 opponent points (x₂) dashed: logistic regression, 67.4% right 0 10 20 30 40 50 60 70 Notre Dame points (x₁) 0 10 20 30 40 50 opponent points (x₂) h₁: won by 9+ h₂: lost by 9+ neither: one-score game

Asking if a game was close or not results in data that is very similar to the XOR problem we saw with perceptrons, where our data is not  

Backpropagation02

Neural Networks: Diagram

input layer the features, no neurons hidden layer 2 neurons output layer 1 neuron Σ σ b₁ + Σ σ b₂ + Σ σ b₃ + ×ω₁ ×ω₂ ×ω₃ ×ω₄ ×ω₅ ×ω₆ x₁ x₂ z₁ h₁ z₂ h₂ z₃ ŷ ND points opp. points P(close) data flows forward, left to right

Combining our three artificial neurons gives us a simple FFN
The organization of neurons in a neural network is called its  .

Backpropagation03

Feed-Forward Networks - Old Exam Question

import torch
import torch.nn as nn
torch.set_printoptions(precision=2)
class SimpleNN(nn.Module):
def __init__(self, input_size, hidden_size, num_classes):
super(SimpleNN, self).__init__()
self.fc1 = nn.Linear(input_size, hidden_size)
self.sig = nn.Sigmoid()
self.fc2 = nn.Linear(hidden_size, num_classes)
def forward(self, x):
z1 = self.fc1(x)
a1 = self.sig(z1)
z2 = self.fc2(a1)
return z2
Backpropagation · Review04

Last Class: Feed-Forward Networks

# Last lecture's 4 games: ND points, opp. points
X = torch.tensor([[50., 43.], [22., 24.], [38., 0.], [14., 42.]]) # ND Score, Opp. Score
y = torch.tensor([[1.], [1.], [0.], [0.]]) # close = 1, not close = 0
torch.manual_seed(0)
model = SimpleNN(2, 2, 1) # fresh: random weights
for name, p in model.named_parameters(): print(name, p.detach())
y_hat = torch.sigmoid(model(X)) # P(close)
print("y_hat (predictions):", y_hat.detach().flatten())
print("mapped predictions (y_hat > 0.5):", (y_hat > 0.5).float().flatten())
criterion = nn.BCELoss()
loss = criterion(y_hat, y)
print("loss:", round(loss.item(), 3))
Output
fc1.weight tensor([[ 0.19, -0.33],
        [-0.65, -0.68]])
fc1.bias tensor([0.44, 0.58])
fc2.weight tensor([[0.15, 0.32]])
fc2.bias tensor([0.06])
y_hat (predictions): tensor([0.52, 0.52, 0.55, 0.52])
mapped predictions (y_hat > 0.5): tensor([1., 1., 1., 1.])
loss: 0.713
Backpropagation · Review05

Condensing the Hidden Layer

input layer the features, no neurons hidden layer 2 neurons Σ σ 0.44 + Σ σ 0.58 + ×0.19 ×−0.33 ×−0.65 ×−0.68 x₁ x₂ z₁ h₁ z₂ h₂ ND points opp. points

We can compactly represent the operations of the hidden layer as a single matrix multiplication.

Z1 = Xω1T + b1
A1 = σ(Z1)

X inputs, one row per game <4, 2> x₁ x₂ 2014 North Carolina 50 43 2015 Clemson 22 24 2012 Wake Forest 38 0 2012 Alabama 14 42 Ω₁ weights, one row per neuron <2, 2> x₁ x₂ neuron 1 0.19 −0.33 neuron 2 −0.65 −0.68 Ω₁ᵀ transposed: one column per neuron <2, 2> neuron 1 neuron 2 x₁ 0.19 −0.65 x₂ −0.33 −0.68 b₁ biases, one per neuron <2> 0.44 0.58
Backpropagation · Review06

Weights as Matrices: Symbolic

Z_1 = X \Omega_1\T + b_1 = \Bb{\begin{matrix} x_{11} & x_{12} \\ x_{21} & x_{22} \\ x_{31} & x_{32} \\ x_{41} & x_{42} \end{matrix}} \Bb{\begin{matrix} \omega_{11} & \omega_{21} \\ \omega_{12} & \omega_{22} \end{matrix}} + \Bb{\begin{matrix} b_1 & b_2 \end{matrix}} Z_1 = \Bb{\begin{matrix} z_{11} & z_{12} \\ z_{21} & z_{22} \\ z_{31} & z_{32} \\ z_{41} & z_{42} \end{matrix}} = \Bb{\begin{matrix} x_{11} \omega_{11} + x_{12} \omega_{12} + b_1 & x_{11} \omega_{21} + x_{12} \omega_{22} + b_2 \\ x_{21} \omega_{11} + x_{22} \omega_{12} + b_1 & x_{21} \omega_{21} + x_{22} \omega_{22} + b_2 \\ x_{31} \omega_{11} + x_{32} \omega_{12} + b_1 & x_{31} \omega_{21} + x_{32} \omega_{22} + b_2 \\ x_{41} \omega_{11} + x_{42} \omega_{12} + b_1 & x_{41} \omega_{21} + x_{42} \omega_{22} + b_2 \end{matrix}} A_1 = \sigma(Z_1) = \Bb{\begin{matrix} a_{11} & a_{12} \\ a_{21} & a_{22} \\ a_{31} & a_{32} \\ a_{41} & a_{42} \end{matrix}} = \Bb{\begin{matrix} \sigma(z_{11}) & \sigma(z_{12}) \\ \sigma(z_{21}) & \sigma(z_{22}) \\ \sigma(z_{31}) & \sigma(z_{32}) \\ \sigma(z_{41}) & \sigma(z_{42}) \end{matrix}} Z_1 = X \Omega_1\T + b_1 = \Bb{\begin{matrix} x_{11} & x_{12} \\ x_{21} & x_{22} \\ x_{31} & x_{32} \\ x_{41} & x_{42} \end{matrix}} \Bb{\begin{matrix} \omega_{11} & \omega_{21} \\ \omega_{12} & \omega_{22} \end{matrix}} + \Bb{\begin{matrix} b_1 & b_2 \end{matrix}}; Z_1 = \Bb{\begin{matrix} z_{11} & z_{12} \\ z_{21} & z_{22} \\ z_{31} & z_{32} \\ z_{41} & z_{42} \end{matrix}} = \Bb{\begin{matrix} x_{11} \omega_{11} + x_{12} \omega_{12} + b_1 & x_{11} \omega_{21} + x_{12} \omega_{22} + b_2 \\ x_{21} \omega_{11} + x_{22} \omega_{12} + b_1 & x_{21} \omega_{21} + x_{22} \omega_{22} + b_2 \\ x_{31} \omega_{11} + x_{32} \omega_{12} + b_1 & x_{31} \omega_{21} + x_{32} \omega_{22} + b_2 \\ x_{41} \omega_{11} + x_{42} \omega_{12} + b_1 & x_{41} \omega_{21} + x_{42} \omega_{22} + b_2 \end{matrix}}; A_1 = \sigma(Z_1) = \Bb{\begin{matrix} a_{11} & a_{12} \\ a_{21} & a_{22} \\ a_{31} & a_{32} \\ a_{41} & a_{42} \end{matrix}} = \Bb{\begin{matrix} \sigma(z_{11}) & \sigma(z_{12}) \\ \sigma(z_{21}) & \sigma(z_{22}) \\ \sigma(z_{31}) & \sigma(z_{32}) \\ \sigma(z_{41}) & \sigma(z_{42}) \end{matrix}}
Backpropagation · Review07

Weights as Matrices: Numeric

Z_1 = X \Omega_1\T + b_1 = \Bb{\red{\begin{matrix} 50 & 43 \\ 22 & 24 \\ 38 & 0 \\ 14 & 42 \end{matrix}}} \Bb{\blue{\begin{matrix} 0.19 & -0.65 \\ -0.33 & -0.68 \end{matrix}}} + \Bb{\begin{matrix} \bnum{0.44} & \bnum{0.58} \end{matrix}} Z_1 = \Bb{\purple{\begin{matrix} -4.25 & -61.16 \\ -3.30 & -30.04 \\ 7.66 & -24.12 \\ -10.76 & -37.08 \end{matrix}}} = \Bb{\begin{matrix} \red{50} \cdot \wnum{0.19} + \red{43} \cdot (\wnum{-0.33}) + \bnum{0.44} & \red{50} \cdot (\wnum{-0.65}) + \red{43} \cdot (\wnum{-0.68}) + \bnum{0.58} \\ \red{22} \cdot \wnum{0.19} + \red{24} \cdot (\wnum{-0.33}) + \bnum{0.44} & \red{22} \cdot (\wnum{-0.65}) + \red{24} \cdot (\wnum{-0.68}) + \bnum{0.58} \\ \red{38} \cdot \wnum{0.19} + \red{0} \cdot (\wnum{-0.33}) + \bnum{0.44} & \red{38} \cdot (\wnum{-0.65}) + \red{0} \cdot (\wnum{-0.68}) + \bnum{0.58} \\ \red{14} \cdot \wnum{0.19} + \red{42} \cdot (\wnum{-0.33}) + \bnum{0.44} & \red{14} \cdot (\wnum{-0.65}) + \red{42} \cdot (\wnum{-0.68}) + \bnum{0.58} \end{matrix}} A_1 = \sigma(Z_1) = \Bb{\gold{\begin{matrix} 0.01 & 0.00 \\ 0.04 & 0.00 \\ 1.00 & 0.00 \\ 0.00 & 0.00 \end{matrix}}} = \Bb{\begin{array}{ll} \sigma(\purple{-4.25}) & \sigma(\purple{-61.16}) \\ \sigma(\purple{-3.30}) & \sigma(\purple{-30.04}) \\ \sigma(\phantom{-}\purple{7.66}) & \sigma(\purple{-24.12}) \\ \sigma(\purple{-10.76}) & \sigma(\purple{-37.08}) \end{array}} Z_1 = X \Omega_1\T + b_1 = \Bb{\red{\begin{matrix} 50 & 43 \\ 22 & 24 \\ 38 & 0 \\ 14 & 42 \end{matrix}}} \Bb{\blue{\begin{matrix} 0.19 & -0.65 \\ -0.33 & -0.68 \end{matrix}}} + \Bb{\begin{matrix} \bnum{0.44} & \bnum{0.58} \end{matrix}}; Z_1 = \Bb{\purple{\begin{matrix} -4.25 & -61.16 \\ -3.30 & -30.04 \\ 7.66 & -24.12 \\ -10.76 & -37.08 \end{matrix}}} = \Bb{\begin{matrix} \red{50} \cdot \wnum{0.19} + \red{43} \cdot (\wnum{-0.33}) + \bnum{0.44} & \red{50} \cdot (\wnum{-0.65}) + \red{43} \cdot (\wnum{-0.68}) + \bnum{0.58} \\ \red{22} \cdot \wnum{0.19} + \red{24} \cdot (\wnum{-0.33}) + \bnum{0.44} & \red{22} \cdot (\wnum{-0.65}) + \red{24} \cdot (\wnum{-0.68}) + \bnum{0.58} \\ \red{38} \cdot \wnum{0.19} + \red{0} \cdot (\wnum{-0.33}) + \bnum{0.44} & \red{38} \cdot (\wnum{-0.65}) + \red{0} \cdot (\wnum{-0.68}) + \bnum{0.58} \\ \red{14} \cdot \wnum{0.19} + \red{42} \cdot (\wnum{-0.33}) + \bnum{0.44} & \red{14} \cdot (\wnum{-0.65}) + \red{42} \cdot (\wnum{-0.68}) + \bnum{0.58} \end{matrix}}; A_1 = \sigma(Z_1) = \Bb{\gold{\begin{matrix} 0.01 & 0.00 \\ 0.04 & 0.00 \\ 1.00 & 0.00 \\ 0.00 & 0.00 \end{matrix}}} = \Bb{\begin{array}{ll} \sigma(\purple{-4.25}) & \sigma(\purple{-61.16}) \\ \sigma(\purple{-3.30}) & \sigma(\purple{-30.04}) \\ \sigma(\phantom{-}\purple{7.66}) & \sigma(\purple{-24.12}) \\ \sigma(\purple{-10.76}) & \sigma(\purple{-37.08}) \end{array}}
Backpropagation · Review08

Weights as Matrices: Equation Conventions

There are a number of different conventions for writing the linear transformation as used in a linear_layer. I use a slightly non-standard one that I believe maps better to PyTorch and the diagrams I show in class.

ConventionEquationWhere you'll see it
Bias folded inZ = Xω
(X gets a column of 1s)
Linear regression, perceptrons and SVMs
Inputs × outputsZ = Xω + bKeras/TensorFlow, many textbooks
Outputs × inputsZ = XωT + bPyTorch nn.Linear and this course

PyTorch stores one row per neuron so it reads like the math: XωT + b

Ω₁ stored by PyTorch: one row per neuron <2, 2> x₁ x₂ neuron 1 0.19 −0.33 neuron 2 −0.65 −0.68 Ω₁ᵀ transposed for XΩᵀ: one column per neuron <2, 2> neuron 1 neuron 2 x₁ 0.19 −0.65 x₂ −0.33 −0.68
Backpropagation · Review09

Improving the Weights

Linear Regressions and OLS

Due to the simplicity of linear regression, we could use a   to directly find the optimal set of weights.

This is due to the   being perfectly convex

Backpropagation · Gradient Descent10

Gradient Descent

Gradient Descent is an optimization method that   tries to find a set of values (weights) that  

A gradient is the   at a given point in a specified direction

Backpropagation · Gradient Descent11

Multivariate Gradient Descent

In the case of a multivariate function, the gradient is a  

∇ ℒ = ∂ ℒ ∂ω₁ ∂ ℒ ∂ω₂ slope along ω₁ slope along ω₂
Backpropagation · Gradient Descent12

Gradient Descent Algorithm

weight (ω) loss (ℒ) initial weight ω old new weight ω new minimum of the loss step size set by the learning rate α \omega_{new} = \omega_{old} - \alpha\,\frac{\partial ℒ}{\partial \omega} \omega_{new} = \omega_{old} - \alpha\,\frac{\partial ℒ}{\partial \omega}
Backpropagation · Gradient Descent13

Common Loss Functions

Backpropagation · Gradient Descent14

Logistic Regression with Gradient Descent

The chain rule enables us to find derivatives of  !

forward x z = x ωᵀ + b z ŷ = σ ( z ) ŷ ℒ = BCE( ŷ , y ) L ω b y backward z = x ωᵀ + b ŷ = σ ( z ) ℒ = BCE( ŷ , y ) L x
\frac{\partial ℒ}{\partial \omega_1} = \hspace{16em} \frac{\partial ℒ}{\partial b_1} =\frac{\partial ℒ}{\partial \omega_1} = \hspace{16em} \frac{\partial ℒ}{\partial b_1} =

Note: (full derivation) of BCE and σ

Backpropagation · Gradient Descent15

Dimensionality of Gradients

forward one prediction per game z₁₁ z₂₁ z₃₁ z₄₁ Z₂ <4, 1> = a₁₁ a₁₂ a₂₁ a₂₂ a₃₁ a₃₂ a₄₁ a₄₂ A₁ <4, 2> @ ω₁₁ ω₁₂ Ω₂ᵀ <2, 1> + b b₂ <1> weights same shape as Ω₂ ∂ℒ ∂ω₁₁ ∂ℒ ∂ω₁₂ ∂ℒ/∂Ω₂ <1, 2> = ∂ℒ ∂z₁₁ ∂ℒ ∂z₂₁ ∂ℒ ∂z₃₁ ∂ℒ ∂z₄₁ (∂ℒ/∂Z₂)ᵀ <1, 4> @ a₁₁ a₁₂ a₂₁ a₂₂ a₃₁ a₃₂ a₄₁ a₄₂ A₁ <4, 2> bias same shape as b₂ ∂ℒ ∂b ∂ℒ/∂b₂ <1> = Σ games ∂ℒ ∂z₁₁ ∂ℒ ∂z₂₁ ∂ℒ ∂z₃₁ ∂ℒ ∂z₄₁ ∂ℒ/∂Z₂ <4, 1> sent backwards same shape as A₁: the next layer's dY ∂ℒ ∂a₁₁ ∂ℒ ∂a₁₂ ∂ℒ ∂a₂₁ ∂ℒ ∂a₂₂ ∂ℒ ∂a₃₁ ∂ℒ ∂a₃₂ ∂ℒ ∂a₄₁ ∂ℒ ∂a₄₂ ∂ℒ/∂A₁ <4, 2> = ∂ℒ ∂z₁₁ ∂ℒ ∂z₂₁ ∂ℒ ∂z₃₁ ∂ℒ ∂z₄₁ ∂ℒ/∂Z₂ <4, 1> @ ω₁₁ ω₁₂ Ω₂ <1, 2>
Backpropagation · Gradient Descent16

Yes, you should probably understand backpropagation

The start of Andrej Karpathy's Medium post of December 19, 2016: when CS231n was offered at Stanford, students had to write the forward and backward pass of each layer in raw numpy, and some complained, why do we have to write the backward pass when frameworks compute them for you automatically?

karpathy.medium.com/yes-you-should-understand-backprop

Backpropagation (Backprop)

Backpropagation is a technique in deep learning to   for a network. Using the chain rule allows us to implement Gradient Descent for multi-layer models

Backpropagation · Backpropagation17

Python Backprop by Hand - Linear Layer

import torch
class LinearLayer:
def __init__(self, n_in, n_out):
self.W = torch.randn(n_out, n_in) * (2/n_in)**0.5
self.b = torch.zeros(n_out)
def forward(self, X):
self.X = X # keep for backward
return  
def backward(self, dY):
self.dW = dY.T @ self.X # dL/dW
self.db = dY.sum(dim=0) # dL/db
return dY @ self.W # dL/dX, sent back
def update(self, lr):
self.W -= lr * self.dW
self.b -= lr * self.db
Backpropagation · Backpropagation18

Python Backprop by Hand - Sigmoid and ReLU

class Sigmoid:
def forward(self, Z):
self.A = 1 / (1 + torch.exp(-Z)) # saved
return self.A
def backward(self, dA):
return dA * self.A * (1 - self.A) # dL/dZ
def update(self, lr):
pass # no weights to learn
class ReLU:
def forward(self, Z):
self.Z = Z # saved
return Z.clamp(min=0)
def backward(self, dA):
return dA * (self.Z > 0).float() # dL/dZ
def update(self, lr):
pass # no weights to learn
Backpropagation · Backpropagation19

Python Backprop by Hand - Neural Network

class Network:
def __init__(self, layers):
self.layers = layers
def forward(self, X):
for layer in self.layers:
X = layer.forward(X)
return X # logits
def backward(self, y_hat, y):
grad = y_hat - y # dL/dy_hat (hard-coded here)
for layer in reversed(self.layers): # chain rule
grad = layer.backward(grad) # back to front
def update(self, lr):
for layer in self.layers:
layer.update(lr)

Our network class is really just more of a wrapper than anything else, most of the work is done in the layer classes

Backpropagation · Backpropagation20

Training a Neural Network by Hand

# Last lecture's 4 games: ND points, opp. points
X = torch.tensor([[50., 43.], [22., 24.], [38., 0.], [14., 42.]]) # ND Score, Opp. Score
y = torch.tensor([[1.], [1.], [0.], [0.]]) # close = 1, not close = 0
torch.manual_seed(0)
model = Network([LinearLayer(2, 2), Sigmoid(), LinearLayer(2, 1)])
bce = torch.nn.BCELoss()
y_hat = torch.sigmoid(model.forward(X)) # P(close)
print('y_hat at epoch 0:', y_hat.flatten(), 'loss:', bce(y_hat, y))
def train(model, X, lr=0.001, epochs=1000):
for epoch in range(epochs):
y_hat = torch.sigmoid(model.forward(X))
model.backward(y_hat, y)
model.update(lr)
return torch.sigmoid(model.forward(X))
y_hat_trained = train(model, X)
print('y_hat at epoch 1000:', y_hat_trained.flatten(), 'loss:', bce(y_hat_trained, y))
Output
y_hat at epoch 0: tensor([0.5038, 0.5367, 0.4577, 0.5866]) loss: tensor(0.7008)
y_hat at epoch 1000: tensor([0.5537, 0.5523, 0.3362, 0.5538]) loss: tensor(0.6004)
Backpropagation · Backpropagation21

Some Issues with Backprop and Activation Functions

There are some issues that appear when you start using backprop, depending on the activation functions you use.

Saturated Sigmoids:

When a neuron's Z is far from 0, the sigmoid  , so its local derivative is effectively 0 and almost no gradient gets back through it. The neuron is saturated.

Dying ReLUs:

A ReLU's local derivative is  , so a neuron that's negative on every input passes no gradient back and never updates again. The neuron is dead.

Backpropagation · Issues with Backprop22

Saturated Sigmoids

Not dissimilar to the issues with the step function, if we are too far outside of the curve of the sigmoid, we get effectively no useful gradient information from it.

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ −8 −4 0 4 8 z 0.00 0.25 0.50 1.00 6 more further right (14.7 to 60.2) saturated: slope ≈ 0 saturated: slope ≈ 0 slope at most 0.25 σ(z) local derivative σ(z)(1 − σ(z))

Purple dots: this section's network's 8 values of Z1 (4 games × 2 neurons) on raw scores, most of them far out on the flat top.

Backpropagation · Issues with Backprop23

Saturated Neurons: Demo

torch.manual_seed(11) # ensure our random init is always the same
layers = [LinearLayer(2, 2), Sigmoid(), LinearLayer(2, 1)]
model = Network(layers)
y_hat = train(model, X, 0.5)
print("A1:", model.layers[1].A)
print("largest dL/dW1:", model.layers[0].dW.abs().max())
print("y_hat:", y_hat.flatten(), "loss:", bce(y_hat, y))
Output
A1: tensor([[1.0000e+00, 1.0000e+00],
        [1.0000e+00, 1.0000e+00],
        [1.0000e+00, 1.0000e+00],
        [1.0000e+00, 1.9767e-04]])
largest dL/dW1: tensor(8.6757e-05)
y_hat: tensor([0.6660, 0.6660, 0.6660, 0.0031]) loss: tensor(0.4782)

Same 2 → 2 → 1 shape as last class (seed 11, α = 0.5), and we keep it for the rest of the section. On raw scores the hidden layer is pinned at 1, so three of the games look identical to the output layer and almost no gradient reaches ω1.

Backpropagation · Issues with Backprop24

Dying ReLUs

Similarly to the edges of a sigmoid, a ReLU gives no useful gradient information if the Z value is less than 0.

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ −8 −4 0 4 8 z 0 1 4 8 start: 6 more further up the ramp after one step: all 8 between −1,670 and −407 dead: slope = 0 slope 1 ReLU(z) = max(0, z) local derivative

Hollow dots: the same 8 values of Z1 at the start, nearly all on the ramp. One gradient step later, all 8 are in the dead zone.

Backpropagation · Issues with Backprop25

Dying ReLUs: Demo

torch.manual_seed(11) # ensure our random init is always the same
layers = [LinearLayer(2, 2), ReLU(), LinearLayer(2, 1)]
model = Network(layers)
y_hat = train(model, X, 0.5)
A1 = model.layers[1].forward(model.layers[0].forward(X))
dead = (A1.sum(dim=0) == 0).sum().item()
print("A1:", A1)
print("dead neurons:", dead, "of 2")
print("y_hat:", y_hat.flatten(), "loss:", bce(y_hat, y))
Output
A1: tensor([[0., 0.],
        [0., 0.],
        [0., 0.],
        [0., 0.]])
dead neurons: 2 of 2
y_hat: tensor([0.5000, 0.5000, 0.5000, 0.5000]) loss: tensor(0.6931)

Same network, same starting weights, only the activation changed. Both neurons start alive, but the first step, with gradients up to 38 from the raw scores, pushes every z below 0 and kills both. After that every gradient is exactly 0.

Backpropagation · Issues with Backprop26

Fixing Both: Standardize the Inputs

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0 10 20 30 40 50 raw scores standardized inputs x −8 −4 0 4 8 z 0.0 0.5 1.0 6 raw z further right sigmoid −8 −4 0 4 8 z 0 4 8 6 raw z further right ReLU z from raw scores z from standardized scores

Standardized inputs sit on both sides of 0, so every z starts near 0: on the steep part of the sigmoid, and positive for some games, negative for others in each ReLU neuron.

Backpropagation · Issues with Backprop27

Fixing Both: Standardize the Inputs

X_s = (X - X.mean(dim=0)) / X.std(dim=0) # mean 0, std 1
torch.manual_seed(11)
layers = [LinearLayer(2, 2), Sigmoid(), LinearLayer(2, 1)]
y_hat = train(Network(layers), X_s, 0.5)
print("sigmoid y_hat:", y_hat.flatten())
print("sigmoid loss:", bce(y_hat, y))
torch.manual_seed(11)
layers = [LinearLayer(2, 2), ReLU(), LinearLayer(2, 1)]
y_hat = train(Network(layers), X_s, 0.5)
print("ReLU y_hat:", y_hat.flatten())
print("ReLU loss:", bce(y_hat, y))
Output
sigmoid y_hat: tensor([0.9953, 0.9929, 0.0056, 0.0091])
sigmoid loss: tensor(0.0067)
ReLU y_hat: tensor([9.9886e-01, 9.9886e-01, 8.8724e-05, 1.3661e-04])
ReLU loss: tensor(0.0006)
Backpropagation · Issues with Backprop28

Backprop Exercise (old final exam question)

import torch
import torch.nn as nn
class Model4_1(nn.Module):
def __init__(self):
super(Model4_1, self).__init__()
self.lin1 = nn.Linear(784, 100)
self.relu = nn.ReLU()
self.lin2 = nn.Linear(100, 10)
def forward(self, x):
out = self.lin1(x)
out = self.relu(out)
out = self.lin2(out)
return out

Note: You can assume softmax and cross entropy loss are used after the final layer, the two yielding a combined derivative of: ŷ − y

Backpropagation · Issues with Backprop29

Backprop Exercise (old final exam question)

ℒ =ℒ =
\frac{\partial ℒ}{\partial b_1} =\frac{\partial ℒ}{\partial b_1} =
\frac{\partial ℒ}{\partial b_1} =\frac{\partial ℒ}{\partial b_1} =
Backpropagation · Issues with Backprop30

The MNIST Dataset

MNIST (LeCun and others, 1998) is the classic first dataset for neural networks: 70,000 handwritten digits (60,000 to train, 10,000 to test), each a 28 × 28 grayscale image. Flattened, that's the 784 inputs of the exam network we just did.

EMNIST (Cohen and others, 2017) adds letters in the same format. Its balanced split has 47 classes: 10 digits, 26 capitals and the 11 lowercase letters that don't look like their capitals.

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ N O T R E D A M E 7 2 a b d ? ? ? ? ? ? ? ? ? ? ? ? police archive (EMNIST): labelled, 28 × 28 Professor Dingler's note: no labels

In Homework 06 and 07 we train on 9,400 of these from the police archive, then point the network at the cut-out letters in Professor Dingler's note.

Backpropagation · Issues with Backprop31

Modern Momentum: ADAM

Plain gradient descent gives every weight the   α. The exam network has 79,510 weights and biases for MNIST, so weights with big, noisy gradients overshoot while weights with tiny gradients barely move. Adam (Adaptive Moment Estimation, Kingma and Ba, 2014) is the optimizer I see most today, and it fixes the step with two running averages:

m \leftarrow \beta_1\, m + (1 - \beta_1)\, \frac{\partial ℒ}{\partial \omega} v \leftarrow \beta_2\, v + (1 - \beta_2) \bp{\frac{\partial ℒ}{\partial \omega}}^2 \omega \leftarrow \omega - \alpha\, \frac{m}{\sqrt{v} + \epsilon} m \leftarrow \beta_1\, m + (1 - \beta_1)\, \frac{\partial ℒ}{\partial \omega}; v \leftarrow \beta_2\, v + (1 - \beta_2) \bp{\frac{\partial ℒ}{\partial \omega}}^2; \omega \leftarrow \omega - \alpha\, \frac{m}{\sqrt{v} + \epsilon}

In PyTorch it's one line, torch.optim.Adam(model.parameters(), lr=0.001), the optimizer Homework 07's CNN trains with.

NOTE: The defaults are β1 = 0.9, β2 = 0.999, and Adam also corrects m and v for starting at 0.

Backpropagation · Issues with Backprop32

ADAM on MNIST

Kingma and Ba trained a 784 → 1000 → 1000 → 10 ReLU network with dropout on MNIST, the same kind of network as the exam question, just wider. Adam falls fastest and ends lowest.

Training cost on a log scale over 200 passes through MNIST for a multilayer network with dropout, for AdaGrad, RMSProp, SGDNesterov, AdaDelta and Adam: Adam's curve ends lowest
Backpropagation · Issues with Backprop33