Unit 02 · Chapter 1

Principal Component Analysis

Principal Component Analysis01

Dimensionality Expansion

With the SVM kernel trick we saw a way of implicitly increasing the dimensionality of our data set.

This allowed us to fit nonlinear problems.

Principal Component Analysis · Dimensionality02

The Curse of Dimensionality

As we add more features (dimensions) to our dataset, our data becomes more spread out (sparse).

As our number of dimensions increases, the volume of the feature space grows exponentially.

Principal Component Analysis · Dimensionality03

The Curse of Dimensionality

AAAWgmp1bWIAAAAeanVtZGMycGEAEQAQgAAAqgA4m3EDYzJwYQAAABZcanVtYgAAAEdqdW1kYzJtYQARABCAAACqADibcQN1cm46YzJwYTpkZDMyNjUzMC0wZTFiLTRjYmQtYmNkNS05OGI1YzBiNTFmN2IAAAADl2p1bWIAAAApanVtZGMyYXMAEQAQgAAAqgA4m3EDYzJwYS5hc3NlcnRpb25zAAAAALxqdW1iAAAARGp1bWRjYm9yABEAEIAAAKoAOJtxE2MycGEuaW5ncmVkaWVudC52MwAAAAAYYzJzaCPsh+N1gu135yLAJYBG6vMAAABwY2JvcqNpZGM6Zm9ybWF0bWltYWdlL3N2Zyt4bWxqaW5zdGFuY2VJRHgseG1wOmlpZDpiZjEzNmYwYy0yZWI4LTRjYmYtYWM2My1lNWMyNjRkNGJiOTBscmVsYXRpb25zaGlwaHBhcmVudE9mAAAB4mp1bWIAAABBanVtZGNib3IAEQAQgAAAqgA4m3ETYzJwYS5hY3Rpb25zLnYyAAAAABhjMnNo1UbAkVtXMIoCfpQxQ57nsQAAAZljYm9yomdhY3Rpb25zgqJmYWN0aW9ua2MycGEub3BlbmVkanBhcmFtZXRlcnOha2luZ3JlZGllbnRzgaJjdXJseC1zZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmluZ3JlZGllbnQudjNkaGFzaFggHP1U92CLNujNK9cvsB+lR8YkeLHeQ9bsDHsOk6X/muSkZmFjdGlvbngdY29tLmFudGhyb3BpYy5jbGF1ZGUucHJvdmlkZWRqcGFyYW1ldGVyc6F4H2NvbS5hbnRocm9waWMub3JpZ2luLWNvbmZpZGVuY2VndW5rbm93bmtkZXNjcmlwdGlvbnhmQ2xhdWRlIHByb3ZpZGVkIHRoaXMgZmlsZSBhdCB0aGUgcmVxdWVzdCBvZiBhIHVzZXIgYW5kIG1heSBoYXZlIGNyZWF0ZWQgb3IgbW9kaWZpZWQgdGhlIGZpbGUgY29udGVudHMubXNvZnR3YXJlQWdlbnShZG5hbWVmQ2xhdWRlcmFsbEFjdGlvbnNJbmNsdWRlZPUAAADIanVtYgAAAEBqdW1kY2JvcgARABCAAACqADibcRNjMnBhLmhhc2guZGF0YQAAAAAYYzJzaNi6wZm2sBWczWCRqbiLldkAAACAY2JvcqVjYWxnZnNoYTI1NmNwYWRMAAAAAAAAAAAAAAAAZGhhc2hYII7YdhjUWwo/0stIJLwfAF9rqhfhIIhTlcyjYccZ1GXKZG5hbWVuanVtYmYgbWFuaWZlc3RqZXhjbHVzaW9uc4GiZXN0YXJ0GQFxZmxlbmd0aBkeBAAAAj5qdW1iAAAAJ2p1bWRjMmNsABEAEIAAAKoAOJtxA2MycGEuY2xhaW0udjIAAAACD2Nib3KlY2FsZ2ZzaGEyNTZpc2lnbmF0dXJleE1zZWxmI2p1bWJmPS9jMnBhL3VybjpjMnBhOmRkMzI2NTMwLTBlMWItNGNiZC1iY2Q1LTk4YjVjMGI1MWY3Yi9jMnBhLnNpZ25hdHVyZWppbnN0YW5jZUlEeCx4bXA6aWlkOmNhMWRjZWUzLTZmMWItNGQ5OS1hMDcyLTE2Y2I1NTg1NzM5ZXJjcmVhdGVkX2Fzc2VydGlvbnODomN1cmx4LXNlbGYjanVtYmY9YzJwYS5hc3NlcnRpb25zL2MycGEuaW5ncmVkaWVudC52M2RoYXNoWCAc/VT3YIs26M0r1y+wH6VHxiR4sd5D1uwMew6Tpf+a5KJjdXJseCpzZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmFjdGlvbnMudjJkaGFzaFggwyTN+dal2gLCQUo1gpJszhqSqacuH9E9G3/EdN8NLCOiY3VybHgpc2VsZiNqdW1iZj1jMnBhLmFzc2VydGlvbnMvYzJwYS5oYXNoLmRhdGFkaGFzaFgg8LLksTl9kwrO9fihvz+DobGbFv1kEwr0mgwO34Fzeu10Y2xhaW1fZ2VuZXJhdG9yX2luZm+jZG5hbWVvQW50aHJvcGljIEZpbGVzZ3ZlcnNpb25lMS4wLjBrc3BlY1ZlcnNpb25lMi40LjAAABA4anVtYgAAAChqdW1kYzJjcwARABCAAACqADibcQNjMnBhLnNpZ25hdHVyZQAAABAIY2JvctKEWQISogEmGCFZAgowggIGMIIBjaADAgECAhRA5aAK7sI50L64g/oGQgU9Z1UTADAKBggqhkjOPQQDAzBJMRcwFQYDVQQKEw5BbnRocm9waWMsIFBCQzEuMCwGA1UEAxMlQW50aHJvcGljIENvbnRlbnQgQ3JlZGVudGlhbHMgUm9vdCBDQTAeFw0yNjA4MDcxODQzNTZaFw0yODA4MDYxOTQzNTZaMEQxFzAVBgNVBAoTDkFudGhyb3BpYywgUEJDMSkwJwYDVQQDEyBBbnRocm9waWMgQ2xhdWRlIENvbnRlbnQgU2lnbmluZzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJh6CmvLUBgFFNU0vUKlOVtE6djd17L5SuwX0LemFisBM3dkd/3cyjxFA3Qo5S46fX0/ihY0VZ7mfb9KF703t5OjWDBWMA4GA1UdDwEB/wQEAwIHgDAVBgNVHSUEDjAMBgorBgEEAYPoXgIBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUzlHiBIFOZFsj+OPEz5o+nMHXXMIwCgYIKoZIzj0EAwMDZwAwZAIwMXMdFJ4BetLLVY7ORuE9noqbbAZOZn/aArXyTwFAZfKrPzxF2vPoJNf1+UCdg1XGAjBwX1zd9WGqYkqmL5SFqw1QySjr1zJfpJM9+1rdDwSPLMOPOjKuiXjoU/pUUeG9RwmhY3BhZFkNngAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPZYQKSep51IFWzEsnzCgCtcTM7n0+nlBCRDOko1XiH7EfXJb1uPl3I+24WwiIFZ7zsI9hQjr7VJdsWGYbKG4mxx4Eg= image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ total yards: 5 cells, all 5 used total yards and first downs: 25 cells, 13 used

d = 80?

Principal Component Analysis · Dimensionality04

Dimensionality Reduction (exercise)

Dimensionality Reduction is an unsupervised learning technique that helps us project our data from a high dimensional space to a lower dimensional space.

See if you can project the cubes into the lower dimensions.

Principal Component Analysis · Dimensionality05

Are All Features Equally Useful?

We may as well not even use X2, there'd likely be no difference in our ability to solve the problem, but we'd get to do less work.

Principal Component Analysis · Dimensionality06

Projecting Data onto an Axis

X1 X2 onto X1 onto X1 X1 X2 onto X2 onto X2
Principal Component Analysis · PCA07

Variance of Data on an Axis

\text{Var}(X_i) = \frac{1}{n}\sum_{j=1}^{n} (x_{ij} - \mu_i)^2 \text{Var}(X_i) = \frac{1}{n}\sum_{j=1}^{n} (x_{ij} - \mu_i)^2
Principal Component Analysis · PCA08

Variance of Data on an Axis

X1 X2 onto X1 onto X1 X1 X2 onto X2 onto X2
Principal Component Analysis · PCA09

Why Do We Care About Variance

\text{Var}(\hat{\omega}_i \mid X_i) = \frac{\sigma^2}{\sum_j (x_{ij} - \mu_i)^2} \text{Var}(\hat{\omega}_i \mid X_i) = \frac{\sigma^2}{\sum_j (x_{ij} - \mu_i)^2}

The more variance we observe in Xi, the more precisely we can estimate its regression coefficient (ωi), all else equal.

Principal Component Analysis · PCA10

Projecting Data onto a New Axis

What if we could find a new axis with higher variance?

PC1 X1 X2
Principal Component Analysis · PCA11

Principal Component Analysis (PCA)

PCA works by discovering the directions (principal components) along which the data varies the most.

PC1 PC2
Principal Component Analysis · PCA12

Exploring PCA

Say we have the following dataset of students' scores in both Math (X1) and Physics (X2):

Student ID (S)Math (x1)Physics (x2)
90179763510.06.0
211.04.0
38.05.0
43.03.0
51.03.2
62.01.0
Principal Component Analysis · PCA13

Exploring PCA

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0 2 4 6 8 10 12 Math (x1) 0 1 2 3 4 5 6 7 Physics (x2) 1 2 3 4 5 6
Principal Component Analysis · PCA14

Finding the Principal Components

  1. Standardize our data
  2. Calculate the covariance matrix
  3. Compute the eigenvalues and eigenvectors
  4. Sort the eigenvalues and vectors from largest to smallest
  5. Choose how many principal components to use
  6. Project our points onto the new principal components
Principal Component Analysis · PCA15

1.) Standardize the Data

We first calculate the average score for math and for physics.

\mu_1 = \frac{10 + 11 + 8 + 3 + 1 + 2}{6} = 5.83 \qquad \mu_2 = 3.70 \mu_1 = \frac{10 + 11 + 8 + 3 + 1 + 2}{6} = 5.83 \qquad \mu_2 = 3.70

We then shift the data so it is centered and scaled around the origin by subtracting the average and dividing by the standard deviation.

x \leftarrow \frac{x - \mu}{\sigma} \qquad \sigma_1 = 3.98 \quad \sigma_2 = 1.59 x \leftarrow \frac{x - \mu}{\sigma} \qquad \sigma_1 = 3.98 \quad \sigma_2 = 1.59
Principal Component Analysis · PCA16

2.) Calculate the Covariance Matrix

From the covariance matrix we can see the relationships between our features.

ND ID (S)Math (x1)Physics (x2)
9017976351.051.45
21.300.19
30.540.82
4−0.71−0.44
5−1.22−0.32
6−0.96−1.70
Principal Component Analysis · PCA17

2.) Calculate the Covariance Matrix

\text{Var}(X_i) = \frac{1}{n}\sum_j (x_{ij} - \mu_i)^2 \qquad \text{Cov}(X_i, X_k) = \frac{1}{n}\sum_j (x_{ij} - \mu_i)(x_{kj} - \mu_k) C = \Hb{\begin{matrix} \text{Var}(X_1) & \text{Cov}(X_2, X_1) \\ \text{Cov}(X_1, X_2) & \text{Var}(X_2) \end{matrix}} \text{Var}(X_i) = \frac{1}{n}\sum_j (x_{ij} - \mu_i)^2 \qquad \text{Cov}(X_i, X_k) = \frac{1}{n}\sum_j (x_{ij} - \mu_i)(x_{kj} - \mu_k); C = \Hb{\begin{matrix} \text{Var}(X_1) & \text{Cov}(X_2, X_1) \\ \text{Cov}(X_1, X_2) & \text{Var}(X_2) \end{matrix}}
Principal Component Analysis · PCA18

2.) Calculate the Covariance Matrix

\text{Var}(X_1) = 1 \qquad \text{Var}(X_2) = 1 \text{Cov}(X_1, X_2) = \text{Cov}(X_2, X_1) = \frac{1.05(1.45) + 1.30(0.19) + \dots + (-0.96)(-1.70)}{6} = 0.7584 \text{Var}(X_1) = 1 \qquad \text{Var}(X_2) = 1; \text{Cov}(X_1, X_2) = \text{Cov}(X_2, X_1) = \frac{1.05(1.45) + 1.30(0.19) + \dots + (-0.96)(-1.70)}{6} = 0.7584 C = \Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} C = \Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}}
Principal Component Analysis · PCA19

Interpreting Covariance

Covariance is a measure of how our data relates to each other.

We could also think of covariance as a measure of how much "informational" overlap our features have.

Principal Component Analysis · PCA20

Interpreting the Covariance Matrix

The total amount of variance in our data is:

\text{Total Var} = \text{Var}(X_1) + \text{Var}(X_2) = 1 + 1 = 2 \text{Total Var} = \text{Var}(X_1) + \text{Var}(X_2) = 1 + 1 = 2

The percentage of total variance per feature is:

\%X_1 = \frac{\text{Var}(X_1)}{\text{Total Var}} = \frac{1}{2} = 50\% \qquad \%X_2 = \frac{1}{2} = 50\% \%X_1 = \frac{\text{Var}(X_1)}{\text{Total Var}} = \frac{1}{2} = 50\% \qquad \%X_2 = \frac{1}{2} = 50\%

So X1 holds 50% of what separates our data and X2 has 50% of what separates our data.

Principal Component Analysis · PCA21

Interpreting the Covariance Matrix

We can also calculate shared linear variation by:

(0.7584)^2 = 0.5752 (0.7584)^2 = 0.5752

So 57.52% of the variation in Physics can be explained using only Math.

Principal Component Analysis · PCA22

3.) Find the Eigenvalues

The eigenvalues represent the amount of variance contained by each principal component we find.

The eigenvalue equation is:

\det(C - \lambda I) = 0 \det(C - \lambda I) = 0 C = \Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} \qquad I = \Hb{\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix}} C = \Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} \qquad I = \Hb{\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix}}

Using the identity matrix means we're solving for covariance being 0 (or an informational overlap of 0).

Principal Component Analysis · PCA23

3.) Compute the Eigenvalues

\det\left(\Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} - \lambda \Hb{\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix}}\right) = 0 \text{subtract } \lambda \text{ on the diagonal} \det\Hb{\begin{matrix} 1-\lambda & 0.7584 \\ 0.7584 & 1-\lambda \end{matrix}} = 0 2 \times 2 \text{ determinant: } ad - bc (1-\lambda)^2 - 0.7584^2 = 0 \text{square root of both sides} 1 - \lambda = \pm 0.7584 \lambda_1 = 1.7584 \qquad \lambda_2 = 0.2416 \det\left(\Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} - \lambda \Hb{\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix}}\right) = 0; \text{subtract } \lambda \text{ on the diagonal}; \det\Hb{\begin{matrix} 1-\lambda & 0.7584 \\ 0.7584 & 1-\lambda \end{matrix}} = 0; 2 \times 2 \text{ determinant: } ad - bc; (1-\lambda)^2 - 0.7584^2 = 0; \text{square root of both sides}; 1 - \lambda = \pm 0.7584; \lambda_1 = 1.7584 \qquad \lambda_2 = 0.2416
Principal Component Analysis · PCA24

3.) Compute the Eigenvectors

While the eigenvalues capture how much variance is explained by each principal component, the eigenvectors show us the directions along which the principal components point.

The eigenvector equation is:

(C - \lambda I)\,V = 0 (C - \lambda I)\,V = 0

λ1 = 1.7584 λ2 = 0.2416

Principal Component Analysis · PCA25

Interpreting the Eigenvalues

You can think of the eigenvalues as the values in the covariance matrix for our two new axes (principal components).

C = \Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} C = \Hb{\begin{matrix} 1.7584 & 0.0 \\ 0.0 & 0.2416 \end{matrix}} C = \Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}}; C = \Hb{\begin{matrix} 1.7584 & 0.0 \\ 0.0 & 0.2416 \end{matrix}}

The total variance in our data is still 2 but now the first axis (PC1) contains 87.92% of the total variance, compared to X1 with 50.00%.

Principal Component Analysis · PCA26

3.) Compute the Eigenvectors

\left(\Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} - 1.7584\,I\right) V_1 = 0 \quad\Rightarrow\quad V_1 = \Hb{\begin{matrix} 0.7071 \\ 0.7071 \end{matrix}} \left(\Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} - 0.2416\,I\right) V_2 = 0 \quad\Rightarrow\quad V_2 = \Hb{\begin{matrix} -0.7071 \\ 0.7071 \end{matrix}} \left(\Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} - 1.7584\,I\right) V_1 = 0 \quad\Rightarrow\quad V_1 = \Hb{\begin{matrix} 0.7071 \\ 0.7071 \end{matrix}}; \left(\Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} - 0.2416\,I\right) V_2 = 0 \quad\Rightarrow\quad V_2 = \Hb{\begin{matrix} -0.7071 \\ 0.7071 \end{matrix}}

Notice that V1 and V2 are orthogonal (perpendicular) to each other.

Principal Component Analysis · PCA27

Orthogonality of Principal Components

PCA can be seen as a rotation of the original data axes to a new set of axes. If the principal components weren't orthogonal, there would be overlap between the axes, meaning that some of the variance would be captured in multiple directions, resulting in less efficient dimensionality reduction.

Principal Component Analysis · PCA28

Orthogonality of Principal Components

PC1 PC2
Principal Component Analysis · PCA29

Orthogonality of Principal Components

Imagine non-orthogonal axes. Moving the X1 of one of the points would inherently move the X2 value of that same point! This would be extremely high covariance, and have large "informational" overlap between features.

Principal Component Analysis · PCA30

4. and 5.) Sorting and Picking PCAs

Using PCA we find a number of principal components equal to the number of original features in our dataset, but we typically aren't interested in all of them. Instead we want a minimum set that can explain the majority of the variance in our data.

Principal Component Analysis · Using PCA31

4. and 5.) Sorting and Picking PCAs

AAAWgmp1bWIAAAAeanVtZGMycGEAEQAQgAAAqgA4m3EDYzJwYQAAABZcanVtYgAAAEdqdW1kYzJtYQARABCAAACqADibcQN1cm46YzJwYTozYTIxMTI0Ny0wMDE0LTRkOWYtYTRiOS0zYjQ2ZWFmNTQ1ZWYAAAADl2p1bWIAAAApanVtZGMyYXMAEQAQgAAAqgA4m3EDYzJwYS5hc3NlcnRpb25zAAAAALxqdW1iAAAARGp1bWRjYm9yABEAEIAAAKoAOJtxE2MycGEuaW5ncmVkaWVudC52MwAAAAAYYzJzaFep3d0eV0HKuJJMuXdByGAAAABwY2JvcqNpZGM6Zm9ybWF0bWltYWdlL3N2Zyt4bWxqaW5zdGFuY2VJRHgseG1wOmlpZDo3NjU3MmRkMC0yYTA1LTQzM2EtODMxZC1mMmYxMDQwMmE2NzFscmVsYXRpb25zaGlwaHBhcmVudE9mAAAB4mp1bWIAAABBanVtZGNib3IAEQAQgAAAqgA4m3ETYzJwYS5hY3Rpb25zLnYyAAAAABhjMnNoV09wfVR6YRGfWGa0Mt82+AAAAZljYm9yomdhY3Rpb25zgqJmYWN0aW9ua2MycGEub3BlbmVkanBhcmFtZXRlcnOha2luZ3JlZGllbnRzgaJjdXJseC1zZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmluZ3JlZGllbnQudjNkaGFzaFggAqFCIx7Dj8LwQSQNF4rcPU4mpuaVEHNVwZuqOsjdJt+kZmFjdGlvbngdY29tLmFudGhyb3BpYy5jbGF1ZGUucHJvdmlkZWRqcGFyYW1ldGVyc6F4H2NvbS5hbnRocm9waWMub3JpZ2luLWNvbmZpZGVuY2VndW5rbm93bmtkZXNjcmlwdGlvbnhmQ2xhdWRlIHByb3ZpZGVkIHRoaXMgZmlsZSBhdCB0aGUgcmVxdWVzdCBvZiBhIHVzZXIgYW5kIG1heSBoYXZlIGNyZWF0ZWQgb3IgbW9kaWZpZWQgdGhlIGZpbGUgY29udGVudHMubXNvZnR3YXJlQWdlbnShZG5hbWVmQ2xhdWRlcmFsbEFjdGlvbnNJbmNsdWRlZPUAAADIanVtYgAAAEBqdW1kY2JvcgARABCAAACqADibcRNjMnBhLmhhc2guZGF0YQAAAAAYYzJzaL4FYhqJMv2vPazMqEkU7PYAAACAY2JvcqVjYWxnZnNoYTI1NmNwYWRMAAAAAAAAAAAAAAAAZGhhc2hYIJQzS6wgU0F1uc3tVIImqFZj2Z9uIKIFFqb6CJQYL91QZG5hbWVuanVtYmYgbWFuaWZlc3RqZXhjbHVzaW9uc4GiZXN0YXJ0GQFxZmxlbmd0aBkeBAAAAj5qdW1iAAAAJ2p1bWRjMmNsABEAEIAAAKoAOJtxA2MycGEuY2xhaW0udjIAAAACD2Nib3KlY2FsZ2ZzaGEyNTZpc2lnbmF0dXJleE1zZWxmI2p1bWJmPS9jMnBhL3VybjpjMnBhOjNhMjExMjQ3LTAwMTQtNGQ5Zi1hNGI5LTNiNDZlYWY1NDVlZi9jMnBhLnNpZ25hdHVyZWppbnN0YW5jZUlEeCx4bXA6aWlkOjg4NGExNTQwLTM3ODQtNGQ1YS05Zjc5LWQyOTgzY2YzYzJiYXJjcmVhdGVkX2Fzc2VydGlvbnODomN1cmx4LXNlbGYjanVtYmY9YzJwYS5hc3NlcnRpb25zL2MycGEuaW5ncmVkaWVudC52M2RoYXNoWCACoUIjHsOPwvBBJA0Xitw9Tiam5pUQc1XBm6o6yN0m36JjdXJseCpzZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmFjdGlvbnMudjJkaGFzaFgg+w6DJf55PSU9MwCjL0pStx6xVl9FSAIjCwGa/F+MXyiiY3VybHgpc2VsZiNqdW1iZj1jMnBhLmFzc2VydGlvbnMvYzJwYS5oYXNoLmRhdGFkaGFzaFgg8cSCquWXKZFvcM/9e3j1TVupYL7rDPmIozlo9sH49ZJ0Y2xhaW1fZ2VuZXJhdG9yX2luZm+jZG5hbWVvQW50aHJvcGljIEZpbGVzZ3ZlcnNpb25lMS4wLjBrc3BlY1ZlcnNpb25lMi40LjAAABA4anVtYgAAAChqdW1kYzJjcwARABCAAACqADibcQNjMnBhLnNpZ25hdHVyZQAAABAIY2JvctKEWQISogEmGCFZAgowggIGMIIBjaADAgECAhRA5aAK7sI50L64g/oGQgU9Z1UTADAKBggqhkjOPQQDAzBJMRcwFQYDVQQKEw5BbnRocm9waWMsIFBCQzEuMCwGA1UEAxMlQW50aHJvcGljIENvbnRlbnQgQ3JlZGVudGlhbHMgUm9vdCBDQTAeFw0yNjA4MDcxODQzNTZaFw0yODA4MDYxOTQzNTZaMEQxFzAVBgNVBAoTDkFudGhyb3BpYywgUEJDMSkwJwYDVQQDEyBBbnRocm9waWMgQ2xhdWRlIENvbnRlbnQgU2lnbmluZzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJh6CmvLUBgFFNU0vUKlOVtE6djd17L5SuwX0LemFisBM3dkd/3cyjxFA3Qo5S46fX0/ihY0VZ7mfb9KF703t5OjWDBWMA4GA1UdDwEB/wQEAwIHgDAVBgNVHSUEDjAMBgorBgEEAYPoXgIBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUzlHiBIFOZFsj+OPEz5o+nMHXXMIwCgYIKoZIzj0EAwMDZwAwZAIwMXMdFJ4BetLLVY7ORuE9noqbbAZOZn/aArXyTwFAZfKrPzxF2vPoJNf1+UCdg1XGAjBwX1zd9WGqYkqmL5SFqw1QySjr1zJfpJM9+1rdDwSPLMOPOjKuiXjoU/pUUeG9RwmhY3BhZFkNngAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPZYQB/FlpcRptXgN4aFAdWYwFEZY8gjr/XXO8uIS/Es+FfqlVV9wyfZoMsj0Kiz2ghZgHvVrrbD9siGfgp2JUpT2TQ= image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 1 5 10 15 20 25 principal component 0 2 4 6 variance (eigenvalue) elbow
Principal Component Analysis · Using PCA32

Why Use PCA

When using PCA, we can often get equivalent accuracy with lower training time* and memory requirements!

In many datasets, not all features are equally useful!

Principal Component Analysis · Using PCA33

6. Project Our Data into the PCA Space

The final step is to use the eigenvalues and eigenvectors of our selected principal components to transform our original data into our new "PC space".

z = x\,V_1 = 0.7071(1.05) + 0.7071(1.45) = 1.77 z = x\,V_1 = 0.7071(1.05) + 0.7071(1.45) = 1.77
Principal Component Analysis · Using PCA34

6. Project Our Data into the PCA Space

PC1 X1 X2
Principal Component Analysis · Using PCA35

Using PCA in Code

Like always it's easy.

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
# math and physics scores for our six students
X = np.array([[10, 6], [11, 4], [8, 5],
[3, 3], [1, 3.2], [2, 1]])
Z = StandardScaler().fit_transform(X) # step 1
pca = PCA(n_components=1).fit(Z) # steps 2 to 5
print(pca.components_.round(4))
print(pca.explained_variance_ratio_.round(4))
print(pca.transform(Z).round(2).ravel()) # step 6
Output
[[0.7071 0.7071]]
[0.8792]
[ 1.77  1.05  0.96 -0.82 -1.08 -1.89]
Principal Component Analysis · Using PCA36

Input Space vs. Feature Space

So far we've worked primarily in input space. Input space is the space of our original measured features.

However PCA finds us new dimensions. When we plot our original points along these new dimensions they could be considered to now be in feature/latent space.

Much of Deep Learning is about transforming our input into latent spaces that we learn.

Principal Component Analysis · Using PCA37

Eigenfaces: Turk and Pentland (1991)

Principal Component Analysis · Using PCA38

Dimensionality Reduction in the Eye

Each eye has well over 100 million light-sensing cells, but only about 1 million fibers in the optic nerve, so the retina has to compress what we see before it goes to the brain.

Color is one example. Our three kinds of cone overlap a lot, so their signals are highly correlated, and the retina recombines them into light–dark, red–green and blue–yellow channels. Buchsbaum and Gottschalk (1983) showed these are close to the principal components of the cone signals.

Oja (1982) showed a single neuron with a Hebbian learning rule ends up finding the first principal component of its inputs.

Principal Component Analysis · Using PCA39

Review

Complete the review: Eigenvectors identify the directions of the principal components, eigenvalues quantify how much variance each component holds. A scree plot helps us choose how many components to keep.

Principal Component Analysis · Review40

Matching: Principal Component Analysis

  E   Principal componentA. Projecting data from a high-dimensional space to a lower-dimensional one
  D   EigenvalueB. The space of our original measured features
  C   EigenvectorC. The direction along which a principal component points
  G   CovarianceD. The amount of variance contained by each principal component
  J   VarianceE. A new axis along which the data varies the most
  I   Scree plotF. As dimensions grow, the data needed to generalize grows exponentially
  A   Dimensionality reductionG. How two features vary together, their "informational" overlap
  F   Curse of dimensionalityH. The new dimensions found by a transformation such as PCA
  B   Input spaceI. Variance explained per component, used to choose how many to keep
  H   Feature (latent) spaceJ. How spread out the data is along an axis
Principal Component Analysis · Review41

Practice Problems

Principal Component Analysis · Review42

Practice Problems, Continued

Principal Component Analysis · Review43

Practice Problems, Continued

Principal Component Analysis · Review44

Practice Problems, Continued

Principal Component Analysis · Review45

Practice Problems, Continued

Principal Component Analysis · Review46