Unit 02 · Chapter 1
Principal Component Analysis
CSE 30124 · Introduction to Artificial Intelligence William Theisen · University of Notre Dame
Dimensionality Expansion
With the SVM kernel trick we saw a way of implicitly increasing the dimensionality of our data set.
This allowed us to fit nonlinear problems .
The Curse of Dimensionality
As we add more features (dimensions) to our dataset, our data becomes more spread out (sparse) .
As our number of dimensions increases, the volume of the feature space grows exponentially .
The Curse of Dimensionality
AAAWgmp1bWIAAAAeanVtZGMycGEAEQAQgAAAqgA4m3EDYzJwYQAAABZcanVtYgAAAEdqdW1kYzJtYQARABCAAACqADibcQN1cm46YzJwYTpkZDMyNjUzMC0wZTFiLTRjYmQtYmNkNS05OGI1YzBiNTFmN2IAAAADl2p1bWIAAAApanVtZGMyYXMAEQAQgAAAqgA4m3EDYzJwYS5hc3NlcnRpb25zAAAAALxqdW1iAAAARGp1bWRjYm9yABEAEIAAAKoAOJtxE2MycGEuaW5ncmVkaWVudC52MwAAAAAYYzJzaCPsh+N1gu135yLAJYBG6vMAAABwY2JvcqNpZGM6Zm9ybWF0bWltYWdlL3N2Zyt4bWxqaW5zdGFuY2VJRHgseG1wOmlpZDpiZjEzNmYwYy0yZWI4LTRjYmYtYWM2My1lNWMyNjRkNGJiOTBscmVsYXRpb25zaGlwaHBhcmVudE9mAAAB4mp1bWIAAABBanVtZGNib3IAEQAQgAAAqgA4m3ETYzJwYS5hY3Rpb25zLnYyAAAAABhjMnNo1UbAkVtXMIoCfpQxQ57nsQAAAZljYm9yomdhY3Rpb25zgqJmYWN0aW9ua2MycGEub3BlbmVkanBhcmFtZXRlcnOha2luZ3JlZGllbnRzgaJjdXJseC1zZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmluZ3JlZGllbnQudjNkaGFzaFggHP1U92CLNujNK9cvsB+lR8YkeLHeQ9bsDHsOk6X/muSkZmFjdGlvbngdY29tLmFudGhyb3BpYy5jbGF1ZGUucHJvdmlkZWRqcGFyYW1ldGVyc6F4H2NvbS5hbnRocm9waWMub3JpZ2luLWNvbmZpZGVuY2VndW5rbm93bmtkZXNjcmlwdGlvbnhmQ2xhdWRlIHByb3ZpZGVkIHRoaXMgZmlsZSBhdCB0aGUgcmVxdWVzdCBvZiBhIHVzZXIgYW5kIG1heSBoYXZlIGNyZWF0ZWQgb3IgbW9kaWZpZWQgdGhlIGZpbGUgY29udGVudHMubXNvZnR3YXJlQWdlbnShZG5hbWVmQ2xhdWRlcmFsbEFjdGlvbnNJbmNsdWRlZPUAAADIanVtYgAAAEBqdW1kY2JvcgARABCAAACqADibcRNjMnBhLmhhc2guZGF0YQAAAAAYYzJzaNi6wZm2sBWczWCRqbiLldkAAACAY2JvcqVjYWxnZnNoYTI1NmNwYWRMAAAAAAAAAAAAAAAAZGhhc2hYII7YdhjUWwo/0stIJLwfAF9rqhfhIIhTlcyjYccZ1GXKZG5hbWVuanVtYmYgbWFuaWZlc3RqZXhjbHVzaW9uc4GiZXN0YXJ0GQFxZmxlbmd0aBkeBAAAAj5qdW1iAAAAJ2p1bWRjMmNsABEAEIAAAKoAOJtxA2MycGEuY2xhaW0udjIAAAACD2Nib3KlY2FsZ2ZzaGEyNTZpc2lnbmF0dXJleE1zZWxmI2p1bWJmPS9jMnBhL3VybjpjMnBhOmRkMzI2NTMwLTBlMWItNGNiZC1iY2Q1LTk4YjVjMGI1MWY3Yi9jMnBhLnNpZ25hdHVyZWppbnN0YW5jZUlEeCx4bXA6aWlkOmNhMWRjZWUzLTZmMWItNGQ5OS1hMDcyLTE2Y2I1NTg1NzM5ZXJjcmVhdGVkX2Fzc2VydGlvbnODomN1cmx4LXNlbGYjanVtYmY9YzJwYS5hc3NlcnRpb25zL2MycGEuaW5ncmVkaWVudC52M2RoYXNoWCAc/VT3YIs26M0r1y+wH6VHxiR4sd5D1uwMew6Tpf+a5KJjdXJseCpzZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmFjdGlvbnMudjJkaGFzaFggwyTN+dal2gLCQUo1gpJszhqSqacuH9E9G3/EdN8NLCOiY3VybHgpc2VsZiNqdW1iZj1jMnBhLmFzc2VydGlvbnMvYzJwYS5oYXNoLmRhdGFkaGFzaFgg8LLksTl9kwrO9fihvz+DobGbFv1kEwr0mgwO34Fzeu10Y2xhaW1fZ2VuZXJhdG9yX2luZm+jZG5hbWVvQW50aHJvcGljIEZpbGVzZ3ZlcnNpb25lMS4wLjBrc3BlY1ZlcnNpb25lMi40LjAAABA4anVtYgAAAChqdW1kYzJjcwARABCAAACqADibcQNjMnBhLnNpZ25hdHVyZQAAABAIY2JvctKEWQISogEmGCFZAgowggIGMIIBjaADAgECAhRA5aAK7sI50L64g/oGQgU9Z1UTADAKBggqhkjOPQQDAzBJMRcwFQYDVQQKEw5BbnRocm9waWMsIFBCQzEuMCwGA1UEAxMlQW50aHJvcGljIENvbnRlbnQgQ3JlZGVudGlhbHMgUm9vdCBDQTAeFw0yNjA4MDcxODQzNTZaFw0yODA4MDYxOTQzNTZaMEQxFzAVBgNVBAoTDkFudGhyb3BpYywgUEJDMSkwJwYDVQQDEyBBbnRocm9waWMgQ2xhdWRlIENvbnRlbnQgU2lnbmluZzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJh6CmvLUBgFFNU0vUKlOVtE6djd17L5SuwX0LemFisBM3dkd/3cyjxFA3Qo5S46fX0/ihY0VZ7mfb9KF703t5OjWDBWMA4GA1UdDwEB/wQEAwIHgDAVBgNVHSUEDjAMBgorBgEEAYPoXgIBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUzlHiBIFOZFsj+OPEz5o+nMHXXMIwCgYIKoZIzj0EAwMDZwAwZAIwMXMdFJ4BetLLVY7ORuE9noqbbAZOZn/aArXyTwFAZfKrPzxF2vPoJNf1+UCdg1XGAjBwX1zd9WGqYkqmL5SFqw1QySjr1zJfpJM9+1rdDwSPLMOPOjKuiXjoU/pUUeG9RwmhY3BhZFkNngAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPZYQKSep51IFWzEsnzCgCtcTM7n0+nlBCRDOko1XiH7EfXJb1uPl3I+24WwiIFZ7zsI9hQjr7VJdsWGYbKG4mxx4Eg=
image/svg+xml
Matplotlib v3.11.2, https://matplotlib.org/
total yards: 5 cells, all 5 used
total yards and first downs: 25 cells, 13 used
d = 80?
Dimensionality Reduction (exercise)
Dimensionality Reduction is an unsupervised learning technique that helps us project our data from a high dimensional space to a lower dimensional space .
See if you can project the cubes into the lower dimensions.
Are All Features Equally Useful?
X1 = Sq. Ftg.
X2 = # of doors
y = Sale price
We may as well not even use X2 , there'd likely be no difference in our ability to solve the problem, but we'd get to do less work .
Projecting Data onto an Axis
X1
X2
onto X1
onto X1
X1
X2
onto X2
onto X2
Variance of Data on an Axis
\text{Var}(X_i) = \frac{1}{n}\sum_{j=1}^{n} (x_{ij} - \mu_i)^2
Variance of Data on an Axis
X1
X2
onto X1
onto X1
X1
X2
onto X2
onto X2
Why Do We Care About Variance
X1 = Sq. Ftg., X2 = # of doors, y = Sale price
\text{Var}(\hat{\omega}_i \mid X_i) = \frac{\sigma^2}{\sum_j (x_{ij} - \mu_i)^2}
The more variance we observe in Xi , the more precisely we can estimate its regression coefficient (ωi ), all else equal.
Projecting Data onto a New Axis
What if we could find a new axis with higher variance ?
PC1
X1
X2
Principal Component Analysis (PCA)
PCA works by discovering the directions (principal components ) along which the data varies the most .
PC1
PC2
Exploring PCA
Say we have the following dataset of students' scores in both Math (X1 ) and Physics (X2 ):
Student ID (S) Math (x1 ) Physics (x2 )
901797635 10.0 6.0
2 11.0 4.0
3 8.0 5.0
4 3.0 3.0
5 1.0 3.2
6 2.0 1.0
Exploring PCA
image/svg+xml
Matplotlib v3.11.2, https://matplotlib.org/
0
2
4
6
8
10
12
Math (x1)
0
1
2
3
4
5
6
7
Physics (x2)
1
2
3
4
5
6
Finding the Principal Components
Standardize our data
Calculate the covariance matrix
Compute the eigenvalues and eigenvectors
Sort the eigenvalues and vectors from largest to smallest
Choose how many principal components to use
Project our points onto the new principal components
1.) Standardize the Data
We first calculate the average score for math and for physics.
\mu_1 = \frac{10 + 11 + 8 + 3 + 1 + 2}{6} = 5.83 \qquad \mu_2 = 3.70
We then shift the data so it is centered and scaled around the origin by subtracting the average and dividing by the standard deviation .
x \leftarrow \frac{x - \mu}{\sigma} \qquad \sigma_1 = 3.98 \quad \sigma_2 = 1.59
Q25. What does it mean to standardize your data? Why do you need to standardize your data before performing PCA?Subtract each feature's average and divide by its standard deviation, so every feature has average 0 and variance 1. PCA looks for the directions with the most variance, so without it a feature measured in big units (dollars, yards) would have the most variance just because of its units and take over the first component.
2.) Calculate the Covariance Matrix
From the covariance matrix we can see the relationships between our features .
ND ID (S) Math (x1 ) Physics (x2 )
901797635 1.05 1.45
2 1.30 0.19
3 0.54 0.82
4 −0.71 −0.44
5 −1.22 −0.32
6 −0.96 −1.70
2.) Calculate the Covariance Matrix
\text{Var}(X_i) = \frac{1}{n}\sum_j (x_{ij} - \mu_i)^2 \qquad \text{Cov}(X_i, X_k) = \frac{1}{n}\sum_j (x_{ij} - \mu_i)(x_{kj} - \mu_k)
C = \Hb{\begin{matrix} \text{Var}(X_1) & \text{Cov}(X_2, X_1) \\ \text{Cov}(X_1, X_2) & \text{Var}(X_2) \end{matrix}}
2.) Calculate the Covariance Matrix
\text{Var}(X_1) = 1 \qquad \text{Var}(X_2) = 1
\text{Cov}(X_1, X_2) = \text{Cov}(X_2, X_1) = \frac{1.05(1.45) + 1.30(0.19) + \dots + (-0.96)(-1.70)}{6} = 0.7584
C = \Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}}
Interpreting Covariance
A positive covariance indicates that when one increases, the other increases (here we have Cov = 0.7584 )
A negative covariance indicates that when one increases, the other decreases
A covariance close to 0 indicates that there's no real relationship between the two variables
Covariance is a measure of how our data relates to each other.
We could also think of covariance as a measure of how much "informational" overlap our features have.
Interpreting the Covariance Matrix
The total amount of variance in our data is:
\text{Total Var} = \text{Var}(X_1) + \text{Var}(X_2) = 1 + 1 = 2
The percentage of total variance per feature is:
\%X_1 = \frac{\text{Var}(X_1)}{\text{Total Var}} = \frac{1}{2} = 50\% \qquad \%X_2 = \frac{1}{2} = 50\%
So X1 holds 50% of what separates our data and X2 has 50% of what separates our data.
Interpreting the Covariance Matrix
We can also calculate shared linear variation by:
(0.7584)^2 = 0.5752
So 57.52% of the variation in Physics can be explained using only Math .
3.) Find the Eigenvalues
The eigenvalues represent the amount of variance contained by each principal component we find.
The eigenvalue equation is:
\det(C - \lambda I) = 0
C = \Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} \qquad I = \Hb{\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix}}
Using the identity matrix means we're solving for covariance being 0 (or an informational overlap of 0).
3.) Compute the Eigenvalues
\det\left(\Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} - \lambda \Hb{\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix}}\right) = 0
\text{subtract } \lambda \text{ on the diagonal}
\det\Hb{\begin{matrix} 1-\lambda & 0.7584 \\ 0.7584 & 1-\lambda \end{matrix}} = 0
2 \times 2 \text{ determinant: } ad - bc
(1-\lambda)^2 - 0.7584^2 = 0
\text{square root of both sides}
1 - \lambda = \pm 0.7584
\lambda_1 = 1.7584 \qquad \lambda_2 = 0.2416
3.) Compute the Eigenvectors
While the eigenvalues capture how much variance is explained by each principal component, the eigenvectors show us the directions along which the principal components point.
The eigenvector equation is:
(C - \lambda I)\,V = 0
λ1 = 1.7584 λ2 = 0.2416
Interpreting the Eigenvalues
You can think of the eigenvalues as the values in the covariance matrix for our two new axes (principal components).
C = \Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}}
C = \Hb{\begin{matrix} 1.7584 & 0.0 \\ 0.0 & 0.2416 \end{matrix}}
The total variance in our data is still 2 but now the first axis (PC1) contains 87.92% of the total variance , compared to X1 with 50.00% .
3.) Compute the Eigenvectors
\left(\Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} - 1.7584\,I\right) V_1 = 0 \quad\Rightarrow\quad V_1 = \Hb{\begin{matrix} 0.7071 \\ 0.7071 \end{matrix}}
\left(\Hb{\begin{matrix} 1 & 0.7584 \\ 0.7584 & 1 \end{matrix}} - 0.2416\,I\right) V_2 = 0 \quad\Rightarrow\quad V_2 = \Hb{\begin{matrix} -0.7071 \\ 0.7071 \end{matrix}}
Notice that V1 and V2 are orthogonal (perpendicular) to each other.
Orthogonality of Principal Components
PCA can be seen as a rotation of the original data axes to a new set of axes. If the principal components weren't orthogonal, there would be overlap between the axes, meaning that some of the variance would be captured in multiple directions, resulting in less efficient dimensionality reduction .
Orthogonality of Principal Components
PC1
PC2
Orthogonality of Principal Components
Imagine non-orthogonal axes. Moving the X1 of one of the points would inherently move the X2 value of that same point! This would be extremely high covariance , and have large "informational" overlap between features.
4. and 5.) Sorting and Picking PCAs
Using PCA we find a number of principal components equal to the number of original features in our dataset, but we typically aren't interested in all of them. Instead we want a minimum set that can explain the majority of the variance in our data.
4. and 5.) Sorting and Picking PCAs
AAAWgmp1bWIAAAAeanVtZGMycGEAEQAQgAAAqgA4m3EDYzJwYQAAABZcanVtYgAAAEdqdW1kYzJtYQARABCAAACqADibcQN1cm46YzJwYTozYTIxMTI0Ny0wMDE0LTRkOWYtYTRiOS0zYjQ2ZWFmNTQ1ZWYAAAADl2p1bWIAAAApanVtZGMyYXMAEQAQgAAAqgA4m3EDYzJwYS5hc3NlcnRpb25zAAAAALxqdW1iAAAARGp1bWRjYm9yABEAEIAAAKoAOJtxE2MycGEuaW5ncmVkaWVudC52MwAAAAAYYzJzaFep3d0eV0HKuJJMuXdByGAAAABwY2JvcqNpZGM6Zm9ybWF0bWltYWdlL3N2Zyt4bWxqaW5zdGFuY2VJRHgseG1wOmlpZDo3NjU3MmRkMC0yYTA1LTQzM2EtODMxZC1mMmYxMDQwMmE2NzFscmVsYXRpb25zaGlwaHBhcmVudE9mAAAB4mp1bWIAAABBanVtZGNib3IAEQAQgAAAqgA4m3ETYzJwYS5hY3Rpb25zLnYyAAAAABhjMnNoV09wfVR6YRGfWGa0Mt82+AAAAZljYm9yomdhY3Rpb25zgqJmYWN0aW9ua2MycGEub3BlbmVkanBhcmFtZXRlcnOha2luZ3JlZGllbnRzgaJjdXJseC1zZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmluZ3JlZGllbnQudjNkaGFzaFggAqFCIx7Dj8LwQSQNF4rcPU4mpuaVEHNVwZuqOsjdJt+kZmFjdGlvbngdY29tLmFudGhyb3BpYy5jbGF1ZGUucHJvdmlkZWRqcGFyYW1ldGVyc6F4H2NvbS5hbnRocm9waWMub3JpZ2luLWNvbmZpZGVuY2VndW5rbm93bmtkZXNjcmlwdGlvbnhmQ2xhdWRlIHByb3ZpZGVkIHRoaXMgZmlsZSBhdCB0aGUgcmVxdWVzdCBvZiBhIHVzZXIgYW5kIG1heSBoYXZlIGNyZWF0ZWQgb3IgbW9kaWZpZWQgdGhlIGZpbGUgY29udGVudHMubXNvZnR3YXJlQWdlbnShZG5hbWVmQ2xhdWRlcmFsbEFjdGlvbnNJbmNsdWRlZPUAAADIanVtYgAAAEBqdW1kY2JvcgARABCAAACqADibcRNjMnBhLmhhc2guZGF0YQAAAAAYYzJzaL4FYhqJMv2vPazMqEkU7PYAAACAY2JvcqVjYWxnZnNoYTI1NmNwYWRMAAAAAAAAAAAAAAAAZGhhc2hYIJQzS6wgU0F1uc3tVIImqFZj2Z9uIKIFFqb6CJQYL91QZG5hbWVuanVtYmYgbWFuaWZlc3RqZXhjbHVzaW9uc4GiZXN0YXJ0GQFxZmxlbmd0aBkeBAAAAj5qdW1iAAAAJ2p1bWRjMmNsABEAEIAAAKoAOJtxA2MycGEuY2xhaW0udjIAAAACD2Nib3KlY2FsZ2ZzaGEyNTZpc2lnbmF0dXJleE1zZWxmI2p1bWJmPS9jMnBhL3VybjpjMnBhOjNhMjExMjQ3LTAwMTQtNGQ5Zi1hNGI5LTNiNDZlYWY1NDVlZi9jMnBhLnNpZ25hdHVyZWppbnN0YW5jZUlEeCx4bXA6aWlkOjg4NGExNTQwLTM3ODQtNGQ1YS05Zjc5LWQyOTgzY2YzYzJiYXJjcmVhdGVkX2Fzc2VydGlvbnODomN1cmx4LXNlbGYjanVtYmY9YzJwYS5hc3NlcnRpb25zL2MycGEuaW5ncmVkaWVudC52M2RoYXNoWCACoUIjHsOPwvBBJA0Xitw9Tiam5pUQc1XBm6o6yN0m36JjdXJseCpzZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmFjdGlvbnMudjJkaGFzaFgg+w6DJf55PSU9MwCjL0pStx6xVl9FSAIjCwGa/F+MXyiiY3VybHgpc2VsZiNqdW1iZj1jMnBhLmFzc2VydGlvbnMvYzJwYS5oYXNoLmRhdGFkaGFzaFgg8cSCquWXKZFvcM/9e3j1TVupYL7rDPmIozlo9sH49ZJ0Y2xhaW1fZ2VuZXJhdG9yX2luZm+jZG5hbWVvQW50aHJvcGljIEZpbGVzZ3ZlcnNpb25lMS4wLjBrc3BlY1ZlcnNpb25lMi40LjAAABA4anVtYgAAAChqdW1kYzJjcwARABCAAACqADibcQNjMnBhLnNpZ25hdHVyZQAAABAIY2JvctKEWQISogEmGCFZAgowggIGMIIBjaADAgECAhRA5aAK7sI50L64g/oGQgU9Z1UTADAKBggqhkjOPQQDAzBJMRcwFQYDVQQKEw5BbnRocm9waWMsIFBCQzEuMCwGA1UEAxMlQW50aHJvcGljIENvbnRlbnQgQ3JlZGVudGlhbHMgUm9vdCBDQTAeFw0yNjA4MDcxODQzNTZaFw0yODA4MDYxOTQzNTZaMEQxFzAVBgNVBAoTDkFudGhyb3BpYywgUEJDMSkwJwYDVQQDEyBBbnRocm9waWMgQ2xhdWRlIENvbnRlbnQgU2lnbmluZzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJh6CmvLUBgFFNU0vUKlOVtE6djd17L5SuwX0LemFisBM3dkd/3cyjxFA3Qo5S46fX0/ihY0VZ7mfb9KF703t5OjWDBWMA4GA1UdDwEB/wQEAwIHgDAVBgNVHSUEDjAMBgorBgEEAYPoXgIBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUzlHiBIFOZFsj+OPEz5o+nMHXXMIwCgYIKoZIzj0EAwMDZwAwZAIwMXMdFJ4BetLLVY7ORuE9noqbbAZOZn/aArXyTwFAZfKrPzxF2vPoJNf1+UCdg1XGAjBwX1zd9WGqYkqmL5SFqw1QySjr1zJfpJM9+1rdDwSPLMOPOjKuiXjoU/pUUeG9RwmhY3BhZFkNngAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPZYQB/FlpcRptXgN4aFAdWYwFEZY8gjr/XXO8uIS/Es+FfqlVV9wyfZoMsj0Kiz2ghZgHvVrrbD9siGfgp2JUpT2TQ=
image/svg+xml
Matplotlib v3.11.2, https://matplotlib.org/
1
5
10
15
20
25
principal component
0
2
4
6
variance (eigenvalue)
elbow
Why Use PCA
When using PCA , we can often get equivalent accuracy with lower training time* and memory requirements !
In many datasets, not all features are equally useful !
6. Project Our Data into the PCA Space
The final step is to use the eigenvalues and eigenvectors of our selected principal components to transform our original data into our new "PC space" .
z = x\,V_1 = 0.7071(1.05) + 0.7071(1.45) = 1.77
6. Project Our Data into the PCA Space
PC1
X1
X2
Using PCA in Code
Like always it's easy.
import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
# math and physics scores for our six students
X = np.array([[10 , 6 ], [11 , 4 ], [8 , 5 ],
[3 , 3 ], [1 , 3.2 ], [2 , 1 ]])
Z = StandardScaler().fit_transform(X) # step 1
pca = PCA(n_components=1 ).fit(Z) # steps 2 to 5
print (pca.components_.round(4 ))
print (pca.explained_variance_ratio_.round(4 ))
print (pca.transform(Z).round(2 ).ravel()) # step 6
Output
[[0.7071 0.7071 ]]
[0.8792 ]
[ 1.77 1.05 0.96 -0.82 -1.08 -1.89 ]
Input Space vs. Feature Space
So far we've worked primarily in input space . Input space is the space of our original measured features .
However PCA finds us new dimensions. When we plot our original points along these new dimensions they could be considered to now be in feature/latent space .
Much of Deep Learning is about transforming our input into latent spaces that we learn .
Eigenfaces: Turk and Pentland (1991)
Dimensionality Reduction in the Eye
Each eye has well over 100 million light-sensing cells, but only about 1 million fibers in the optic nerve, so the retina has to compress what we see before it goes to the brain.
Color is one example. Our three kinds of cone overlap a lot, so their signals are highly correlated, and the retina recombines them into light–dark, red–green and blue–yellow channels. Buchsbaum and Gottschalk (1983 ) showed these are close to the principal components of the cone signals.
Oja (1982 ) showed a single neuron with a Hebbian learning rule ends up finding the first principal component of its inputs.
Review
Complete the review: Eigenvectors identify the directions of the principal components , eigenvalues quantify how much variance each component holds . A scree plot helps us choose how many components to keep .
Matching: Principal Component Analysis
E Principal component A. Projecting data from a high-dimensional space to a lower-dimensional one
D Eigenvalue B. The space of our original measured features
C Eigenvector C. The direction along which a principal component points
G Covariance D. The amount of variance contained by each principal component
J Variance E. A new axis along which the data varies the most
I Scree plot F. As dimensions grow, the data needed to generalize grows exponentially
A Dimensionality reduction G. How two features vary together, their "informational" overlap
F Curse of dimensionality H. The new dimensions found by a transformation such as PCA
B Input space I. Variance explained per component, used to choose how many to keep
H Feature (latent) space J. How spread out the data is along an axis
Practice Problems
Q20. Draw (approximately) the two principal components on the plot and label them.
image/svg+xml
Matplotlib v3.11.2, https://matplotlib.org/
0
2
4
6
8
10
12
Math (x1)
0
1
2
3
4
5
6
7
Physics (x2)
1
2
3
4
5
6
PC1 runs up and to the right through the middle of the cloud (along the direction both scores rise together), PC2 crosses it at a right angle through the average point (5.83, 3.70).
Practice Problems, Continued
Q21. Why are the principal components always orthogonal to each other? What do we do with the covariance matrix to ensure this happens?They're the eigenvectors of the covariance matrix, and a symmetric matrix (Cov(X1, X2) = Cov(X2, X1)) always has perpendicular eigenvectors. Each new component is the direction of most remaining variance at right angles to the ones before it, so no variance is counted twice and the new features have zero covariance.
Q22. What is dimensionality reduction, and when may we use it?Projecting data from a high-dimensional space into a lower-dimensional one while keeping as much of what separates the data as possible. Use it when there are many (often redundant) features: to train faster with less memory, to fight the curse of dimensionality, or to plot the data in 2D or 3D.
Practice Problems, Continued
Q23. What are the roles of eigenvalues and eigenvectors in PCA?The eigenvectors of the covariance matrix are the directions of the principal components (the new axes). The eigenvalues are the variance along each one, so sorting by eigenvalue ranks the components and tells us how many to keep.
Q24. If I have a dataset with 5 input features and then transform the 5 features into 5 principal components, did I lose any information in my dataset? Why or why not? If I only select and use the top 2 principal components did I lose any information?Keeping all 5 loses nothing: it's only a rotation, and the total variance is unchanged (rotate back and you get the data exactly). Keeping the top 2 loses the variance on the other 3 components, usually a small share.
Practice Problems, Continued
Q48. When we use PCA, we often get more principal components back than we'd realistically use. Given the scree plot below, indicate how many principal components you'd choose to use. Why don't we use every component?
image/svg+xml
Matplotlib v3.11.2, https://matplotlib.org/
1
2
3
4
5
6
7
8
principal component
0
1
2
3
4
5
6
eigenvalue
Use the elbow: the eigenvalues drop sharply from component 1 (about 5.5) to component 2 (about 1.6), then level off, so keep 1 or 2. The later components explain very little variance (mostly noise), and keeping them defeats the purpose of reducing the dimensions.
Practice Problems, Continued
Q56. In Homework03, why did we use SVD during PCA instead of the covariance matrix?The SVD of the centered data gives the principal components directly (the right singular vectors) without ever forming the covariance matrix. Forming XᵀX is more expensive with many features and less numerically stable, so the SVD is faster and more accurate. The variances come from the singular values, σ²/(n − 1).