PCA (Principal Component Analysis)

PCA (Principal Component Analysis) transforms a set of correlated variables into a smaller number of uncorrelated variables, called principal components, that retain most of the variance.

Visualize the dimensionality reduction

In this 3D simulator, a cloud of points is projected onto a 2-dimensional plane (PC1 and PC2) while losing as little information as possible.

PCA visualizer

From three dimensions to two

Project to 2DAuto-rotate

Loading 3D scene…

Drag to rotate, scroll to zoom150 points

Full view

99.0% of variance

PC1 and PC2 capture almost all of the variation; PC3 is the residual direction.

Explained variance

PC1

83.0%

PC2

16.0%

PC3

1.0%

Covariance eigenvalues: 4.05 · 0.78 · 0.05

What to look for

• The components are the eigenvectors of the covariance matrix, ordered by eigenvalue.
• Projecting means keeping the coordinates along PC1 and PC2 and discarding PC3.
• If the last eigenvalue is small, little information is lost.

What it is for

1. Reducing noise: by keeping the components that explain most of the variance, we discard directions dominated by noise (like the red axis PC3).
2. Visualizing: it lets you project high-dimensional data down to 2 or 3 dimensions for plotting.
3. Gaining efficiency: models train faster with fewer variables.

Geometric intuition

The idea is to “rotate” the data space until we look at it from the most informative angle: the one that maximizes the spread of the points.

The algorithm step by step

1. Centering and standardization

So that variables with large scales (like “income”) do not dominate those with small scales (like “age”), we rescale each column:

This leaves each variable with zero mean and unit variance. If you only center (without dividing by ), you work with the original covariance matrix; if you standardize, with the correlation matrix.

2. Covariance matrix

We compute the matrix , which summarizes the linear relationships between all pairs of variables:

3. Eigenvalues and eigenvectors

The principal axes are the eigenvectors of . They are obtained by solving the characteristic equation:

• Eigenvalues : indicate importance; each one is the variance captured along its direction.
• Eigenvectors : indicate direction; they define the new axes (the lines in the simulator).

The fraction of variance explained by component is .

4. Projection

We choose the eigenvectors with the largest eigenvalues, collect them as columns of , and project:

holds the reduced data, ready for plotting or for training a model.

Caution