Principal Components Analysis

High correlations between some pairs of variables.

We're used to doing regression analyses

• treating one variable as the predictor (\( X= \) Petal Length)
• one variable as the outcome (\( Y= \) Petal Width)

\[ \widehat{Y} = -0.402 + 0.424X \]

View this as a cloud of points - try to find transformed versions of the data which describe its most important properties: its location and its spread.

The mean location is \[ \bar{X}_1 = 4.906 \qquad \bar{X}_2 = 1.676 \]

Transformed variables \[ \begin{aligned} Y_1 &= 0.9098 (X_1-4.906) + 0.4151 (X_2-1.676)\\ Y_2 &= 0.4151 (X_1-4.906) - 0.9098 (X_2-1.676) \end{aligned} \] We can calculate \( (Y_1,Y_2) \) from any observation \( (X_1,X_2) \)

We can also reconstruct \( (X_1,X_2) \) from \( (Y_1,Y_2) \): \[ \begin{aligned} X_1 &= 4.906 + 0.9098 Y_1 + 0.4151 Y_2\\ X_2 &= 1.676 + 0.4515 Y_1 - 0.9098 Y_2 \end{aligned} \]

Here \( Y_1 \) is more informative than \( Y_2 \):

• It explains the greatest amount of variation in the relationship between \( X_1 \) and \( X_2 \)
• If we had to choose which one of the two we'd rather know: we'd choose \( Y_1 \)
• \( Y_1 \) alone allows a good guess at both \( X_1 \) and \( X_2 \)

We have a matrix of data \( X \):

• \( n \) rows (e.g. \( n=100 \) irises)
• \( p \) columns (e.g. \( p=4 \) variables)

In Principal Components Analysis we find transformed versions of the data

• Each Principal Component (PC) is a linear combination of the data \[ \begin{aligned} Y_1 &= a_{10} + a_{11} X_1 + a_{12} X_2 + \ldots + a_{1p} X_p\\ Y_2 &= a_{20} + a_{21} X_1 + a_{22} X_2 + \ldots + a_{2p} X_p\\ \ldots &\\ Y_p &= a_{p0} + a_{p1} X_1 + a_{p2} X_2 + \ldots + a_{pp} X_p\\ \end{aligned} \] The numbers \( a_{10} \), \( a_{11} \) etc can be calculated from the data.

• Each Principal Component (PC) explains some of the variation in the data
• The PCs are ordered so that the first explains the most variation, and the last explains the least
• There are \( p \) PCs (if \( n\geq p \))
• Together all of the PCs reconstruct the data exactly
• If the first few PCs explain most of the variance, then we can make an approximation the data using just those PCs.

Scree Plot: shows the variance explained by each PC

• 1 PC explains most of the variance
• 3 PCs explain almost all of the variance

• 40 people with 10 images each - total of 400 images
• Each image is 112 \( \times \) 92 pixels (10304 pixels in all)
• Each image is grey scale (pixel values \( \in [0,1] \))

• We can run a Principal Components Analysis on these data
• Data set is \( X \): 400 rows \( \times \) 10304 columns
• There are only \( n=400 \) PCs (since \( n\lt p \) )
• The “average location” is the mean of all the images

Need at 50 PCs to explain 80% of the variance.

• Storing a library of PCs means any face can be represented by just 50 weights
• \( \Rightarrow \) compression of the data

• PCA summarises the key features of multivariate data \( X \)
• It allows a reduced representation of each data item:
  • store only a library of PCs
  • and for each image a set of weights of of each PC

Next Lecture

• The theory behind PCA
• How to carry out a PCA in practice
• Variations on PCA - centering and scaling
• Intepretation of PCs