Principal Component Analysis (PCA)¶
PCA Definition from CMPUT 340¶
Preface¶
Standard Deviation: The average distance a data point is from the mean.
Variance: Another measure of the spread of data.
Covariance: Covariance
Definition¶
The principal components are the orthogonal vectors which correspond to highest variance of the data.
When using PCA, the data must be standardized (ie. centering the mean of the data by zero, and scaling it by the standard deviation)
PCA looks at how different features contribute to the spread of the data.
PCA Definition from CMPUT 467¶
A standard dimensionality reduction technique where the input vector \(\mathbf{x} \in \mathbb{R}^{1\times d}\) is projected into a lower dimensional \(\mathbf{h} \in \mathbb{R}^{1\times p}\) spanned by the space of the principal components. The projection to lower dimensions has the property that it removes noise and keeps the most meaningful directions, thus helps speed up learning by reducing the number of features and promoting generaliztion by avoiding overfitting to noise
These principal components are the directions of maximal variance in the data.
To obtain these p principal components \(\mathbf{D} \in \mathbb{R}^{p \times d}\), the common solution technique is to obtain the singular value decomposition of the data matrix \(\mathbf{X = U\Sigma V^\top} \in \mathbb{R}^{n \times d}\) which gives:
Where \(\mathbf{\Sigma_{p}} \in \mathbb{R}^{p\times p}\) consists of the p largets singular values (in descending order) and \(\mathbf{U_{p}}\in \mathbb{R}^{n\times p}\) and \(\mathbf{V_{p}} \in \mathbb{R}^{p \times d}\) are the corresponding singular vectors. ie. \(\mathbf{U_{p}} = \mathbf{U}_{:1:p}\) (and the same for V)
PCA creates new features and the new \(\mathbf{H}\) is not a subset of the original \(\mathbf{X}\).
Note that PCA is equivalent to a linear autoencoder (Autoencoders 1)
References¶
Probabilistic Principal Component Analysis (PPCA) (this goes into Probabilistic PCA)