Probabilistic Principal Component Analysis (PPCA)¶

Definition¶

We assume that there is a Gaussian distribution over the latent variables, such that:

\[ p(\mathbf{h}) = \cal{N}(\mathbf{\mu = 0, I}) \]

Reflecting that the most latent variables will be around zero, with decreasing probability that they are large away from zero.

Additionally, we model that each \(\mathbf{h}\) can produce different \(\mathbf{x}\):

\[ p(\mathbf{x|hD}) = \mathcal{ N }(\mathbf{\mu=hD}, \sigma^2 \mathbf{I}) \]

for some \(\sigma > 0\).

Our goal is to find \(\mathbf{D}\) under these assumptions, where:

\[ p(\mathbf{x}| \mathbf{D}) = \int p(x|\mathbf{hD})p(\mathbf{h})d\mathbf{h} \]

We want to maximize the likelihood of the data, so find the dictionary that makes the data most likely under the Gaussian assumptions and we get:

\[ p(\mathbf{x|D}) = \mathcal{N}(\mathbf{\mu = 0, D^\top D + \sigma^2I}) \]

Result: we see that \(\mathbf{x}\) lives in a low dimensional space (given that the term \(\mathbf{D^\top D} \in \mathbb{R}^{d\times d}\) is low-rank since \(p<d\)). The covariance \(D^\top D + \sigma^2I\) implies that \(\mathbf{x}\) lives primarily in this space with small noise from \(\sigma^2\) making it slightly deviate from the plane.

We obtain the maximum likelihood estimator for \(\mathbf{D}\):

\[ \begin{align} \sigma^2_{MLE} &= \frac{1}{d-p}\sum_{j=p+1}^d\sigma^2_{j} \\ \mathbf{D}_{MLE} &= (\mathbf{\Sigma_{p}} - \sigma^2_{MLE}\mathbf{I})^{1/2}\mathbf{V}_{p}^\top \end{align} \]

This result is very similar to the Principal Component Analysis (PCA) solution, but the difference is that \(\mathbf{\Sigma_{p}}\) is shifted downwards by \(\sigma_{MLE}\). The other difference is whether we put the singular values in \(\mathbf{h \text{ or } D}\), but this difference is superficial and is convention.

If we want to get the most likely \(\mathbf{h}\) for a given input, we optimize:

\[ \begin{align} \arg\!\min_{\mathbf{h} \in \mathbb{R}^p} -\ln p(\mathbf{h|x, D}) &= \arg\!\min_{\mathbf{h} \in \mathbb{R}^p} - \ln p(\mathbf{x|h,D} ) - \ln p(\mathbf{h}) \\ &= \arg\!\min_{\mathbf{h} \in \mathbb{R}^p} ||\mathbf{x - hD||^2_{2}} + \sigma^2_{MLE}||\mathbf{h}||^2_{2} \end{align} \]

References¶

Finding the dictionary to get latent factors Latent Factors and Factor Analysis Principal Component Analysis (PCA)