Connection of Generative Models to PPCA

Preface

A refresher on Probability Principal Component Analysis: Probabilistic Principal Component Analysis (PPCA)

Definition

Remark: we want to have a generative model that is easy is sample from, and learns a good approximation of the true underlying distribution.

Generating samples from PPCA is very straightforward, with the procedure being:

1. For \(h_{1}, h_{2}, \dots h_{p}\), sample \(h_{i} \sim \mathcal{N}(0,1)\)
2. Obtain latent vector \(\mathbf{h} = [h_{1}, h_{2}, \dots h_{p}]\)
3. Sample \(\mathbf{x} \sim \mathcal{N}(\boldsymbol{\mu} = \mathbf{hD}, \sigma^2 \mathbf{I})\)

However, we are still limited in the types of distributions we can represent over our features \(\mathbf{x}\). This is because our assumptions are too simple; we are assuming Gaussian's distributions, and in particular for \(p(\mathbf{x}| \mathbf{h, D})= \mathcal{N}(\boldsymbol{\mu} = \mathbf{hD}, \sigma^2 \mathbf{I})\), we assume the mean is linear in \(\mathbf{h}\).

But, we can assume the mean to be a more complex function \(f(\mathbf{h})\). This gives allows us to interpret \(p(\mathbf{x|h, D})\) differently: \(\mathbf{h}\) are the inputs, \(\mathbf{D}\) are the parameters of the learned function \(f\), and \(\mathbf{x}\) are the outputs.

This is where Variational Autoencoder (VAE)s come into play.

References

Generative Models using Data Representations Generative Models