Generative Models

Definition

A class of statistical models to generate new data points that resembles the training data. These models try to learn the underlying distribution of the input data (dataset \(\mathcal{D}\)), to produce synthetic data that is similar to \(\mathcal{D}\). Formally,

\[ \mathbf{x} \sim p_{\theta} \]

which means that we sample a new datapoint from the distribution that our generative model predicts given the parameters \(\theta\).

Generative models are designed around two criteria:

1. Facilitating sampling (how easy is it to sample from this distribution)
2. Facilitating learning the model

For a generative model to be useful, it must be efficient to sample from (using a generative model means sampling from it).

Example

There are simple generative models, for example MLE, where our goal is to model the distribution \(p(\mathbf{x})\) using the parameters \(\mathbf{\theta}\), assuming a dataset \(\cal D\). The objective of MLE is to find the parameters \(\theta\) of the distribution that make the data most likely. This is formalized as:

\[ \mathbf{MLE} : \theta_{mle} = \arg\!\max_{\mathbf{\theta}}p(\mathcal{D} | \mathbf{\theta}) = \arg\!\min_{\mathbf{\theta}} - \sum_{i=1}^n - \ln p(\mathbf{x_{i}| \mathbf{\theta}}) \]

For simple distributions (Gaussian), this reduces to an easy formula with easy to compute gradients, but once we look at complex distributions, learning becomes more complicated.

References

Simple Generative Models, Mixture Models Evaluating Generative Models