Evaluating Generative Models

Definition

We can evaluate the generalization error for generative models, but is defined differently from predictor models.

Once we define our metric (ie. KL Divergence), and the sample estimator, then we can use all the same strategies (test sets, cross validation).

A natural choice is the KL divergence between two different distributions: the learned generative model \(p(x | \theta)\), and the true generative model \(p_{\text{true}}(x)\):

\[ D_{KL}(p_{\text{true}}||p(x|\theta)) = \int_{\mathcal{X}}p_{\text{true}}(x)\ln \frac{p_{\text{true}}(x)}{p(x|\theta)}dx \]

It is shown that:

\[ D_{KL}(p_{\text{true}}||p(x|\theta)) = -\mathbb{E}[\ln p(X |\theta)] + \text{a term that depends only on } p_{\text{true}} \]

Therefore we can estimate the generalization error by estimating the expected log likelihood of data generated by \(p_{\text{true}}(x)\), under model \(p(x|\theta)\). We can do this using a sample average:

\[ \text{Perf}(\theta) := \frac{1}{m} \sum_{x \in \mathcal{D}_{test}}\ln(p|\theta) \]

\(\text{Perf}(\theta)\) is how well the model generalizes to the test data, so we want higher Perf.

Thus, if we want to minimize generalization error:

\[ GE(\theta) = -\frac{1}{m}\sum_{x \in \mathcal{D}_{test}}\ln(p|\theta) \]

References

Kullback-Leibler Divergence (KL)