Simple Generative Models, Mixture Models¶
Preface¶
We discuss how to use and learn mixture models. Assume that each mixture component has the parameters \(\boldsymbol{\theta}_{k}\), giving
Mixture models provide enough complexity to be used as generative models , to model realistic distributions, and make it useful to generate potential outcomes using these learned distributions.
We discuss first how mixture models are used as generative models, then discuss how to learn them.
Using Mixture Models¶
Mixture models are easy to sample from, as long as it is easy to sample from each component \(x \sim p_{k}\).
We must sample \(k\) proportionally to probabilities \(p[k] = w_{k}\). To do this we discretize the interval [0, 1] into \(m\) buckets, with the first bucket of size \(w_{1}\), the second bucket of size \(w_{2}\) etc.
Then we can uniformly sample from this interval [0,1] and return its bucket.
Once we have component \(k\) selected, we sample \(x \sim p_{k}\). A common procedure to sample from \(p_{k}\) is if the inverse CDF exists, is to sample \(u \sim [0,1]\) uniformly and transform it using the inverse CDF. Thus we have:
This means we can sample from a uniform \(U\), and apply the transformation \(F^{-1}(U)\), so it is as if we sampled directly from \(X\).
Learning Mixture Models¶
To learn mixture models, we introduce auxiliary variables that correspond to the mixing component. For mixture models, we only need one auxiliary variable, \(k\), which corresponds to the \(k\)'th distribution of the mixture distribution.
We jointly optimize over the auxiliary variable and the original variable \(\boldsymbol{\theta}\), to obtain simpler iterative updates that converge quickly to a stationary point. This algorithm is called Expectation-Maximization (EM Algorithm).
References¶
Mixture Models Cumulative Distribution Function (CDF) Generative Models using Data Representations