Reparametrization Trick in VAE's (Machine Learning)¶
Definition¶
The issue is that we cannot differentiate \(\mathbf{h} \sim q(\mathbf{h|x}, \boldsymbol{\theta})\) because the distribution \(q\) itself doesn't have a closed form to differentiate with.
To combat this, we re-express this stochasticity independent of our parameters \(\mu, \sigma\) and creating an equation that is differentiable \(w\).\(r\).\(t\) \(\mu, \text{and }\sigma\).
\[
h_{i} = \mu_{i}(\mathbf{x}) + \sigma_{i}(\mathbf{x}) \cdot \epsilon, \ \ \ \ \ \ \ \ \ \ \ \epsilon \sim \mathcal{N}(0, 1)
\]
Now if we want to backpropagate with respect to the mean or std deviation, we are able to do so easily, since we changed our stochasticity from internal to external.