Uncertainty Estimation for Neural Networks using Ensembles¶
Preface¶
Obtaining the posterior over parameters in neural networks is much more difficult than in Bayesian Linear Regression or Gaussian processes. This is because:
- Neural networks have millions of parameters, so when obtaining the posterior \(p(f_{\mathbf{W}} (\mathbf{x}) | \mathcal{D})\), where \(f_{\mathbf{W}}(\mathbf{x})\) is the NN, we get an intractable high-dimensional distribution.
- The posterior is some complex distribution (non-Gaussian) and is difficult to represent or approximate.
- Unlike conjugate priors in Bayesian Linear Regression (ie. Gaussian prior + Gaussian likelihood \(\implies\) Gaussian posterior), NN's lack any closed-form posteriors.
Sampling from \(p(f_{\mathbf{W}}(\mathbf{x}) | \mathcal{D})\) is also computationaly expensive and infeasible.
Instead we use ensembles.
Definition¶
Ensembles is a generic idea that can be layered onto any learning algorithm. The idea is to simulate different weights that we could have learned if we had seen different data.
Recall that our uncertainty comes from insufficient data, ie. we only have seen one dataset. If we had seen a different dataset, we would make a different prediction for \(\mathbf{x}\).
In an ideal scenario, we get \(m\) different datasets of size \(n\), estimate \(\mathbf{w}_{1}, \dots, \mathbf{w}_{m}\) from the datasets \(\mathcal{D}_{1}, \dots, \mathcal{D}_{m}\) and observe the spread in predictions. We could compute a mean \(\mu = \frac{1}{m}\sum_{i=1}^m \hat{y}_{i}\) and a \(95\%\) confidence interval \(v\) over \(\hat{y}_{1}, \dots, \hat{y}_{m}\) to get our uncertainty \([\mu - v,\ \mu + v]\).
But we don't get \(m\) different datasets to work with. So what can we do?
We treat our dataset \(\mathcal{D}\) as an empirical distribution and resample from it. This is called Bootstrap Resampling. We sample \(n\) data points from \(\mathcal{D}\), with replacement (after selecting an item, you put it back before the next selection) to obtain \(\mathcal{D}_{1}\), and then do this \(m\) times to get \(m\) datasets.
Then we train \(m\) neural networks independently on each dataset, and then we're done.
Downsides¶
A large ensemble will be needed to get an effective uncertainty estimate, which can be expensive.
An ensemble can often be an underestimate of uncertainty, especially if our dataset \(\mathcal{D}\) is small. (not enough unique info to work with).