Estimating Uncertainty for Nonlinear Models¶
Preface¶
We want to extend uncertainty estimation for nonlinear models. That is, we assume that
where the true underlying function \(f^*\) can be a nonlinear function, and we assume that \(\epsilon \sim \mathcal{N}(0, \sigma^2)\) for some unknown variance \(\sigma^2 > 0\).
Definition¶
The simplest way to extend to the nonlinear setting is to first map \(\mathbf{x}\) to a new set of features \(\boldsymbol{\phi}(\mathbf{x})\) and then do Bayesian Linear Regression on \(\boldsymbol{\phi}(\mathbf{x})\).
We can simply compute:
- \(\boldsymbol{\Sigma}_{n} = \sigma_{y}^2\boldsymbol{\Lambda}^{-1}\), using \(\boldsymbol{\Lambda} = \sum_{i=1}^{n} \boldsymbol{\phi}_{i}^{\top}\boldsymbol{\phi}_{i}+\lambda \mathbf{I}\) and,
- \(\boldsymbol{\mu}_{n} = \boldsymbol{\Lambda}^{-1}\sum_{i=1}^n \boldsymbol{\phi}_{i}^{\top}y_{i}\)
Where we define \(\boldsymbol{\phi}_{i} = \boldsymbol{\phi}(\mathbf{x}_{i})\).
Then we have that:
Kernel Trick¶
Another way we can extend to the nonlinear setting is by using the Kernel Trick: Kernel Trick in Machine Learning.
We use the Kernel Trick in Bayesian Linear Regression: Kernelizing Bayesian Linear Regression (Gaussian Process Regression)
Uncertainty Estimation for Neural Networks using Ensembles¶
A different approach with Neural Networks: Uncertainty Estimation for Neural Networks using Ensembles