Generalization Error

Definition

Given our sample (subset) of the total outcomes, we want our model to be able to accurately predict all possible outcomes. The accuracy of our predictions depends on the cost function:

\[ \mathcal{Y \times Y} \to [0,\infty) \]

Where \(\cal{Y \times Y}\) represents the total possible combinations of true and predicted values.

We want to find a function \(f\) such that our generalization error is minimal:

\[ GE(f) = \mathbb{E}[\text{cost}(f(X), Y)] \]

The generalization error formula is:

\[ \text{Generalization Error = } \text{Bias}^2 \text{+ Variance + Irreducible Error} \]

Specifically, for a given input \(x\) using the squared error loss, we have:

\[ \mathbb{E}\left[ (y - \hat{f}(x))^2 \right] = \underbrace{\left( \mathbb{E}[\hat{f}(x)] - f(x) \right)^2}_{\text{Bias}^2} + \underbrace{\mathbb{E}\left[ (\hat{f}(x) - \mathbb{E}[\hat{f}(x)])^2 \right]}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{Irreducible Error}} \]

Example

For regression, we use a mean squared error cost (which we aim to minimize the error), \((\hat{y}- y)^2\). Then it's corresponding generalization error is:

\[ GE(f) = \mathbb{E}[(f(X) - Y)^2] \]