Ordinary Least Squares (OLS)¶
Definition¶
The formulation of solving a linear regression problem.
The maximum likelihood formulation for linear regression is:
\[
\mathbf{w_{MLE}} = \arg\!\min_{\mathbf{w} \in \mathbb{R}^d} \sum_{i=1}^n(\mathbf{x_{i}w}-y_{i})^2
\]
The matrix version of the MLE formulation is written as:
\[
\mathbf{w_{MLE}} = \arg\!\min_{\mathbf{w} \in \mathbb{R}^d} ||\mathbf{Xw-y}||_{2}^2
\]
Taking the gradient of \(\mathbf{w_{MLE}}\) (so we can set it to zero for a minima):
\[
\mathbf{w} = \left( \sum_{i=1}^n \mathbf{x_{i}^\top}x_{i}\right)^{-1} \sum_{i=1}^n \mathbf{x_{i}}^\top y_{i}
\]
In matrix form, this is equivalent to:
\[
\mathbf{w} = \mathbf{(X^\top X)^{-1}X^\top y}
\]
Thus we have our prediction:
\[
\hat{y} = \mathbf{Xw} = \mathbf{X(X^\top X)^{-1}X^\top y}
\]
Where \(\mathbf{y}\) is the line projected onto the column space of \(\mathbf{X}\). This means that our prediction \(\hat{\mathbf{y}}\) is the most accurate representation of \(\mathbf{y}\) given our current feature space.
Stability and the Bias-Variance of the OLS Solution¶
The OLS solution can be unstable, but we can use regularization to mitigate this issue. Bias-Variance of OLS