SGD has Implicit Regularization¶
Definition¶
SGD runs gradient descent with batches of data instead of the entire dataset. This results in stochastic noise. Stochastic noise has two key effects:
- Escaping Sharp Minima: Since we don't use full gradient descent, we're not traversing down the steepest slope everytime, rather we're moving a bit more randomly (stochastic noise). This allows us to escape sharp minima, which has poor generalization and less robust to small changes in input data.
- Prefers Flat Minima: SGD tends to converge to flatter minima, which are wider and more stable. Flat Minima are associated with better generalization because small changes in the weights do not significantly affect the loss, making the model more robust to the variation's of input data.
SGD follows a random trajectory due to the noise in the gradient estimates. This random trajectory biases SGD to certain solutions:
- Low Norm Solutions: SGD often converges to solutions with smaller weight norms, which is similar to ℓ2 Regularization (ie. weight decay).
- Simple Solutions: SGD tends to favour simple functions instead of finding overly complex functions. This helps prevent overfitting.
In the context of Overparametrized Neural Networks:
- SGD allows the model to find solutions that fit the data in a smooth/simple way, rather than having a highly oscillatory (overfitting) function.
- SGD often converges to solutions that use low-rank weight matrices, effectively becoming a form of dimensionality reduction.