ℓ1 Regularization¶
Definition¶
Using the linear regression objective as an example, we implement an l1 objective by introducing the Laplace Distribution as a prior on the weights:
\[
c(\mathbf{w}) = ||\mathbf{Xw-y}||^2_{2} + \lambda||\mathbf{w}||_{1}
\]
This objective function is also called the Lasso.
The subscripts 2 and 1 are the L2 Norm and L1 Norm.
Similar to ℓ2 Regularization, Lasso penalizes large weights, but also introduces sparsity.
This means that Lasso places a higher emphasis on zero-ing out weights, which can be intuitively seen by the graph of the Laplace Distribution. Zero-ing out the weights is essentially feature selection, where Lasso gets rid of "irrelevant features"
Lasso doesn't have a closed form, so we must use gradient descent: Gradient Descent with Lasso