Proximal Methods¶
Definition¶
A way to set constraints on the weights. This is followed in two steps:
- Do gradient descent to update the weight (gradient step)
- Project the weight back into its constraints (projection step)
Then to formulate this idea:
\[
\min_{\mathbf{w} \in \mathbb{R^d}} c(\mathbf{w}) + r(\mathbf{w})
\]
\(c(\mathbf{w})\) is the typical cost function and assume to be everywhere differentiable. \(r(\mathbf{w})\) is a nonsmooth function where we encode constraints.
Check out Proximal Update.
It is feasible to solve the Proximal Operator only when:
- There is a closed-form solution
- There is an efficient iterative algorithm
Example¶
\(c(\mathbf{w})\) is the ordinary least squares and if we want the constraint set to be \(\cal{F} = [-1, 1]^d\), then \(r(\mathbf{w})\) is set as:
\[
r(\mathbf{w}) = \begin{cases}
0 & \text{if } \mathbf{ w} \in \cal{F} \\
\infty & \text{else}
\end{cases}
\]
Thus to attain a minimum, we must have \(\mathbf{w} \in \cal{F}\), otherwise our objective function is \(\infty\)