Generalization Error for Logistic Regression¶
Definition¶
We want to minimize the 0-1 cost. We get the generalization error by:
\[
GE(f) = \mathbb{E}[\mathbb{I}(f(X) \neq Y)]
\]
Here we are getting the expectation of the true distribution.
We can then find an approximate p(y|x) such that when p(y|x) > 0.5, f(x) = 1 else 0.
We typically use the sigmoid function \(p(y|x) = \sigma(x^Tw)\). For this cost, we substitute the cross-entropy loss and approximate the true distribution \(p_{true}(y|x)\)