Multinomial Logistic Regression

Definition

MLR corresponds to predicting which "class" is most likely to occur.

We use the softmax function to calculate the probabilities of each class occuring (set by a vector with its indices being the classes), and then take its argmax as the prediction.

We then use maximum likelihood estimation to derive the negative log likelihood formula such that we can get a numerical measurement of how our predictions against the target values (the true y's).

However, we can pivot around a class by making it 0 and removing it from \(\mathbf{W} \in \mathbb{R}^{d \times m}\) where d is the number of features, and m is the number of classes. This means we have a new weight vector \(W\in \mathbb{R}^{d \times m-1}\) since we can get class m's probability by subtracting 1 with the rest of our probabilities.

This allows us to make our solution unique, and we learn fewer parameters.

Example

Predicting if it will either rain, snow, or flurry tomorrow. This would correspond to a one-hot encoded vector [1 0 0] if it was raining. We can only have one class be present, so it cannot rain and snow