GLM Preface

Definition

Revisiting OLS (ordinary least squares):

We know that we have a dataset with \(n\) samples: \(D = \{ (\mathbf{x_{i}}, y_{i}) \}_{i=1}^n\), where we assumed \(D\) was drawn from the distribution:

\[ p(\mathbf{x}, y) = p(\mathbf{x}) \cdot p(y|\mathbf{x}). \]

We also assumed that the underlying relationship between the features and targets were linear:

\[ Y = \sum_{j=1}^dX_{j}\mathbf{\omega}_{j} + \epsilon \]

We want to generalize this formulation; thus, we separate the underlying linear relationship between the features and targets from the fact that Y was normally distributed. So we can write:

\[ \begin{align} &1. \ p(y|\mathbf{x}) = \cal{N}(\mu(\mathbf{x}), \sigma^2) \\ &2. \ \mathbb{E}[Y|\mathbf{x}] = \mathbf{x \omega} \end{align} \]

1 corresponds the fact that Y is normally distributed with a mean dependent on the feature vector. 2 corresponds to the relationship between the features and target, formalized with \(\mathbf{x \omega}\).

By separating these two things, we can have Y be drawn from different distributions, and the underlying relationship between features and target need not be linear.

Example

Suppose we want to solve a problem, where it's more reasonable to say that the target variabe \(y\) was drawn from a Poisson distribution.

Then we can easily say:

\[ p(y|\mathbf{x} = \text{Poisson}(\mathbf{\lambda}) = \frac{\lambda^y \exp(-\lambda)}{y!}, \]

where \(\lambda\) is the mean of the distribution: \(\mathbb{E}[Y|\mathbf{x}] = \lambda\).

\(\mathbb{E}[Y|\mathbf{x}] = \lambda\) formalizes the relationship between the features and target, but we need to rewrite in terms of the feature \(\mathbf{x}\). Since \(\lambda > 0\) we cannot model \(\lambda = \mathbf{xw} \in \mathbb{R}\), since it can go negative.

However, we can say: \(\lambda = \exp(\mathbf{xw})\). In our example \(g = \exp()\), where \(g\) is the exponential transfer.

Then our formulation will be:

\[ \begin{align} \\ &1. \ p(y|\mathbf{x}) = \text{Poisson}(\lambda(\mathbf{x})) = \frac{\exp(\mathbf{xw}y)\exp(-\exp(\mathbf{xw}))}{y!} \\ &2. \ \mathbb{E}[Y | \mathbf{x}] = \exp(\mathbf{xw}) \end{align} \]

Note that when using MLE, we can only use gradient descent since the MLE for this distribution does not have a closed form solution.

References

Ordinary Least Squares (OLS) Generalized Linear Models (GLM)