Natural Exponential Family

Definition

A class of probability distributions with the following form:

\[ p(y|\theta) = \exp(\theta y-a(\theta) +b(y)) \]

where, \(\theta \in \mathbb{R}\) is the parameter of the distribution. \(a: \mathbb{R} \rightarrow \mathbb{R}\) is a log normalizer function. \(b: \mathbb{R} \rightarrow \mathbb{R}\) is a function of y. Since it is not a function of \(\theta\), it is usually ignored during optimization

The Gaussian, Gamma, Poisson and binomial distributions are part of the natural exponential family.

\(a(\theta)\) ensures that we have a valid pmf/pdf (ie. the integral or sum equal 1), and the derivative of \(a(\theta)\) is the transfer function

Example

The Poisson distribution can be expressed as:

\[ p(y|\lambda) = \exp(y\log\lambda - \lambda -\log y!) \]

$\theta = \log\lambda$
$a(\theta) = \exp(\theta)$
$b(y) = -\log y!$