Score function, Fisher Information, Cramer-Rao Lower Bound and Efficiency¶

Score Function¶

Definition

The score function is the derivative of the log-likelihood \(\ell'(\theta)\).

Fisher Information¶

The fisher information, denoted as \(I(\theta)\) is the variance of the score.

\[ I(\theta) = -\mathbb{E}[\ell''(\theta)] = \mathbb{E}[\ell'(\theta)^{2}]. \]

Derivation of Fisher

Start with the fact that any density integrates to 1

\[ 1 = \int p(x | \theta)dx. \]

Take the derivative of both sides

\[ 0 = \frac{d}{dx}\int p(x|\theta)dx = \int p'(x|\theta)dx = \int \frac{p'(x|\theta)}{p(x|\theta)} p(x|\theta)dx. \]

Notice that \(\frac{d}{dx}\ell(\theta) = \frac{d}{dx}\log p(x|\theta) = \frac{p'(x|\theta)}{p(x|\theta)}\), so we have that

\[ 0 = \int l'(\theta)p(x|\theta)dx. \]

We'll take the derivative of both sides again

\[ \begin{align} 0 &= \int \ell''(\theta)p(x|\theta)dx + \int l'(\theta) p'(x|\theta)dx \\ &= \int \ell''(\theta)p(x|\theta)dx + \int l'(\theta)^{2} p(x|\theta)dx \\ \end{align} \]

\[ \therefore -\int \ell''(\theta)p(x|\theta)dx =\int l'(\theta)^{2} p(x|\theta)dx \]

Which is what we set as our fisher info.

Intuition of Fisher Information

On average the score is zero - this is the slope of the likelihood function at the true value \(\theta\). The FI measures the average curvature of the likelihood function at the true parameter value. If this function is really curved at \(\theta\), then it is easier to distinguish the maximum of the likelihood, thus there is more "information".

Cramer-Rao Lower Bound¶

Mean Squared Error

\(MSE = \mathbb{E}_{\theta}[(\hat{\theta}(X_{1}, \dots, X_{n}) - \theta)^{2}]\).

The MSE can be decomposed into variance + squared-bias. We want estimators that are optimal in the sense of minimizing the MSE. If we have an unbiased estimator, minimizing MSE is equivalent to minimizing variance.

Thus, we have a term called the Minimum Variance Unbiased Estimator, which is the best estimator in terms of the MSE within the class of unbiased estimators.

Cramer-Rao Lower Bound

If I have an unbiased estimator \(\hat{\theta}(X_{1}, \dots, X_{n})\) for \(g(\theta)\). That is, \(\mathbb{E}[\hat{\theta}] = g(\theta)\), then

\[ Var(\hat{\theta}) \leq \frac{g'(\theta)^{2}}{nI(\theta)}, \]

where \(I(\theta)\) is the Fisher Info for a single observation.

An unbiased estimator is said to be efficient if it attains the CR lower bound.

References¶

Maximum Likelihood Estimation