MGFs and the Central Limit Theorem

Prerequisite Knowledge

Recall

A moment generating function (MFG) of a random variable \(X\) is defined as \(M_{X}(t) = E(\exp(tX))\). They are useful for determining the distribution of simple functions of random variables.

If \(X\) and \(Y\) have the same MGF \(M_{X}(t) = M_{Y}(t)\) for all \(t\) then \(X\) and \(Y\) have the same distribution.

If we have a sequence of independent random variables \(X_{i}\), then we have a nice property regarding their sum:

\[ \mathbb{E}\left[ \exp\left( t \cdot \sum_{i=1}^{n} X_{i}\right) \right] = \prod_{i=1}^{n} \mathbb{E}[\exp(tX_{i})] = \prod_{i=1}^{n} M_{X_{i}}(t) \]

---

MGFs and Convergence in Distribution

Theorem

If there exists \(\epsilon > 0\) such that \(M_{X_{n}}(t) \to M_{X}(t)\) for all \(|t| \leq \epsilon\) then \(X_{n}\) converges in distribution to \(X\).

This is one primary way to prove convergence in distribution results.

Example

If \(X_{n} \sim \text{Binomial}(\mu /n, n)\) then this converges to \(\text{Poisson}(\mu)\).

Proof: Show MGF for \(X_{n}\).

\[ \begin{align} M_{X_{n}} &= (1-\mu / n + \mu e^{t} / n)^{n} \\ &= (1 + \mu (e^{t} -1) / n)^{n} \to \exp(\mu(e^{t} - 1)) \end{align} \]

Show MGF for \(X\) (Poisson).

\[ \begin{aligned} M_X(t) &= E[e^{tX}] \\[6pt] &= \sum_{x=0}^{\infty} e^{tx} \cdot \frac{e^{-\mu}\,\mu^x}{x!} \\[6pt] &= e^{-\mu} \sum_{x=0}^{\infty} \frac{e^{tx}\,\mu^x}{x!} \\[6pt] &= e^{-\mu} \sum_{x=0}^{\infty} \frac{(e^t)^x\,\mu^x}{x!} \\[6pt] &= e^{-\mu} \sum_{x=0}^{\infty} \frac{(e^t \mu)^x}{x!} \\[6pt] &= e^{-\mu} \cdot e^{e^t \mu} \qquad \left(\text{since } \sum_{x=0}^{\infty} \frac{y^x}{x!} = e^y \text{ with } y = e^t\mu\right) \\[6pt] &= \exp\!\left(e^t\mu - \mu\right) \\[6pt] &= \exp\!\left(\mu(e^t - 1)\right) \end{aligned} \]

---

Central Limit Theorem

Definition

If \(X_{i}\) are i.i.d. with finite variance \(\sigma^{2}\) and mean \(\mu\) then

\[ \sqrt{ n }(\bar{X}_{n} - \mu) \xrightarrow d \mathcal N(0, \sigma^{2}). \]

Example

If \(Z\) is a standard normal, then \(Z^{2}\) has a \(\chi^{2}(1)\) distribution. This distribution has mean 1 and variance 2.

The distribution of \(\sum_{i=1}^{n}Z^{2} \sim \chi^{2}(n)\) with mean \(n\) and variance \(2n\).

Then if \(C_{n}\) is \(\chi^{2}(n)\) then

\[ \begin{align} \sqrt{ n } \left( \frac{1}{n}C_{n} - 1 \right) \sim \mathcal N(0, 2) \\ \sqrt{ n } \left( \frac{1}{n}C_{n} - 1 \right) / \sqrt{ 2 } \sim \mathcal N(0, 1) \\ \end{align} \]