Convergence in Probability¶

Prerequisite Knowledge¶

Recall

A sequence of scalars converges or has a limit when $\lim_{ n\to \infty }x_{n} = y$.

i.e. $\lim_{ n \to \infty } x_{n} = 0$, for $x_{n}= \frac{1}{n}$.

We want to generalize this idea of convergence to random variables $X_{n}$. Roughly speaking, we can think of convergence as the pdf (or pmf) of one RV converging to the pdf/pmf of another.

Definition

A probability distribution $P$ refers to a function that takes in a set or event $A$ and returns the probability $0 \leq P(A) \leq 1$ of $A$ occurring. The distribution $P$ of a random variable $X$ is defined as $P_{X}(A) := \text{Pr}(X \in A)$

$P_{X}(A)$ - "The probability that $X$ takes a value of $A$".

$P$ is the most general object since not every distribution has a pmf or pdf (an example is the Cantor distribution).

Note:

$\text{Pr}$ and $P$ are not the same.
$\text{Pr}$ takes in an event sampled from a sample space $\Omega$.
$P$ is a function that takes in a real number.

Convergence in Probability Definition¶

Definition

A sequence of random variables $X_{n}$ 'converges in probability' to a random variable $Y$ if for all $\epsilon > 0$ we have

\[ \lim_{ n \to \infty } \text{Pr}(|X_{n} - Y| > \epsilon) = 0. \]

We denote this convergence by $X_{n} \xrightarrow{p} Y$.

Convergence in probability provides a bound ($\epsilon$) of how much one RV differs from another. We can think of it like one random variable gradually "mimicking" another over time.

Note that the "strictness" in the inequality does not matter (i.e. $|X_{n} - Y| > \epsilon$ or $|X_{n} - Y| \geq \epsilon$).

Example

Suppose I make a friend in university and every year we fill out a questionnaire. Over the course of four years, my response $X_{n}$ converges to his $Y$, assuming that this friend doesn't change his preferences over time.

Convergence in probability is often used when $Y$ is a fixed constant. The classic example is the weak law of large numbers, which says that:

If $X_{1}, \dots, X_{n}$ (where $n$ denotes sample size) are i.i.d with $\mathbb{E}[X_{i}]= \mu$ and $Var(X_{i}) = \sigma^{2} < \infty$, then the sequence $S_{n} := \frac{1}{n}\sum_{i=1}^{n}X_{i}$ converges in probability to $\mu$.

Using Convergence in Probability to prove WLLN¶

In order to prove the weak law of large numbers (WLLN), we require two tools:

Proving (and using) Markov's Inequality
Proving (and using) Chebyshev's Inequality

Markov's Inequality Proof¶

Proof

I want to show that if $X \geq 0$ is a nonnegative RV, then $P(X>\epsilon) \leq \frac{\mathbb{E}[X]}{\epsilon}$. (This is Markov's Inequality)

I'll first define an indicator function for an event $A$

\[ I(A) = \begin{cases} 1 \ \text{if $A$ occurs} \\ 0 \ \text{else} \end{cases}, \ \ \text{where $A = (\epsilon, \infty)$} \]

Then I'll use the fact $P(X \in A) = \mathbb{E}[I(X \in A)]$. A quick proof for this:

\[ P(X \in A) = P(X > \epsilon) = \int_{\epsilon}^{\infty} p(x)dx = \int_{0}^{\infty}I(X \in A)p(x)dx = \mathbb{E}[I(X \in A)] \ \ \blacksquare \]

Next, note that

\[ I(X > \epsilon) \leq \frac{X}{\epsilon}I(X > \epsilon) \leq \frac{X}{\epsilon}. \]

To show that this inequality holds true, I'll split this into two cases. 1) $X > \epsilon$, and 2) $X\leq \epsilon$.

Case 1): when $X>\epsilon$, its trivial to see that $\frac{X}{\epsilon} > 1$ and so will always be greater than the indicator function.
Case 2): when $X\leq \epsilon$ all indicator functions equal $0$, so the statement holds.

Thus:

\[ P(X > \epsilon) = \mathbb{E}[I(X > \epsilon)] \leq \mathbb{E}\left[ \frac{X}{\epsilon}I(X>\epsilon) \right] \leq \mathbb{E}\left[ \frac{X}{\epsilon} \right] = \frac{\mathbb{E}[X]}{\epsilon} \ \ \blacksquare \]

A small remark: the reason the inequality $I(X > \epsilon) \leq \frac{X}{\epsilon}I(X > \epsilon) \leq \frac{X}{\epsilon}$ holds upon adding expectation is due to the monotonicity of expectation.

Chebyshev's Inequality Proof¶

Proof

I want to show that given $\mathbb{E}[X] = \mu$ and $Var(X) = \sigma^{2} < \infty$, then

\[ P(|X-\mu| > \epsilon) \leq \frac{\sigma^{2}}{\epsilon^{2}}. \]

I can simply use the just-proven Markov's Inequality:

\[ P(|X-\mu| > \epsilon) = P(|X-\mu|^{2} > \epsilon^{2})\leq \frac{\mathbb{E}[|X-\mu|^{2}]}{\epsilon^2} = \frac{\sigma^{2}}{\epsilon^{2}} \ \ \blacksquare \]

WLLN Proof¶

Proof

I want to show that $\frac{1}{n}\sum_{i=1}^{n}X_{i} \xrightarrow{p} \mu$.

Lets look at $P\left( \left|\frac{1}{n}\sum X_{i} - \mu \right| > \epsilon\right)$ for $\epsilon >0$.

I want to apply Chebyshev's Inequality, so lets check the conditions.

$\mathbb{E}\left[ \frac{1}{n} \sum X_{i}\right] = \frac{1}{n}\sum \mathbb{E}[X_{i}] = \frac{1}{n} n \mu = \mu$
$Var\left( \frac{1}{n} \sum X_{i} \right) = \frac{1}{n^{2}} \sum Var(X_{i}) = \frac{1}{n^2} n \sigma^{2} = \frac{\sigma^{2}}{n}$

The conditions hold, so lets apply Chebyshev's Inequality.

\[ P\left( \left|\frac{1}{n}\sum X_{i} - \mu \right| > \epsilon\right) \leq \frac{\sigma^{2} / n}{\epsilon^{2}} \]

Then apply the limit.

\[ \lim_{ n \to \infty } P\left( \left|\frac{1}{n}\sum X_{i} - \mu \right| > \epsilon\right) = 0. \]

Thus, $\frac{1}{n}\sum_{i=1}^{n}X_{i} \xrightarrow p \mu \ \ \blacksquare$.

From this proof we can make the following observation.

Remark

If we have a sequence of random variables $X_{n}$ with the following properties:

$\mathbb{E}[X_{n}] = \mu$
$Var(X_{n}) = \sigma^{2}(n) \to 0$ *

Then $X_{n}$ also converges in probability to $\mu$.

*$\sigma^{2}(n)$ denotes the variance of $X_{n}$ as a function of $n$ (e.g. $\frac{\sigma^{2}}{n}$).

Properties of Convergence in Probability¶

Properties

Let $X_{n}, Y_{n}$ converge in probability to $X,Y$, let $a,b \in \mathbb{R}$ be scalars and let $g$ be a continuous function.

$X_{n} + Y_{n} \xrightarrow p X + Y$
$X_{n}Y_{n} \xrightarrow p XY$
If $X_{n} \xrightarrow p a$ then $g(X_{n}) \xrightarrow p g(a)$

Proposition 1 Proof¶

Proof

\[ \begin{align} \text{Pr}(|X_{n} + Y_{n} - (X + Y)| > \epsilon) &= \text{Pr}(|X_{n} - X + Y_{n} - Y| > \epsilon) & (1)\\ &\leq \text{Pr}(|X_{n} - X| + |Y_{n} - Y| > \epsilon) & \text{triangle inequality } (2)\\ &\leq \text{Pr}\left( |X_{n} - X| > \frac{\epsilon}{2} \text{ or } |Y_{n}- Y| > \frac{\epsilon}{2}\right) & *^{1} \ (3) \\ &\leq \text{Pr}\left( |X_{n} - X| > \frac{\epsilon}{2} \right) + \text{Pr}\left( |Y_{n}-Y| > \frac{\epsilon}{2} \right) & *^{2} \ (4)\\ & \to 0 + 0 \ \ \blacksquare \end{align} \]

$*^{1}$: To show this holds true, I'll use this result:

\[ \text{If } A \subseteq B \text{ then } P(A) \leq P(B) \]

i.e. "If A occurs, B occurs" or "if A implies B, $P(A) \leq P(B)$".

Then since $A \implies B \iff \neg B \implies \neg A$ (contrapositive), its trivial to see that

\[ \left\{ |X_{n} - X| \leq \frac{\epsilon}{2} \text{ and } |Y_{n} - Y| \leq \frac{\epsilon}{2} \right\} \implies \{ |X_{n}-X| + |Y_{n} -Y| \leq \epsilon\} \]

holds true. And so the inequality in $(3)$ is true.

$*^{2}$: I use the result

\[ P(A \text{ or } B) = P(A \cup B) = P(A) + P(B) - P(A \cap B) \leq P(A) + P(B) \]

Proposition 2 Proof (WIP)¶

Proposition 3 Proof¶

Proof

By continuity of $g$ for every $\epsilon > 0$, there exists a $\delta > 0$ such that if $|X_{n}-a| \leq \delta$ then $|g(X_{n}) - g(a)| \leq \epsilon$. Thus

\[ \text{Pr}(|g(X_{n}) -g(a)| > \epsilon) \leq \text{Pr}(|X_{n} - a| > \epsilon) \to 0\ \ \ \ \ \ \ \text{contrapositive} \]

How does this relate to statistics?¶

We are interested in the behaviour of an estimator $\hat{\theta}(X_{1}, \dots , X_{n})$ of some parameter $\theta$ and would $\hat{\theta}$ to converge to $\theta$ as the sample size $n$ increases. If $\hat{\theta}(X_{1}, \dots, X_{n}) \xrightarrow p \theta$, then $\hat{\theta}$ is a consistent estimator.

Asymptotic Unbiasedness and a variance converging to zero imply consistency.

Here is an example utilizing the properties of Convergence in Probability

Example

The sample variance $\frac{1}{n}\sum_{i=1}^{n}(X_{i} - \bar{X})^{2} = \left( \frac{1}{n}\sum_{i=1}^{n}X_{i}^{2} \right)-\bar{X}^{2}$.

By the WLLN we have

\[ \frac{1}{n}\sum_{i=1}X_{i}^{2} \xrightarrow p \mathbb{E}[X_{i}^{2}], \]

and by WLLN and the property of convergence under the application of the continuous function $g(x) = x^{2}$ we have

\[ \bar{X}^{2} \xrightarrow p \mathbb{E}[X_{i}]^{2}. \]

Finally, by convergence in probability under sums, the sample variance converges in probability to

\[ \mathbb{E}[X_{i}^{2}] - \mathbb{E}[X_{i}]^{2} = Var(X_{i}) \]

References¶

Probability Distribution Random Variable Consistency (statistics) The Bias and Mean Square Error of Point Estimators Variance Expected Value and Variance of a Random Variable