Convergence in Distribution¶
Preface¶
Convergence in Probability to a fixed parameter \(\theta\) is simple in that it either converges or doesn't. The notion of convergence in distribution allows us to get more precise information on the distribution.
Convergence in Distribution Definition¶
Definition
Suppose we have CDFs \(F_{X_{n}}, F_{X}\) of the random variables \(X_{n}, X\) and \(C(F_{X})\) are the set of points where the distribution of \(F_{X}\) is continuous. We say that \(X_{n}\) converges in distribution to \(X\) if \(F_{X_{n}}(z) \to F_{X}(z), \ \ \forall z\in C(F_{X})\).
The distribution of \(X\) is called the asymptotic or limiting distribution.
The reason for having \(C(F_{X})\) is because we want this definition to capture convergence for constant random variables. Think of a constant R.V as a point mass with probability 1.
For continuous CDF's, this continuity condition can be ignored since the function is continuous everywhere. For discrete limiting distributions however, this continuity condition needs to be accounted for.
Example
Take a constant R.V (point mass) \(X_{n} = \frac{1}{n}\). \(X_{n}\) should converge to a point mass at zero.
The CDF of the constant R.V zero is \(F(z) = I(z\geq 0)\) and has a discontinuity at zero. However, the CDF of \(\frac{1}{n}\) is \(F_{X_{n}}(z)=I\left( z \geq \frac{1}{n} \right)\). This converges pointwise to \(F\) everywhere except at the point \(z=0\).
To make this tangible:
For any finite \(n\), \(F_{X_{n}}(0)\) doesn't converge to \(F(0)\), the discontinuity. So we use a workaround and define a continuous set of points \(C\).
We denote convergence in distribution of \(X_{n}\) to \(X\) as \(X_{n} \xrightarrow d X\). If \(X\) has a distribution \(\mathcal{N}(0, 1)\), then we can say \(X_{n} \xrightarrow d \mathcal N (0, 1)\). This highlights that we only care about the distribution of the R.V's.
Difference between Convergence in Probability and Convergence in Distribution¶
For convergence in probability, the two R.V's need to be linked or must "talk to each other" in some way.
Coin Flip
Let two R.V's \(X_{n}, Y_{n}\) be the n'th time I flip two separate coins. They converge in distribution because they are the same distribution but they don't converge in probability.
To see why this is the case, we examine the convergence in probability definition
We can associate the term \(|X_{n} - Y_{n}|\) to two cases:
- The coins match, in which case the term is zero.
- The coins don't match, in which the term is one.
Since the coins are fair, both cases happen 50% of the time. This means that if we choose an epsilon \(\epsilon = 0.5\) (in truth, any epsilon holds for this statement), no matter how many times we flip these two coins, they never stop bouncing between zero and one, and thus never converge.
An Example Highlighting Convergence in Distribution¶
Convergence in Distribution is nice because it allows us to find a distribution that we can use after proving and estimator \(\hat{\theta}\) is consistent. With a distribution, we are able to perform hypothesis tests and create confidence intervals.
Scaled Maximum
For \(X_{i} \sim \text{Unif}[0, \theta]\), we found an estimator \(\hat{\theta} = \text{max} \ X_{i}\).
Now we want to find a distribution at which \(\text{max} \ X_{i}\) converges to. (In general, this is difficult).
Consider \(n(\theta - \text{max} \ X_{i})\).
This is the CDF of the exponential distribution with pdf \(\frac{1}{\theta}\exp(-t / \theta)\).
(1): Rearranging variables
(2): Using the compliment. \(P(A) = 1-P(A^{c})\).
(3): By definition, \(F_{X}(s) = P(X < s)\). And since we have it that the maximum of the set of \(X_{i}\)'s be less than some value, it is equivalent to say that all \(X_{i}\)'s in the set are less than that value.
(Technically I skipped a step between (2) and (3) where I should have shown the intersection between all \(X_{i}\)'s, but I think it intuitively makes sense as to why have a power of \(n\) on the \(F_{X_{i}}\)).
(4): Substituting the CDF with the values and rearranging to get this specific form.
(5): A well-known result:
De Moivre - Laplace CLT¶
Definition
If we have \(X_{i} \sim \text{Bernoulli}(p)\) then \(\sqrt{ n }(\bar{X}-p) \xrightarrow d \mathcal N(0, p(1-p))\).
Note that in this case we have a sequence of discrete distributions (Binomials that are shifted and scaled) and these converge to a continuous distribution.
Properties of Convergence in Distribution¶
Properties
Let \(X_{n}\) be a sequence of R.V's and \(a\) a scalar constant.
- If \(X_{n} \xrightarrow p X\) then \(X_{n} \xrightarrow d X\) (The converse doesn't hold).
- If \(X_{n} \xrightarrow d a\) then \(X_{n} \xrightarrow p a\)
- If \(X_{n} \xrightarrow d X\) and \(g\) is a continuous function, then \(g(X_{n}) \xrightarrow d g(X)\).
Slutsky's Theorem¶
Definition and proof of Slutsky's Theorem.
The hypothesis that we have convergence to constants is important because we can show that if \(X_{n} \xrightarrow d X\) and \(Y_{n} \xrightarrow d Y\), it does not mean \(X_{n}+Y_{n} \xrightarrow d X +Y\).
Counterexample
Let \(X_{n}, X, Y_{n},Y \sim \mathcal N(0, 1)\) and \(X,Y\) independent of everything else but \(Y_{n} = -X_{n}\). Naturally \(X_{n} \xrightarrow d X\) and \(Y_{n} \xrightarrow d Y\).
Then \(X_{n} + Y_{n} = 0\) always, which is different than \(X+Y \sim \mathcal N(0,2)\).
Boundedness in Probability¶
For real-valued sequences, boundedness plays an important role - bounded sequences have convergent subsequences. We can similarly define what it means for a sequence of random variables to be bounded in probability. In english: "no mass can escape out to \(\infty\) in the tail".
Definition
For all \(\epsilon\) there exists a \(B\) and an integer \(N\) such that \(\text{Pr}(|X_{n}| \geq B) \leq \epsilon\) for all \(n \geq N\).
Proposition
Consider \(X_{n}, Y_{n}\) with \(X_{n}\) bounded in probability and \(Y_{n} \xrightarrow p 0\). Then \(X_{n}\cdot Y_{n} \xrightarrow p 0\).
Proof Choose a \(B\) large enough so that \(\text{Pr}(|X_{n}| > B) \leq \epsilon\). Then
References¶
Cumulative Distribution Function (CDF) Convergence in Probability