Kullback-Leibler Divergence (KL)

Definition

The KL Divergence, denoted as

\[ D_{KL}(P||Q), \]

is a type of statistical distance, used to measure how "different" two probability distributions are.

The formula is

\[ D_{KL}(P||Q) = \sum_{x \in \mathcal{X}} P(x)\log\left( \frac{P(x)}{Q(x)} \right). \]

The KL divergence of \(P\) from \(Q\) is the expected excess surprise from using \(Q\) as a model instead of \(P\), when the actual distribution is \(P\).

The more similar the two distributions are to each other, the smaller the number. If the two distributions are the same, then

\[ D_{KL}(P||Q) = 0. \]

Note that this distance is not a metric, and thus doesn't satisfy the Triangle Inequality.

References

Entropy (Probability) Cross-Entropy (Probability) Surprise or Uncertainty (Probability)