Covariance

Definition

Standard deviation and variance only operate on 1-dimension, so that you can only calculate the standard deviation for each dimension independently of the other dimensions.

However, it's useful to be able to find out how much the dimensions vary from the mean with respect to each other. Covariance is this measure.

Covariance is always measured between 2 dimensions, if you calculate the covariance between a dimension and itself, you get the variance.

• If you had a 3-dimensional dataset (\(x\),\(y\),\(z\)), then you could measure the covariance between the \(x\) and \(y\) dimensions, the \(y\) and \(z\) dimensions, and the \(x\) and \(z\) dimensions.

Furthermore, the covariance matrix shows the relationship between different features (ie. matrix columns) in the dataset. For a 3-dimensional dataset, the covariance matrix would look like:

\[ Cov(x,y,z) = \begin{bmatrix} cov(x,x) & cov(x,y) & cov(x,z) \\ cov(y,x) & cov(y,y) & cov(y,z) \\ cov(z,x) & cov(z,y) & cov(z,z) \end{bmatrix} \]

If the value is positive, the two dimensions increase together. If the value is negative, then one dimension increases and the other decreases. If the value is zero, the two dimensions are independent of each other.

References

Covariance Matrix