Gaussian Process

Definition

A Gaussian Process defines a distribution over a function:

\[ f(x) \sim \mathcal{GP}(m(\mathbf{x}), \ \mathbf{k}(\mathbf{x},\mathbf{x'})) \]

Where the \(m(\mathbf{x}) = \mathbb{E}[f(\mathbf{x})]\) is the mean, and \(\mathbf{k}(\mathbf{x}, \mathbf{x'})\) is the covariance of the multivariate Gaussian. Recall that \(\mathbf{k}(\mathbf{x}, \mathbf{x'})\) is the kernel.

The key property is that: For any finite set of inputs \(\mathbf{x}_{1}, \dots ,\mathbf{x}_{n}\), the corresponding outputs \([f(\mathbf{x}_{1}), \dots, f(\mathbf{x}_{n})]\) follow a multivariate Gaussian. (The set means all possible outputs for all inputs).

Note that a Gaussian Process is an infinite COLLECTION of finite multivariate Gaussians, which depend on the inputs that are chosen. So for any given set of inputs, we get a finite-dimensional multivariate Gaussian distribution over the possible outputs. But since we can pick so many different sequences of inputs, there are basically an infinite amount of finite-dimensional multivariate gaussians.

---

Example

Lets say you take 100 random data points from your dataset \(\mathcal{D} = \{ \mathbf{x}_{i} \}_{i=1}^n\), so we have a set \([ \mathbf{x}_{i}, \dots \mathbf{x}_{i+99}]\).

Then our mean \(m(\mathbf{x})\) is:

\[ \boldsymbol{\mu} = \begin{bmatrix} m(\mathbf{x}_i) \\ m(\mathbf{x}_{i+1}) \\ \vdots \\ m(\mathbf{x}_{i+99}) \end{bmatrix} \in \mathbb{R}^{100} \]

And our covariance is:

\[ K = \begin{bmatrix} k(\mathbf{x}_i, \mathbf{x}_i) & k(\mathbf{x}_i, \mathbf{x}_{i+1}) & \cdots & k(\mathbf{x}_i, \mathbf{x}_{i+99}) \\ k(\mathbf{x}_{i+1}, \mathbf{x}_i) & k(\mathbf{x}_{i+1}, \mathbf{x}_{i+1}) & \cdots & \vdots \\ \vdots & & \ddots & \\ k(\mathbf{x}_{i+99}, \mathbf{x}_i) & \cdots & & k(\mathbf{x}_{i+99}, \mathbf{x}_{i+99}) \end{bmatrix} \in \mathbb{R}^{100 \times 100} \]

Where each entry \(\mathbf{K}_{jk}= \mathbf{k}(\mathbf{x}_{j}, \mathbf{x}_{k})\) measures the similarity (covariance) between the function values at inputs \(\mathbf{x}_{j}\) and \(\mathbf{x}_{k}\). And we measure this similarity for all possible inputs.

References

Kernel Model