Why do we project to higher dimensions?

Definition

We project to higher dimensions because a higher dimensional space separates data points such that we're able to use a linear line to properly separate data points.

Clover's Theorem: states that a dataset that is not linearly separable may become linearly separable once projected into a higher dimensional space using a non-linear transformation.

Then, How do we pick this projection to higher dimensions; how do we pick this data representation?

Typically, there are two criterion: By expanding the dimensionality, does it bring us closer to the space of the true function? (less bias) How much more computation are we going to do if we keep increasing the dimension space?

Example

Red points are negative, blue points are positive.

We want a function \(f(\mathbf{x})\) such that if \(f < 0\) , we classify a point as negative, and if \(f\geq 0\) we classify the point as possitive.

Here we find such an \(f(\mathbf{x}) = x_{1}^2+x_{2}^2-1\).

We could easily learned a linear function given a 3-dimensional feature vector: \(\Phi = [x_{1}^2, x_{2}^2, 1]\).