Radial Basis Function Networks

Definition

RBF Networks provide nonlinearity, similar to polynomial transformations.

Assume we're given a dataset \(D = \{ (\mathbf{x_{i}}, y_{i})\}\), then pick p points to act as centers, denoted as:

\[ \mathbf{c_{1}, c_{2}, \dots ,c_{p}} \in \cal X \]

These centers are usually chosen:

1. to uniformly cover \(\cal X\)
2. selected as a subset of \(D\)
3. computed using some clustering technique over the data

The resulting basis functions - giving a fixed representation - consists of p new features:

\[ \phi_{1}(\mathbf{x}), \phi_{2}(\mathbf{x}), \dots \phi_{p}(\mathbf{x}) \in \mathbb{R} \]

where each

\[ \phi_{j}(\mathbf{x}) := \exp\left( -\frac{1}{2\sigma^2} ||\mathbf{x-c_{j}}||^2_{2}\right) \]

\(\sigma\) gives the width of the Gaussians. The resulting features are between 0 and 1, indicating how close a feature \(\mathbf{x}\) is to its corresponding center \(\mathbf{c_{j}}\). If they are similar then the value tends to 1, if they are dissimilar the value tends to zero.

The size of \(\sigma\) determines if the feature value is large or small. If \(\sigma\) is small, then most features are near zero. If \(\sigma\) is large then most features will be non-negligible.

Using enough RBF Networks allow us to approximate any function, however there are hyperparameters that need to be chosen, namely: \(\sigma\), the number of centers and how we choose them. Usually we find \(\sigma\) using cross-validation.

From RBF Networks, we obtain an RBF feature matrix:

And for a given input \(\mathbf{x}\), the prediction y will be predicted as:

\[ f(\mathbf{x}) = \mathbf{\phi(x)^\top w} \]

References

L2 Norm