Autoencoders

Definition

A type of NN architecture that focuses on dimensionality reduction, feature extraction and data compression.

There is the encoder, Bottleneck Layer, and decoder.

The encoder compresses the input \(\mathbf{x}\) into a smaller dimensional space (called the latent space).

The latent space is the compressed representation of \(\mathbf{x}\), and captures the most important features.

The decoder maps the latent representation back to the original space, aiming to produce an output \(\mathbf{\hat{x}}\) that is close to the original input \(\mathbf{x}\).

The neural network is trained on minimizing the difference between \(\mathbf{\hat{x}}\) and \(\mathbf{x}\) using something like MSE.

---

In-Depth Math

An autoencoder is a neural network that sets the target to the feature vector: for a given input \(\mathbf{x}\) we have \(\mathbf{y = x}\). For example, we can learn an NN with the loss: \(||f_{2}(\mathbf{xW^{(2)}}) \mathbf{W^{(1)}} - \mathbf{x}||^2_{2}\).

The NN architecture is chosen such that we try to remove redundant information in \(\mathbf{x}\) and the output \(\mathbf{\hat{y}}\) holds the most significant information in \(\mathbf{x}\). Removes noise in \(\mathbf{x}\) so we don't overfit that noise.

Let's consider an NN architecture with one hidden layer, with both transfer functions being the identity, this gives the loss:

\[ ||\mathbf{x_{i}W^{(2)}W^{(1)} - x_{i}}||_{2}^2 \]

The input and output dimensions are size d and the hidden dimension is size p.

Thus we have: \(\mathbf{W^{(2)}} \in \mathbb{R}^{d \times p}, \ \mathbf{W}^{(1)}\in \mathbb{R}^{p\times d}\). Notice that if \(p \geq d, \ \mathbf{W^{(2)}W^{(1) }= I}\), and so we get zero error since: \(||\mathbf{x_{i}I - x_{i}}||_{2}^2 = 0\). This is the reason why we have the Bottleneck Layer, with p < d to force the transformation to be lossy.

Then, the optimization for our data, to ensure that the NN retains only the most significant content in the input is:

\[ \min_{\mathbf{W}^{(2)} \in \mathbb{R}^{d\times p}, \ \mathbf{W}^{(1)}\in \mathbb{R}^{p \times d}} ||\mathbf{XW}^{(2)}\mathbf{W}^{(1)}- \mathbf{ X}||_{2}^2 \]

Assume we have \(\mathbf{X = U\Sigma V^\top}\) with singular values \(\sigma_{1}, \dots , \sigma_{d} \geq 0\). And further we have it that \(\sigma_{1}, \dots , \sigma_{p} > 0\), ie. the first p singular values are non-zero.

Then we select: \(\mathbf{W}^{(2)} = \mathbf{V}_{p}\) and \(\mathbf{W}^{(1)} = \mathbf{V}_{p}^\top\), where \(\mathbf{V_{p}} = [\mathbf{v_{1}, \dots, v_{p}}] \in \mathbb{R}^{d\times p}\). This produces the prediction:

\[ \begin{align} \mathbf{\hat{X}} &= \mathbf{U\Sigma V}^\top \mathbf{V}_{p}\mathbf{V}_{p}^\top \ \ \ \ \ \ \ \ \ \ \ \text{since } \mathbf{V}^\top \mathbf{V}_{p} = [\mathbf{I}_{p}; \mathbf{0}] \in \mathbb{R}^{d \times p} \\ &= \mathbf{U}[\mathbf{\Sigma_{p};0}]\mathbf{V}_{p}^\top \\ &= \mathbf{U}_{p}\mathbf{\Sigma}_{p}\mathbf{V_{p}^\top} \end{align} \]

This solution is the best p-rank approximation to \(\mathbf{X}\). This solution is the same returned by PCA: the top p right-singular vectors of \(\mathbf{X}\).

This solution is not unique without further constraints, we can change the weights in either \(\mathbf{W}^{(1)}\) or \(\mathbf{W}^{(2)}\) and get the same overall behaviour in the autoencoder.

---

Connection with PCA and NN's

There is a connection between Principal Component Analysis (PCA) and Neural Networks; both try to learn important features.

We can specify a neural network to have the same solution as a PCA, and this special network is called an Autoencoder. An autoencoder has a linear activation and a Bottleneck Layer.

---

Architectures

Denoising Autoencoder

References

Principal Component Analysis (PCA) Neural Networks