Direct Methods for Missing Data¶
Definition¶
The goal is to learn a function conditioned on all of the available data: the features that are available, and information on which features are missing,
where \(\mathcal{A}\) is a matrix that represents on the \(i\)'th sample, the \(j\)'th value is missing. And similarly for \(\mathcal{M}\) but for missing data.
We should be able to learn a more accurate function this way, compared to imputing data from Matrix Completion (Imputing Data).
This is because the two-stage approach has bias, depending on the imputation, whereas \(f(\mathbf{x_{\mathcal{A}},\mathcal{M}})\) is conditioned only on the true given data.
Approach¶
If we want to learn a neural network \(f\), then a simple yet effective strategy is to set elements in \(\mathbf{x}_{\mathcal{M}}= 0\), and introduce an indicator vector \(\mathcal{M}\), which is the same size as input vector \(\mathbf{x}\).
Thus, \(\mathcal{M}\) is a binary vector that is 0 if the value exists, and 1 if not, which basically just tells the neural network where the missing values are. Think of this indicator vector like an if-statement: If \(x_{j}\) is available, then use the weights on \(x_{j}\) to produce the features on the first layer. If \(x_{j}\) is not available, then the weights on \(x_{j}\) multiply by zero and do not add to the sum.
Formalizing this, lets consider a simple linear setting for a target \(y \in \mathbb{R}\). We have weights \(\mathbf{w}_{x} \in \mathbb{R}^{d \times 1}\) on \(\mathbf{x} \in \mathbb{R}^{1 \times d}\) and \(\mathbf{w}_{m} \in \mathbb{R}^{d \times 1}\) on \(\mathbf{m} \in \mathbb{R}^{1 \times d}\).
We stack these and use
Notice that:
We still have a linear function on the available \(x_{j}\) but shift our predictions by the missing values \(\mathbf{w}_{m,j}\).
The neural network can input these two stacked vectors and learn a complex function that conditions on both the available features and the missingness pattern.