Maximum Likelihood Estimation

Definition

In the problem setting of point estimation, suppose we have i.i.d observations \(X_{i}\) from a distribution with pdf or pmf \(p(x | \theta)\). Using the data, we want to find the parameter \(\theta\). That is, we want to produce an estimate \(\hat{\theta}(X_{1}, \dots , X_{n})\).

The MLE is a strategy to finding \(\theta\) by maximizing the likelihood function \(L(\theta) = \prod_{i=1}^{n}p(X_{i}| \theta)\) over the parameter space \(\Theta\). \(L\) is a joint density function over \(\theta\) instead of \(x_{i}\). This means that the MLE is \(\hat{\theta}_{MLE} = \arg\max_{\theta \in \Theta}L(\theta)\).

Equivalently, we can maximize the log-likelihood, which provides more convenient derivations since it breaks the product into a sum. We denote the log-likelihood function as \(\ell (\theta; \ X_{1}, \dots , X_{n})\). We can then write \(\ell (\theta) = \sum_{i=1}^{n}\ell(\theta | X_{i})\), where \(\ell(\theta ; X_{i}) = \log P(X_{i} | \theta)\).

Note: \(\arg\max f = \arg\min -f\). So a maximization problem is easily converted to a minimization problem.

To solve a maximization or minimization problem, we use calculus to find the first derivative, set it to zero. Then find the second derivative to determine whether \(f\) is concave or convex.