Statistical Inference

1 Point Estimation

A point estimator $ \hat{\Theta} $ is a function of the sample data used to estimate an unknown parameter $ \theta $.

Bias

$ B(\hat{\Theta}) = E[\hat{\Theta}] - \theta $
An estimator is unbiased if $ B(\hat{\Theta}) = 0 $.

Mean Squared Error (MSE)

$ \text{MSE}(\hat{\Theta}) = E[(\hat{\Theta} - \theta)^2] $
$ \text{MSE} = \text{Var}(\hat{\Theta}) + [B(\hat{\Theta})]^2 $

Consistency: An estimator is consistent if it converges in probability to the true parameter as $ n \to \infty $.

2 Maximum Likelihood Estimation (MLE)

The MLE method finds the parameter $ \theta $ that maximizes the Likelihood Function $ L(\theta) $, which is the probability of observing the given data.

$$ \hat{\theta}_{MLE} = \arg\max_{\theta} L(\theta; x_1, \dots, x_n) $$

MLE Explorer: Coin Toss

Suppose we flip a coin $ n $ times and get $ k $ heads. The likelihood function for the probability of heads $ p $ is $ L(p) \propto p^k (1-p)^{n-k} $.
Adjust $ n $ and $ k $ to see how the likelihood function changes and where the maximum lies.

Total Flips ($ n $): 10

Heads ($ k $): 5

MLE Estimate: $ \hat{p} = $ 0.50

3 Interval Estimation (Confidence Intervals)

A $ (1-\alpha)100\% $ Confidence Interval is a random interval $ [L, U] $ that contains the true parameter $ \theta $ with probability $ 1-\alpha $.

$$ P(L \leq \theta \leq U) = 1 - \alpha $$

Common CI for Mean (known $ \sigma $): $$ \bar{X} \pm z_{\alpha/2} \frac{\sigma}{\sqrt{n}} $$

Confidence Interval Simulator

We generate 20 samples from a Normal distribution with $ \mu=0, \sigma=1 $. For each sample, we calculate the 95% CI.
Green intervals contain the true mean (0), Red ones do not. In the long run, ~95% should be green.