Title: Part II / Statistical Modelling
Tags: Linear::Models Generalised::Linear::Models Exponential::Families

Q: How does the OLS derivation go?
A: 1. Log-likelihood function: \[ l(\beta, \sigma^2) = -\frac{n}{2} \log \sigma^2 - \frac{1}{2\sigma^2} \| Y - X\beta \|^2 + \text{constant} \]2. OLS estimator of \( \beta \): \[ \hat{\beta} = \arg \min_{\beta} \| Y - X\beta \|^2 \]Closed-form solution (normal equations): \[ \hat{\beta} = (X^T X)^{-1} X^T Y \]
3. MLE of \( \sigma^2 \): \[ \hat{\sigma}^2_{\text{MLE}} = \frac{1}{n} \| Y - X\hat{\beta} \|^2 = \frac{1}{n} \| R \|^2 \]4. Unbiased estimator of \( \sigma^2 \): \[ \hat{\sigma}^2 = \frac{1}{n-p} \| Y - X\hat{\beta} \|^2 = \frac{1}{n-p} \| R \|^2 \]5. Residual Sum of Squares (RSS): \[ \| R \|^2 = \| Y - X\hat{\beta} \|^2 \]

Q: What is that Hat matrix and what is the vector of residuals?
A: \[\mathbf{H= X(X^T X)^{-1}X^T}\]\[\mathbf{R = Y - X}\hat{\beta}\]\(\mathbf{H} \) is a projection matrix (since it is idempotent and symmetric) onto the column space of \(\mathbf{X}\) (\(\mathcal{C}(\mathbf{X}) = \{\mathbf{X \beta | \beta \in \mathbb{R}^p}\}\))

Q: What is the OLS estimator \(\hat{\beta_1}\) when we have \[\hat{\beta} = \begin{pmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \end{pmatrix}\]
A: Define projections onto the space of \( X_0 \):
\[\tilde{X}_1 = (I - P_0)X_1, \quad \tilde{Y} = (I - P_0)Y\]
Then, the OLS estimator for \( \tilde{Y} \) on \( \tilde{X}_1 \):
\[\hat{\beta}_1 = (\tilde{X}_1^T \tilde{X}_1)^{-1} \tilde{X}_1^T \tilde{Y}\]
Using Gram-Schmidt, we obtain:
\[P - P_0 = P_{\tilde{X}_1} = \tilde{X}_1 (\tilde{X}_1^T \tilde{X}_1)^{-1} \tilde{X}_1^T\]
Decomposing \( Y \):
\[Y = X_0 \hat{\beta}_0 + X_1 \hat{\beta}_1 + R\]
Thus:
\[\tilde{X}_1 \hat{\beta}_1 = (P - P_0)Y\]
\[P_{\tilde{X}_1} Y = \tilde{X}_1 \hat{\beta}_1\]

Q: What is the density function for the multivariate normal distribution and how is \(AZ + b\) distributed for a fixed matrix and vector?
A: \[f(\mathbf{z}) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2}}e^{-(\mathbf{z-\mu})^T \Sigma^{-1}(\mathbf{z- \mu})/2}\]\[\mathbf{AZ + b} \sim N_k(\mathbf{A}\mu + \mathbf{b}, \mathbf{A\Sigma A^T})\]

Q: What are the t-distribution and F-distribution?
A: For \(Z \sim N(0,1), S \sim \chi_d^2\) and \(Z \perp S\) then \[\frac{Z}{\sqrt{S/d}} \sim t_d\]For \(S_1 \sim \chi_{d_1}^2, S_2 \sim \chi_{d_2}^2\) and \(S_1 \perp S_2\) we have \[\frac{S_1 / d_1}{S_2/d_2} \sim F_{d_1,d_2}\]

Q: What is the test for comparing a full model \(\mu = X\beta\) with a submodel \(\mu = X_0 \beta_0\)?
A: GLR statistic is given by \[\frac{\sup_{\beta \in \mathbb{R}^p} L(\beta, \sigma^2)}{\sup_{\beta_0 \in \mathbb{R}^{p_0}, \beta_1 = 0} L(\beta, \sigma^2)} = \left\{ 1 + \frac{\| (P - P_0) Y \|^2}{\| (I - P) Y \|^2} \right\}^{n/2}\]Rejects if the ratio between the RSS of full model is much smaller than the RSS of the submodel and therefore the ratio is very big. Under the null hypothesis we have \(Y = X\beta + \epsilon = X_0 \beta_0 + \epsilon\), therefore the ratio is \[\frac{\| (P - P_0) \epsilon\|^2}{\| (I - P) \epsilon \|^2}\]Because \(P-P_0, I-P\) are project matrices we have \[F = \frac{\| (P - P_0) Y\|^2 /(p - p_0)}{\| (I - P) Y \|^2 /(n-p)} \sim F_{p-p_0, n-p}\]under \(H_0\).

Q: What is the generalised least squares estimator?
A: \[\hat{\beta}_{GLS} = (X^T \Sigma^{-1}X)^{-1}X^T \Sigma^{-1} Y\]

Q: What are homoscedasticity and heteroscedasticity?
A: \[Y_i = \mathbf{X}_i^T \boldsymbol{\beta} + \epsilon_i, \quad i = 1, \dots, n,\]
where
\[(\epsilon_i, \mathbf{X}_i), \quad i = 1, \dots, n, \text{ are independent and identically distributed (IID) };\]
\[\mathbb{E}(\epsilon_i \mid \mathbf{X}_i) = 0;\]
\[\text{Var}(\epsilon_i \mid \mathbf{X}_i) = \sigma^2(\mathbf{X}_i).\]When \(\sigma^2 (X_i) = \sigma^2\) is a constant we say the noise is homoscedastic, otherwise the noise is heteroscedastic.

Q: What is the ommited-variable bias?
A: If we have two models \[Y_i = X_i^T\beta + \epsilon_i\]\[Y_i = X_i^T \beta^* + Z_i^T \gamma^* + \epsilon^*\]In general, \(\beta \neq \beta^*\)

Q: What is the coefficient of determination?
A: \[R^2 = \frac{|| \mathbf{\hat{\mu}} - \overline{Y}\mathbf{1}||^2}{|| \mathbf{Y} - \overline{Y}\mathbf{1}||^2} = 1 - \frac{|| \mathbf{Y} - \mathbf{\hat{\mu}}||^2}{|| \mathbf{Y} - \overline{Y}\mathbf{1}||^2}\]Measures the proportion of variance of \(Y_i\) that can be explained by the linear model.

Q: What is the leverage of the ith observation?
A: \[H_{ii}\]where H is the hat  matrix. We have \[\hat{\mu}_i = (HY)_i = H_{ii}Y_i + \sum_{k \neq i} H_{ik}Y_k\]How much the observed value \(Y_i\) determines the fitted value \(\hat{\mu}_i\)

Q: What is a quantile-quantile(Q-Q) plot?
A: We use this to check normality of nosie. The studentized or standardized residueal of the ith observation is \[\tilde{R}_i = \frac{R_i}{\hat{\sigma}\sqrt{1 - H_{ii}}}\]If the normal linear model is correct, these should be close to \(\epsilon_i / \sigma\) which follows a standard normal. We plot the sample quantiles of \((\tilde{R}_1, \dots , \tilde{R}_n)\) against the theoretical quantiles of N(0,1).

Q: What is a residual vs fitted plot?
A: Just plots the studentized residual against the predicted value. It can tell us whether \(\mathbb{E}(\epsilon_i | X_i) = 0\)

Q: What is a scale-location plot?
A: Shows the square root of the absolute value of the standardized residual against the fitted value. This is used to check the homoscedasticity assumption \(\text{Var}(\epsilon_i | X_i) = \sigma^2\) under which the \(\sqrt{|\tilde{R}_i|}\) should have an average value around 1.

Q: What is a residuals vs leverage plot?
A: Shows \(\tilde{R}_i\) against \(H_{ii}\) and is used to identify outliers with a large leverage. We say an observation \((X_i, Y_i)\) is an outlier if \(|R_i|\) is much larger than what is expected if the noise is normally distribution.

Q: What is Cook's distance?
A: \[D_i = \frac{||\mathbf{X} (\hat{\beta} - \hat{\beta}_{(-i)}) ||^2}{p \hat{\sigma}^2} = \frac{1}{p}\frac{H_{ii}}{1 - H_{ii}}\tilde{R}_i^2\]It is a standardized change of the fitted values when the ith observation is removed. A large value of \(D_i\) indicates that the ith observation has a large influence on the fitted values

Q: What is the bias-variance decomposition of MSPE?
A: \[MSPE(\mathbf{x}) = \mathbb{E}[\{g(\mathbf{x}) - \mathbf{x}^T \beta_n\}^2] + \text{Var}(\mathbf{x}^T \hat{\beta}) + \sigma^2\]Which is the bias of the observation, the variance and the irreducible variance of the data.
For the OLS estimator it can be shown that \[\sum_{i=1}^n \text{Var}(X_i^T \hat{\beta} | X) = p\sigma^2\]which shows us that as we increase complexity to decrease bias, we increase the variance.

Q: What are Mallows' \(C_p\) and \(LOO-CV\)?
A: \[C_p = ||\mathbf{Y -\hat{\mu}}||^2 + 2p\sigma^2\]is an unbiased estimator of \(\sum_{i=1}^n MSPE(\mathbf{X}_i)\). We can esimate \(\sigma^2\) using \(\hat{\sigma}^2\) obtained from full model
\[LOO-CV = \sum_{i=1}^n (Y_i - \hat{\mu}_{(-i)})^2 = \sum_{i=1}^n \frac{(Y_i - \hat{\mu}_i)^2}{(1 - H_{ii})^2}\]

Q: What are AIC and BIC model selection criteria?
A: Akaike's information criterion attempts to estimate \(\mathbb{E}_f\{ -2 \log f(Y_{n+1} ; \hat{\theta}) | \hat{\theta}\}\) which is what the MLE for a parametric model \(Y_i \sim f(y ; \theta)\) is expected to converge to
\[AIC = -2 \sum_{i=1}^n \log f(Y_i ; \hat{\theta}) + 2 \dim (\Theta)\]The correction term penalizes the log-likelihood function evaluated at the same data used to fit the model. Dividing this by n is a consistent estimator of its target.
\[BIC(\Theta_k) = -2 \sum_{i=1}^n \log f(Y_i ; \hat{\theta}_k) + \dim (\Theta_k) \log n\]where \(\hat{\theta}_k\) is the MLE over a Euclidean model space \(\Theta_k\). It is motivated by the Bayesian perspective on model selection. This provides a much more principles way to select "true" regressors than naive selection rules based on statistical significance of regression coefficients.
Compared with AIC, BIC puts a larger penalty on model complexity and thus selects a smaller model. In practice, a rule of thumb is that AIC is more suitable for predictions and BIC is more suitable for selecting the "correct" model.

Q: What are the different algorithms for model section involving subsets?
A: Best subset algorithm selects the smallest RSS for every degree of freedom k
Forward stepwise and backward stepwise start with the null and full model respectively and greedily add or remove regressors depending on which decreases/increases the RSS the least

Q: What are ridge regression and lasso regression?
A: Ridge:\[\text{minimize } ||\mathbf{Y - X \beta}||^2 + \lambda ||\beta||_2^2\]
Lasso: \[\text{minimize } ||\mathbf{Y - X \beta}||^2 + \lambda ||\beta||_1\]

Q: Definition. Suppose \(f_0(\mathbf{y}), \mathbf{y} \in \mathcal{Y} \subseteq \mathbb{R}^d\) is a density function with respect to a measure. By exponential tilting, we mean a collection of density functions given by \[f(\mathbf{y; \theta}) = [...]\]What are the different parts of this called?
A: e^{\theta^T T(y) - K(\theta)}f_0(y)
Back Extra: \(\theta\in\mathbb{R}^p\) is the natural parameter or canonical parameter
\(T(y) \in \mathbb{R}^p\) is called the sufficient statistic
\(f_0(y)\) is called the carrying density
\(K(\theta)\) is called the cumulant function
The exponential family is called minimal if T(y) is linearly independent.

Q: Definition. The cumulants of the probability distribution are given by [...]
A: \(\kappa_r = K^{(r)}(0)\) where \(K(t) = \log M(t) = \log (\mathbb{E}(e^{tY}))\).
Back Extra: The first two cumulants are the mean and variance.

Q: Definition. The mean-variance relationship is given by \[[...]\]
A: \mu'(\theta) = K''(\theta) = V(\theta) \ge 0
Back Extra: Comes from the fact that \[\mu(\theta) = K'(\theta) = \mathbb{E}_{\theta}(Y)\]\[V(\theta) = K''(\theta) = \text{Var}_{\theta}(Y)\]

Q: What is deviance in exponential families?
A: A measure of how one distribution in an exponential family differs from another.
\[\begin{align*} D(\theta_1,\theta_2) &= 2 \mathbb{E}_{\theta_1}\left\{ \log\frac{f(Y;\theta_1)}{f(Y;\theta_2)} \right\} \\ &= 2 \mathbb{E}_{\theta_1}\{ (\theta_1-\theta_2)Y-K(\theta_1)+K(\theta_2)\} \\ &= 2 \{ (\theta_1-\theta_2)\mu_1-K(\theta_1)+K(\theta_2)\}. \end{align*}\]

Q: What is the Box-Cox transformation?
A: \[Y \mapsto Y^{(\lambda)} = \begin{cases} (Y^{\lambda} - 1)/\lambda, & \text{if } \lambda \neq 0 \\ \log Y, & \text{if }\lambda = 0\end{cases}\]

Q: What are the three components of GLMs?
A: (i) Modelling the distribution of \(Y_i\) given \(X_i\) using an exponential family
(ii)A linear predictor \(\eta_i = X_i^T \beta\)
(iii) A strictly increasing and smooth link function g that relates the predictor \(\eta_i\) with the conditional expectation \(\mu_i = E(Y_i |X_i); \eta_i = g(\mu_i)\) and \(\mu_i = g^{-1}(\eta_i)\)

Q: What is overdispersion?
A: The phenomenon that the variance of Y is larger than what is expected from a theoretical model. A common mechanism for overdispersion and underdispersion is unaccounted structure in the sample.

Q: What is the deviance additivity theorem?
A: It says that the deviance between the observations and the submodel can be decomposed as \[D^{(n)}(Y, \hat{\mu}_0) = D^{(n)}(Y, \hat{\mu}) + D^{(n)}(\hat{\mu}, \hat{\mu}_0)\]The equation follows from the following relation between the deviance and log-likelihood\[D^{(n)}(\hat{\mu}, \hat{\mu}_0) = 2\{l(\hat{\mu}) - l (\hat{\mu}_0)\}\]

Q: How do we carry out Newton-Raphson for linear models?
A: We have \[l(\beta) \approx l(\beta^*) + (\beta - \beta^*)U(\beta^*) + \frac{1}{2}(\beta - \beta^*)^TH(\beta^*)(\beta - \beta^*)\]So then we need \[U(\beta^*) +H(\beta^*)(\beta - \beta^*) = 0\]So we update our parameter by \[\beta^{(t)} = \beta^{(t-1)} - \{H(\beta^{(t-1)})\}^{-1} U(\beta^{(t-1)})\]

Q: What is a binomial regression model?
A: Assumed that responses \(Y_1 , \dots , Y_n\) are independent and \[Y_i \sim \frac{1}{n_i} \text{Binomial}(n_i, \mu_i) , \quad i = 1,\dots , n\]where \(n_i\) is known but \(\mu_i\) is unknown.

Q: What is the logit function?
A: \[g(\mu) = \theta(\mu) = \log \frac{\mu}{1-\mu}\]Canonical link for binomial regression

Q: What is the natural parameter and cumulant function for Poisson regression?
A: \[f(y_i ; \mu_i, \sigma^2) = e^{\frac{1}{\sigma^2}\{y_i \log \mu_i - \mu_i\}}f_0(y_i; \sigma^2)\]\[\theta_i = \log(\mu_i), K(\theta) = e^{\theta}\]

Q: What is the canonical link function in Poisson regression and what type of model does it lead to?
A: \[g(\mu) = \theta(\mu) = \log(\mu)\]So the model becomes the log-linear model and we get \[\mu_i = e^{X_i^T \beta} = \prod_{j=1}^p (e^{\beta_j})^{X_{ij}}\]

Q: What is the Poisson trick?
A: If \(Y_i \sim \text{Poisson}(\mu_i), i=1,\dots,L\) independently. Let \(Y_+ = \sum_{i=1}^L Y_i\) then \[Y_+ \sim \text{Poisson}(\mu_+)\] and \[Y_1 , \dots ,Y_L | Y_+ \sim\text{Multinomial}(Y_+, \pi)\]where \(\pi_i = \mu_i / \mu_+\) and \(\mu_+ = \sum_{i=1}^L \mu_i\)

Q: How can we test independence in contingency tables using Poisson regression?
A: Our null hypothesis becomes \[\mu_{jk} = \mu_+\pi_j^A \pi_k^B \quad \text{ for all }j,k\]which is equivalent to the log-linear model \[H_0: \log \mu_{jk} = \alpha + \beta_j^A + \beta_k^A\]which is a submodel of the saturated model \[H_1: \log\mu_{jk} = \alpha +\beta_j^A + \beta_k^A + \beta_{jk}^{AB}\]so we are testing nested models in Poisson regression