Title: Part IB / Statistics
Tags: Hypotheses Estimation Multivariate_Normal Linear_Model Confidence_Intervals Bayesian_Estmation

Q: Definition. An estimator is a [...] The distribution of \(T(X)\) is called its [...]
A: "statistic" of a function of the data \(T(X) = \hat{\theta}\) which we use to approximate the true parameter \(\theta\).; sampling distribution.
Back Extra: Take \(X_1, \dots, X_n\) be i.i.d like \(N (\mu , 1)\). \(\hat{\mu} = \frac{1}{n} \sum_{i=1}^n X_i\). The sampling distribution of \(\hat{\mu}\) is \(N(\mu, 1/n)\)

Q: Definition. The bias of \(\hat{\theta}\) is [...]
A: \[\text{bias}(\hat{\theta}) = \mathbb{E}_{\theta}(\hat{\theta}) - \theta\]
Back Extra: The bias is a function of true parameter \(\theta\), which is not explicit in notation \(\text{bias}(\hat{\theta})\)

Q: Definition. We say \(\hat{\theta}\) is unbiased if [...]
A: \[\text{bias}(\hat{\theta} ) = 0 \text{ for all } \theta \in \Theta\]
Back Extra: \(\hat{\mu}\) is unbiased because \(\mathbb{E}_{\mu}(\hat{\mu}) = \mu\) for all \(\mu \in \mathbb{R}\)

Q: Definition. The mean squared error of \(\hat{\theta}\) is [...]
A: \[\text{mse}(\hat{\theta}) = \mathbb{E}_{\theta}((\hat{\theta} - \theta)^2)\]
Back Extra: This is a function of \(\theta\). There is bias-variance decomposition. Write \(\text{mse}(\hat{\theta}) = \mathbb{E}_{\theta}((\hat{\theta} - \mathbb{E}_{\theta} \hat{\theta} + \mathbb{E}_{\theta} \hat{\theta} - \theta)^2) = \text{Var}_{\theta}(\hat{\theta}) + \text{bias}(\hat{\theta})^2\)

Q: Definition. A statistic T is sufficient for \(\theta\) if [...]
A: the conditional distribution of X given \(T(X)\) does not depend on \(\theta\).
Back Extra: \(\theta\) can be a vector and \(T(X)\) can also be vector-valued.

Q: Theorem. Factorisation criterion. T is sufficient for \(\theta\) iff \(f_X(x| \theta) = g(T(x), \theta)h(x)\) for some functions g,h.
A: (discrete proof) Suppose \(f_X(x | \theta) = g(T(x), \theta) h(x)\). If \(T(x) = t\), then
This doesn't depend on \(\theta\) so T is sufficient.
Conversely, suppose T is sufficient. Then \[\mathbb{P}_{\theta}(X = x, T(X) = t(x)) = \mathbb{P}_{\theta}(X = x | T(X) = T(x))\mathbb{P}_{\theta}(T(X) = T(x))\] The first term does not depend on the theta at T is sufficient, so call this h(x) and call the second term \(g(T(x), \theta)\)

Q: Definition. A sufficient statistic T(x) is minimal sufficient if [...]
A: it is a function of every other sufficient statistic.
Back Extra: If \(T'\) is another sufficient statistic, then \[T'(x) = T'(y) \implies T(x) = T(y)\]
Be definition any two minimal sufficient statistics are in bijection with each other.

Q: Theorem. Suppose T(X) is a statistic such that \(\frac{f_X(x | \theta)}{f_X(y | \theta)}\) is constant as a function of theta if and only if \(T(x) = T(y)\). Then T is minimal sufficient.
A: For any value t of T, let \(z_t\) be a representative from the equivalence class \(\{x : T(x) = t\}\). Then, \[f_X(x | \theta) = f_X(z_{T(x)} | \theta) \cdot \frac{f_X(x | \theta)}{f_X(z_{T(x)} | \theta)}\] So call the first term \(g(T(x), \theta)\) and the second term is constant as a function of \(\theta\) so call this h(x).
So by factorisation criterion. T(X) is sufficient.
To prove T is minimal, let S be any other sufficient statistic. By factorisation criterion \(\exists g_S, h_S\) s.t. \[f_X(x | \theta) = g_S(S(x), \theta) h_S(x)\] suppose \(S(x) = S(y)\), then \[\frac{f_X(x | \theta)}{f_X(y | \theta)} = \frac{g_S(S(x), \theta) h_S(x)}{g_S(S(y), \theta) h_S(y)} = \frac{h_S(x)}{h_S(y)}\] does not depend on \(\theta\) so by the condition in the theorem \(T(x) = T(y)\) so \(S(x) = S(y) \implies T(x) = T(y)\).

Q: Theorem. Rao-Blackwell. Let T be a sufficient statistic for \(\theta\). Let \(\tilde{\theta}\) be an estimation of \(\theta\) with \(\mathbb{E} \tilde{\theta}^2 < \infty\) for all \(\theta\). Define a new estimation \(\hat{\theta} = \mathbb{E}(\tilde{\theta} | T(X))\). Then for all \(\theta\), \[\mathbb{E}((\hat{\theta} - \theta)^2) \leq \mathbb{E}((\tilde{\theta} - \theta)^2)\] The inequality is strict unless \(\tilde{\theta}\) is a function of \(T(X)\).
A: By Tower property \[\mathbb{E}(\hat{\theta}) = \mathbb{E}[\mathbb{E}(\tilde{\theta} | T(X))] = \mathbb{E}(\tilde{\theta})\] Hence \(\text{bias}(\hat{\theta}) = \text{bias}(\tilde{\theta})\). Now to compare variances \[\text{Var}(\tilde{\theta}) = \text{Var}(\mathbb{E}(\tilde{\theta} | T)) + \mathbb{E}(\text{Var}(\tilde{\theta} | T))\]
So this gives us \(\text{Var}(\tilde{\theta}) \leq \text{Var}(\hat{\theta})\). Furthermore \(\mathbb{E}(\text{Var}(\tilde{\theta} | T)) = 0\) iff \(\text{Var}(\tilde{\theta} | T) = 0\) with probability 1 iff \(\tilde{\theta}\) is a function of T(X).

Q: Definition. The likelihood function is \[L : [...]\] The maximum likelihood estimator is the value of \(\theta\) maximising L.
A: \theta \mapsto f_X(X| \theta)
Back Extra: The log-likelihood is denoted \[l(\theta) = \log L(\theta)\]

Q: Definition. A (100-\(\gamma\))% confidence interval (where \(0 < \gamma < 1\)) for a parameter \(\theta\) is [...]
A: a random interval \((A(X), B(X))\) such that \[\mathbb{P}(A(X) \leq \theta \leq B(X)) = \gamma\] for all values of \(\theta\)

Q: Proposition. If T monotone increasing, and \((A(X), B(X))\) is a \((100-\gamma\))% confidence interval for \(\theta\), then \((T(A(X)), T(B(X)))\) is [...]
A: a \(100-\gamma\)% C.I. for \(T(\theta)\).
Back Extra: Remark, when \(\theta\) is a vector we would talk about "confidence sets"

Q: Definition. The prior distribution \(\phi(\theta)\) represents [...] Conditional on \(\theta\), the data X has a pdf (or pmf) \(f_X( \cdot | \theta)\). Having observed \(X = x\), we combine the information in the data with prior to "update" our beliefs. This is captured by the posterior distribution which is the conditional distribution of the parameter given \(X = x\): \[\pi (\theta | x) = [...]\]
A: the investigators beliefs or information about \(\theta\) before collecting any data.; \frac{\pi (\theta) f_X(x | \theta)}{f_X(x)}

Q: Definition. A simple hypothesis is one that [...]. Otherwise we call it a [...] A test of the null hypothesis \(H_0\) is defined by the critical region \(C \subseteq X\). [...]
A: fully specifies the distribution of X; composite hypothesis.; When \(X \in C\), we reject the null. When \(X \not \in C\), we say "we fail to reject the null" or "find no evidence against it"

Definition. In hypothesis testing we can make 2 errors:
-Type 1 error:  We reject \(H_0\) when \(H_0\) is true
-Type 2 error:  We fail to reject \(H_0\), when \(H_0\) is false
We define \[\alpha = \mathbb{P}_{H_0}(H_0 \text{ rejected}) = \mathbb{P}_{H_0} (X \in C)\] \[\beta = \mathbb{P}_{H_1}(H_0 \text{ not rejected}) = \mathbb{P}_{H_1}(X \not \in C)\] so these are the probabilities of Type 1 and Type 2 errors respectively. The size of the test is \(\alpha\) and the power is \(1 - \beta\)

Q: Definition. Let \(H_0\) and \(H_1\) be simple, with X having pdf \(f_i\) under \(H_i\), \(i = 0,1\). The likelihood ratio statistic is [...]
A: \[\Lambda_X(H_0; H_1) = \frac{f_1(X)}{f_0(X)}\]
Back Extra: So we reject the null hypothesis when this ratio goes above a certain number.

Q: Theorem. Suppose \(f_0, f_1\) are nonzero on the same sets and suppose there exists k such that the LRT with critical region \[C = \{x ; f_1(x)/ f_0(x) > k\}\] has size \(\alpha\). The, out of all tests with size \(\leq \alpha\), the test with the smallest \(\beta\) is the LRT.
A: Let \(\overline{C}\) be the complement of C. We know the LRT has : \[\alpha = \int_C f_0(x)dx\] \[\beta = \int_{\overline{C}} f_1 (x)dx\]
Let \(C^*\) be the critical region of another test of size \(\alpha^* \leq \alpha\).
WTP: \(\beta \leq \beta^*\) or \(\beta - \beta^* \leq 0\)

Q: Definition. For any test with critical region of the form \(\{x ; T(x) > k\}\), where T is some statistic, we usually report the p-value or observed significance level: \[p = [...]\] where \(x^*\) is the observed data.
A: \mathbb{P}_{H_0}(T(X) > T(x^*))
Back Extra: In English, this is the probability under the null of sampling an observation "more extreme" than \(x^*\), as measured by T.

Q: Proposition. Under \(H_0\), p-value is \(\text{Unif}[0,1]\)
A: Let F be distribution function of T \[\mathbb{P}_{H_0} (p < u) = \mathbb{P}_{H_0}(1 - F(T) < u) = \mathbb{P}_{H_0} (F(T) > 1 -u)\] \[ = \mathbb{P}_{H_0}(T > F^{-1}(1-u)) = 1 - F(F^{-1}(1-u)) = u\]

Q: Definition. The power function is [...] The size of a test with composite null \(H_0\) is the worst case Type 1 error probability, i.e. [...] We say that a test of \(H_0\) against \(H_1\) is uniformly most powerful of size \(\alpha\) if
[...]
A: \(W(\theta) = \mathbb{P}_{\theta}(X \in C)\).; \[\alpha = \sup_{\theta \in \Theta_0} W(\theta)\]; i) \(\sup_{\theta \in \Theta_0} W(\theta) = \alpha\)
ii) For any other test \(C^*\) of size \(\leq \alpha\), with power functions \(W^*\), we have \(W(\theta) \geq W^*(\theta)\) for all \(\theta \in \Theta_1\)

Q: Theorem. Wilks' Theorem. Suppose \(\Theta_0 \subset \Theta_1\) (null is special case of alternative) and \[\dim (\Theta_1) - \dim(\Theta_0) = p\]
If \(X = (X_1, \dots , X_n)\) iid, then under regularity conditions as \(n \to \infty\) [...]
A: the limiting distribution of \(2 \log \Lambda_X(H_0 ; H_1)\) under the null is \(\chi^2_p\)
Back Extra: So for any \(\theta \in \Theta_0\), any \(l > 0\) \[\mathbb{P}_{\theta}(2 \log \Lambda_X(H_0 ; H_1) \leq l) \to \mathbb{P}(\Xi \leq l)\] where \(\Xi ~ \chi^2_p\)

Q: Definition. Pearson statistic. Let \(o_i = N_i\) the "observed number of type i" and let \(e_i = n \hat{p_i}\) the "expected number of type i under null. Then Pearson's chi-square statistic is [...]
A: \[\sum_i \frac{(o_i - e_i)^2}{e_i}\]
Back Extra: This comes from \(2 \log \Lambda \approx \sum_i \frac{(o_i - e_i)^2}{e_i}\)

Q: Proposition. Under the null hypothesis of independence, \(H_0 : X_1 \perp Y_1\), the datasets \((X,Y), (X,Y_{\pi^1}, \dots, (X, Y_{\pi^B})\) are exchangeable, i.e. \[((X,Y), (X,Y_{\pi^1}), \dots , (X, Y_{\pi^B})) =^d \sigma ((X,Y), (X,Y_{\pi^1}), \dots , (X,Y_{\pi^B}))\] for any permutation \(\sigma\) of \(\{1, \dots , B+1\}\)
A: Simple but tedious proof which we skip

Q: Theorem. Define the acceptance region A of a test as the complement of the critical region. Let \(X \sim f_X( \cdot | \theta)\) for some \(\theta \in \Theta\).
(1) Suppose for each \(\theta_0 \in \Theta\) there is a test of \(H_0 : \theta = \theta_0\) of size \(\alpha\) with acceptance region \(A(\theta_0)\). Then the set \[I(X) = \{\theta : X \in A(\theta)\}\] is a \(100(1-\alpha)%\) confidence set
(2)Suppose \(I(X)\) is a \(100(1-\alpha)%\) confidence set for \(\theta\). Then \[A(\theta_0) = \{x : \theta_0 \in I(x) \}\] is the acceptance region of a size \(\alpha\) test for \(H_0 : \theta = \theta_0\) for each \(\theta_0 \in \Theta\)
A: Note that in each case \[\theta_0 \in I(X) \iff X \in A(\theta_0)\] For (1) we went to show
For (2) want to show

Q: Definition. We say X has a multivariable normal distribution if [...]
A: for any \(t \in \mathbb{R}^n\), the r.v. \(t^T X\) has a normal distribution

Q: Proposition. If X is a multivariate normal, then \(AX +b\) is multivariate normal.
A: Take any \(t \in \mathbb{R}^m\), then \[t^T(AX + b) = (A^T t)^T X + t^T b\] as X is MVN, \((A^T t)^T X \sim N(\mu, \sigma^2)\) for some \((\mu, \sigma^2)\). Then \(t^T (AX + b) \sim N(\mu + t^T b, \sigma^2)\). As \(t \in \mathbb{R}^m\) was arbitrary, \(AX + b\) must be MVN.

Q: Proposition. A MVN distribution is fully specified by its mean and variance.
A: Let \(X_1\) and \(X_2\) each be MVN with mean \(\mu \in \mathbb{R}^n\) and variance \(\Sigma \in \mathbb{R}^{n \times n}\). We will show their mgf's are equal; hence they have the same distribution. Take \(X_1\): \[\mathbb{E}[e^{1 \cdot t^T X_1}] = M_{t^T X_1}(1) = \exp (1 \cdot \mathbb{E}[t^T X_1] + \frac{1}{2} \text{Var}(t^T X_1) \cdot 1^2 ) = \exp (t^T \mu + \frac{1}{2} t^T \Sigma t)\] as this only depends on \(\mu\) and \(\Sigma\), we would obtain the same mgf for \(X_2\).

Q: Definition. We say \(P \in \mathbb{R}^{n \times n}\) is an orthogonal projection if it is:
[...]
A: (i) Idempotent : \(P^2 = P\)
(ii) Symmetric : \(P^T = P\)
Back Extra: Also if for all \(v \in \text{col}(P)\), \(P_v = v\), and for all \(w \in \text{col}(P)^{\perp}\)(i.e. \(w^T v = 0 \forall v \in \text{col}(P)\)), \(Pw = 0\)
We can show that these definition are equal pretty easily.

Q: Corollary. If P is an orthogonal projection, then so is \(I - P\)
A: It is clearly symmetric and \[(I-P)(I-P) = I - 2P + P^2 = I - P\]

Q: Proposition. If \(P \in \mathbb{R}^{n \times n}\) is an orthogonal projection then \[P = U U^T\] where the columns of \(U \in \mathbb{R}^{n \times k}\) form an orthonormal basis for \(\text{col}(P)\); and \(k = \text{rank}(P)\)
A: \(UU^T\) is clearly symmetric and idempotent and \(\text{col}(P) = \text{col}(UU^T)\).
Note that \(k = \text{rank}(P) = \text{Tr}(U^TU) = \text{Tr}(UU^T) = \text{Tr}(P)\)

Q: Theorem. If X is a MVN, \(X \sim N(0, \sigma^2 I)\), and P is an orthogonal projection matrix, then
i) \(PX \sim N(0, \sigma^2 P)\) and \((I-P)X \sim N(0, \sigma^2 (I-P))\) are independent
ii) \(\frac{|| PX||^2}{\sigma^2} \sim \chi^2_{\text{rank}(p)}\)
A: The vector \(\left ( \begin{matrix} P \\ I-P \end{matrix}\right ) X\) is MVN as it is a linear function of X. It's distribution is fully specified by its mean and variance which we can calculate.
Let \(Z \sim N(0, \sigma^2 P)\) and \(Z ' \sim N(0, \sigma^2 (I-P))\) be independent ; then \(\left ( \begin{matrix} Z \\ Z' \end{matrix} \right ) \sim N(0, \sigma^2 \left [ \begin{matrix} P & 0 \\ 0 & I-P \end{matrix} \right ])\). Hence \((Z,Z')^T\) and \((PX, (I-P)X)^T\) have the same distribution and PX, (I-P)X independent.
For (ii) note that \[\frac{||PX||^2}{\sigma^2} = \frac{(PX)^T PX}{\sigma^2} = \frac{X^T (UU^T)^T (UU^T) X}{\sigma^2}\] where columns of U form basis for \(\text{col}(P)\). \[= \frac{||U^TX||^2}{\sigma^2} = \sum_{i=1}^{\text{rank}(P)} \frac{(U^TX)_i^2}{\sigma^2}\] But \(U^TX \sim N(0, \sigma^2 U^TU) = N(0, \sigma^2 I)\), so \(\frac{(U^TX)_i}{\sigma}\) are iid \(N(0,1)\) for \(i = 1, \dots , \text{rank}(P)\). Hence \[||PX||^2 / \sigma^2 \sim \chi^2_{\text{rank}(P)} = \chi^2_{\text{Tr}(P)}\] as rank is trace for projections.

Q: Theorem. (i) \(\overline{X} \sim N(\mu, \sigma^2/n)\)
(ii) \(S_{XX} / \sigma^2 \sim \chi^2_{n-1}\)
(iii) \(\overline{X} , S_{XX}\) independent
A: Let P be a matrix such that \(PX = \vec{1}\overline{X}\) where \(\vec{1} = (1,1,\dots,1)^T \in \mathbb{R}^n\), i.e.
Easy to check \(P = P^T = P^2\), hence P is orthogonal projection. Now we can write \[X = (X_1, \dots , X_n)^T = \mu \vec{1} + \epsilon\] with \(\epsilon \sim N(0, \sigma^2 I)\)
Note : \(\overline{X}\) is a function of \(PX = \mu \vec{1} + P \epsilon\)
\(S_{XX} = \sum_i (X_i - \overline{X})^2 = ||X- \vec{1} \overline{X}||^2 = ||(I-P)X||^2 = ||(I-P)\epsilon||^2\) hence \(S_{XX}\) is a function of \((I-P)\epsilon\)
But \(P\epsilon \perp (I-P)\epsilon\) by previous theorem, hence \(\overline{X} \perp S_{XX}\). The theorem also implies \[\frac{S_{XX}}{\sigma^2} = \frac{||(I-P)\epsilon||^2}{\sigma^2} \sim \chi^2_{\text{Tr}(I-P)}\] where \(\text{Tr}(I-P) = n - \text{Tr}(p) = n -1\).
Finally \(\overline{X} = (PX)_1 = \mu + (P\epsilon)_1\) where \(P\epsilon \sim N(0, \sigma^2 P)\).
Hence \((P\epsilon)_1 \sim N(0, \sigma^2 P_{11}) = N(0, \sigma^2 /n)\) and \(\overline{X} \sim N(\mu, \sigma^2 / n)\)

Q: Theorem. Gauss-Markov. Let \(\beta^* = CY\) be any other linear estimator of \(\beta\), which is unbiased : \(\mathbb{E}[\beta^*] = \beta\). Then for any \(t \in \mathbb{R}^p\): \[\text{Var}(t^T \hat{\beta}) \leq \text{Var} (t^T \beta^*)\]where \(\hat{\beta}\) is the least squares estimator. So we say that the least squares estimator is the best linear unbiased estimator.
A: Note \[\text{Var}(t^T \beta^*) - \text{Var}(t^T \hat{\beta}) = t^T (\text{Var} \beta^* - \text{Var} \hat{\beta}) t \geq 0 \text{ for all t} \iff \text{Var}\beta^* - \text{Var} \hat{\beta} \text{ is positive semi-definite}\]Recall \(\beta^* = CY, \hat{\beta} = (X^T X)^{-1} X^T Y\). Let \(A = C - (X^T X)^{-1} X^T\). Note \[\mathbb{E}[AY] = \mathbb{E}[\beta^*] - \mathbb{E}[\hat{\beta}] = 0\] as they are both unbiased estimators. Also \[\mathbb{E}[AY] = A \mathbb{E}[Y] = AX\beta = 0\] for all \(\beta \in \mathbb{R}^p\) so \(AX = 0\). Now \[\text{Var}(\beta^*) = \text{Var}((A + (X^TX)^{-1} X^T)Y) = (A + (X^TX)^{-1} X^T) \text{Var}(Y) (A+(X^T X)^{-1} X^T)^T = \sigma^2 (AA^T + (X^T X)^{-1} + AX(X^TX)^{-1} + (X^TX)^{-1} X^T A^T)\]last two terms are 0 so \[= \sigma^2 AA^T + \text{Var}(\hat{\beta})\]So indeed \(\text{Var} \beta^* - \text{Var} \hat{\beta} = \sigma^2 AA^T\) is positive semi-definite.

Q: Proposition. \(P = X(X^TX)^{-1}X^T\) is the orthogonal projection onto col(X).
A: P is clearly symmetric, and \[P^2 = X(X^TX)^{-1} X^T X (X^T X)^{-1} X^T = X(X^TX)^{-1}X^T = P\] hence P is an orthogonal projection. Also \(\text{col}(P) = \text{col}(X)\), because for any a, \(Pa = X[(X^TX)^{-1}Xa]\) is a linear combination of columns in X, and conversely if \(b = Xc\) is in \(\text{col}(X)\), then \(b = Xc = X(X^TX)^{-1} X^T X c = Pb\) is also in \(\text{col}(P)\)

Q: Corollary. (i) \(\hat{\beta} \sim N(\beta, \sigma^2 (X^T X)^{-1})\)
(ii) \(\frac{\hat{\sigma}^2}{\sigma^2} \cdot n \sim \chi_{n-p}^2\)
(iii)\(\hat{\beta}, \hat{\sigma}^2\) are independent (!)
A: For (i) we already knew \(\mathbb{E}\hat{\beta} = \beta, \text{var} \hat{\beta} = \sigma^2 (X^TX)^{-1}\). Simply note that \(\hat{\beta} = (X^T X)^{-1} X^T Y\) is linear in \(Y \sim N(X \beta, \sigma^2 I)\); thus \(\hat{\beta}\) is also normal.
For (ii) note \[\frac{n}{\sigma^2} \hat{\sigma}^2 = \frac{||(I-P)Y||^2}{\sigma^2} = \frac{||(I-P)(X\beta + \epsilon)||^2}{\sigma^2} = \frac{||(I-P)\epsilon||^2}{\sigma^2} \sim \chi_{\text{Tr}(I - P)}^2\] by theorem
\(\text{Tr}(I-P) = n - \text{Tr}(P) = n - \text{rank}(P) = n-p\) this is because we assume X full rank.
For (iii) observe that \(\hat{\sigma}^2\) is a function of \((I-P)\epsilon\) and \[\hat{\beta} = (X^TX)^{-1} X^T Y = (X^T X)^{-1} X^T (X \beta + \epsilon) = \beta + (X^T X)^{-1} X^T \epsilon = \beta + (X^TX)^{-1}X^T P \epsilon\] so \(\hat{\beta}\) is a function of \(P\epsilon\). But \(P\epsilon \perp (I-P)\epsilon\) by the theorem, so \(\hat{\beta} \perp \hat{\sigma}^2\)