Title: Part II / Principles of Statistics
Tags: Likelihood::Inference Decision::Theory Nonparametric::Inference::and::Monte::Carlo::Techniques Multivariate::Analysis Bayesian::Inference

Q: Definition. A statistical model is [...]
A: a family of distributions \(\{P_{\theta} : \theta \in \Theta\}\) where \(\Theta\) is a parameter space

Q: Definition. We say \(\theta_0\) is identifiable if [...]
A: whenever \(\theta \in \Theta\) satisfies \(P_{\theta} = P_{\theta_0}\), we have \(\theta = \theta_0\)

Q: Definition. Suppose \(l = l(\cdot, X): \Theta \to \mathbb{R}\) is differentiable on \(\text{int} \Theta \subseteq \mathbb{R}^p\). The random function \(S : \text{int} \Theta \to \mathbb{R}^p\) given by \[[...]\]is known as the score function of score. In the (special case) where we have i.i.d data \(X_1 ,\dots, X_n\) we will notate this as \(S_n\)
A: S(\theta) := \nabla_{\theta} l(\theta) = \left ( \frac{\partial}{\partial \theta_1} l(\theta), \dots, \frac{\partial}{\partial \theta_p}l(\theta)\right)^T

Q: Lemma. Let \(g: \mathbb{R}^d \to \mathbb{R}\). Under appropriate regularity conditions we have that for \(\theta \in \text{int} \Theta\) \[\mathbb{E}_{\theta}[S(\theta)g(X)] = \nabla_{\theta} \mathbb{E}_{\theta} g(X)\]
A: \[\begin{align} \mathbb{E}_{\theta}[S(\theta)g(X)] &= \int \nabla_{\theta} \log (f(x, \theta))g(x)f(x,\theta) dx \\ &= \int \nabla_{\theta} f(x,\theta) g(x)dx \\ &= \nabla_{\theta} \int f(x,\theta) g(x) dx \\ &=\nabla_{\theta} \mathbb{E_{\theta}} g(X) \end{align}\]Taking g = 1 in the above, we see in particular that \(\mathbb{E}_{\theta} [S(\theta)] = 0\)

Q: Definition. The FIsher information matrix \(I(\theta) \in \mathbb{R}^{p\times p}\) for \(\theta \in \text{int} \Theta\) is given by the variance of the score \[[...]\]
A: I(\theta) := \text{Cov}_{\theta} (S(\theta)) = \mathbb{E}(S(\theta)S(\theta)^T)
Back Extra: Note that if \(X_1 ,\dots, X_n \sim^{\text{i.i.d}}f(\cdot, \theta)\) we have \(I_n(\theta) = nI_1(\theta)\) and the Fisher information tensorises.

Q: Theorem. Cramer-Rao Lower Bound. Suppose the model \(\{P_{\theta} : \theta \in \Theta\}\) where \(\Theta \subseteq \mathbb{R}^p\) is sufficiently 'regular'. For a function \(\phi : \Theta \to \mathbb{R}\), consider estimating \(\phi(\theta)\) with an estimator \(\hat{\phi}\). Then for any \(\theta \in \text{int} \Theta\) where \(I(\theta)\) is invertible, \[\text{Var}_{\theta}(\hat{\phi}) \geq \nabla_{\theta}\mathbb{E}_{\theta}(\hat{\phi})^T I(\theta)^{-1} \nabla_{\theta}\mathbb{E}_{\theta} (\hat{\phi})\]
A: Let \(v = I^{-1/2} (\theta)u \) for an arbitrary unit vector u. We have \[|v^T \nabla_{\theta} \mathbb{E}_{\theta} (\hat{\phi})| = |v^T \mathbb{E}_{\theta} (S(\theta)\hat{\phi})| = |\mathbb{E}_{\theta}\{v^T S(\theta) (\hat{\phi} - \mathbb{E}_{\theta}(\hat{\phi}))\}| \leq (\mathbb{E}_{\theta} \{(v^T S(\theta))^2\})^{1/2} \left(\text{Var}_{\theta} (\hat{\phi})\right )^{1/2}\]Rearranging, squaring and writing \(b := \nabla_{\theta}(\mathbb{E}_{\theta}(\hat{\phi}))\) for notational simplicity, we obtain \[\text{Var}_{\theta}(\hat{\phi}) \geq \frac{(v^T b)^2}{v^T I(\theta) v}\]Substituting \(v = I(\theta)^{-1/2} u\) we get \[\text{Var}_{\theta}(\hat{\phi}) \geq (b^T I(\theta)^{-1/2} u)^2\]This is true for all unit vectors u, so maximising over u by taking \[u = \frac{I(\theta)^{-1/2} b}{||I(\theta)^{-1/2} b||^2} = \frac{I(\theta)^{-1/2}b}{\sqrt{b^T I(\theta)^{-1} b}}\]we finally arrive at \[\text{Var}_{\theta} (\hat{\phi}) \geq b^T I(\theta)^{-1} b\] as required

Q: Proposition. Under regularity conditions we have \(I(\theta) = - \mathbb{E}_{\theta} [\nabla_{\theta}^2 \log (f(X, \theta))]\) for \(\theta \in \text{int}\Theta\). \[I_{jk}(\theta) = -\mathbb{E}_{\theta}\left ( \frac{\partial^2}{\partial \theta_j \partial \theta_k} \log (f(X,\theta)) \right )\]
A: We have \[\frac{\partial^2}{\partial \theta_j \partial \theta_k} \log(f(x,\theta)) = \frac{\partial}{\partial \theta_j} \left ( \frac{\frac{\partial}{\partial \theta_k} f(x,\theta)}{f(x,\theta)}\right) = \frac{\frac{\partial^2}{\partial \theta_j \partial \theta_k} f(x,\theta)}{f(x,\theta)} - \frac{\frac{\partial}{\partial \theta_J} f(x,\theta) \frac{\partial}{\partial \theta_k} f(x,\theta)}{f(x,\theta)^2}\]Now, interchanging differentiation and integration \[\int \frac{\partial^2}{\partial \theta_j \partial \theta_k} f(x,\theta) dx = \frac{\partial^2}{\partial \theta_j \partial \theta_k} \int f(x,\theta) dx =0\]And the result follows by taking expectations.

Q: Theorem. Consider a model \(\{f(\cdot, \theta) : \theta \in \Theta \}\) and suppose \(X \sim f(\cdot, \theta_0)\) where \(\theta_0\) is identifiable. Then \(\theta_0\) is the unique maximiser of \[\theta \mapsto \mathbb{E}_{\theta_0} \log f(X,\theta) = \int_{\chi} \log (f(x,\theta))f(x,\theta_0) d\theta\]where \(\chi = \{x : f(x,\theta_0) > 0\}\)
A: We make use of the fact that \(\log u \leq u-1\) for every \(u \geq 0\), so also \(\frac{1}{2}\log u \leq \sqrt{u} - 1\) \[\begin{align} \mathbb{E}_{\theta_0} \log f(X,\theta) - \mathbb{E}_{\theta_0} \log f(X,\theta_0) &= \mathbb{E}_{\theta_0} \log \left ( \frac{f(X,\theta)}{f(X,\theta_0)}\right ) \\ &\leq 2 \mathbb{E}_{\theta_0} \left ( \sqrt{\frac{f(x,\theta)}{f(x,\theta_0)}} - 1\right ) \\ &= 2 \int_{\chi} \sqrt{f(x,\theta) f(x,\theta_0)} dx - 2 \\ &= - \int_{\chi} (\sqrt{f(x,\theta) } - \sqrt{f(x,\theta_0)})^2 dx + \int_{\chi} f(x,\theta) dx -1 \\ & \leq 0\end{align}\]with equality if and only if \(f(\cdot , \theta) = f(\cdot, \theta_0)\) which occurs if and only if \(\theta = \theta_0\) by identifiability.

Q: Definition. Let \((X_n)_{n \in \mathbb{N}}\) and X be random vectors taking values in \(\mathbb{R}^d\).
(i) We say \(X_n\) converges to \(X\) almost surely as \(n \to \infty\) and write \(X_n \to^{a.s} X\) if \[[...]\] (ii) We say \(X_n\) converges to \(X\) in probability, and write \(X_n \to^p X\), if for all \(\epsilon > 0\), \[[...]\](iii)We say \(X_n\) converges in distribution and write \(X_n \to^d X\) if \[[...]\]at all points where \(t \mapsto \mathbb{P}(X \le t)\) is continuous
A: \mathbb{P}(\omega \in \Omega : || X_n(\omega) - X(\omega) ||_{\infty} \to 0) = \mathbb{P}(||X_n - X||_{\infty} \to 0) = 1; \mathbb{P}(||X_n - X||_{\infty} > \epsilon) \to 0; \mathbb{P}(X_n \le t) \to \mathbb{P}(X \le t)
Back Extra: \[X_n \to^{a.s} X \implies X_n \to^p X \implies X_n \to^d X\]

Q: Theorem. Continuous Mapping Theorem. Let \(g: \mathbb{R}^d \to \mathbb{R}^m\) be continuous at every point of a set C such that \(\mathbb{P}(X \in C) = 1\). Then \[[...]\]
A: X_n \to^{a.s / p / d} X \implies g(X_n) \to^{a.s / p/d} g(X)

Q: Lemma. Slutsky's Lemma. Suppose \(X_n \to^d X\) and \(Y_n \to^p c\) where \(c\) is deterministic
(i) If \(X_n\) and \(X\) are random vectors of the same dimension, \([...]\)
(ii)If \(Y_n\) is real-valued, \([...]\). Moreover if \(c \neq 0\) then \([...]\)
(iii)We also have the matrix version of (ii)
A: X_n + Y_n \to^d X +c; Y_n X_n \to^d cX; Y_n^{-1}X_n \to c^{-1}X

Q: Theorem. Dominated Convergence Theorem. Suppose a sequence of real-valued random variables \((X_n)_{n \in \mathbb{N}}\) satisfies \(X_n \to^p X\), and there exists a random variable \(W\) that dominates the \(X_n\) in the sense that \(|X_n(\omega)| \geq |X(\omega)|\) for all \(\omega \in \Omega\). Then[...]
A: \(\mathbb{E}|X| < \infty\) and \(\mathbb{E}X_n \to \mathbb{E}X\)

Q: Theorem. Strong Law of Large Numbers. Let \(X_1 ,\dots, X_n\) be i.i.d taking values in \(\mathbb{R}^d\) with \(\mathbb{E}||X_1||_{\infty} < \infty\). Then \[[...]\]
A: \overline{X}_n = \frac{1}{n}\sum_{i=1}^n X_i \to^{a.s} \mathbb{E}(X)

Q: Theorem. Weak Law of Large Numbers. Let \(X_1, \dots , X_n\) be i.i.d real-valued random variables with \(\text{Var}(X_1) < \infty\). Then \[\overline{X}_n \to^p \mathbb{E}[X_1]\]
A: Applying Markov's inequality to \(\{\overline{X}_n - \mathbb{E}(X_1)\}^2\) we have \[\mathbb{P}(\{ \overline{X}_n - \mathbb{E}(X_1)\}^2 > \epsilon^2) \le \epsilon^{-2} \mathbb{E}\{\overline{X}_n - \mathbb{E}(X_1)\}^2\]But,
\[\mathbb{E}\{\overline{X}_n - \mathbb{E}(X_1)\}^2 = \text{Var}(\overline{X}_n) = \text{Var}(X_1) / n\]so \[\mathbb{P}(|\overline{X}_n - \mathbb{E}(X_1)| > \epsilon) \le \epsilon^{-2}\text{Var}(X_1)/n \to 0\]as \(n \to \infty\)

Q: Theorem. Central Limit Theorem. Let \(X_1, \dots , X_n\) be i.i.d taking values in \(\mathbb{R}^d\) with finite variance \(\Sigma\). Then \[[...]\]
A: \sqrt{n}\{\overline{X}_n - \mathbb{E}(X_1)\} \to^d N_d(0, \Sigma)

Q: Theorem. Uniform Law of Large Numbers. \[[...]\]
A: \sup_{\theta \in \Theta} \left |\frac{1}{n} \sum_{i=1}^n g(X_i,\theta) - \mathbb{E}(g(X,\theta)) \right | \to^p 0

Q: Definition. We say an estimator \(\hat{\theta}_n = \hat{\theta}_n (X_1, \dots , X_n)\)(not necessarily the MLE) is consistent for estimating a parameter \(\theta_0\)(corresponding to the true distribution) if [...]
A: \(\hat{\theta}_n\to^p \theta_0\)

Q: Theorem. Suppose we have appropriate regularity conditions. An MLE exists (almost surely) and any MLE is consistent
A: First note that on the almost sure event \(\{X_1,\ldots,X_n \in \mathcal{X}\}, \bar{\ell}_n(\theta)\) is continuous, so a maximiser on the compact set \(\Theta\) exists. Also, the regularity assumptions accommodate the following ULLN:
\[\sup_{\theta\in\Theta} |\bar{\ell}_n(\theta) - \bar{\ell}(\theta)| \stackrel{p}{\to} 0.\]
Now fix \(\epsilon > 0\) and let \(\Theta_- := \{\theta \in \Theta: \|\theta-\theta_0\|_\infty \geq \epsilon\}\). Note that for any MLE \(\hat{\theta}_n\),
\[\{\bar{\ell}_n(\theta_0) > \sup_{\theta\in\Theta_-} \bar{\ell}_n(\theta)\} \subseteq \{\|\hat{\theta}_n - \theta_0\|_\infty < \epsilon\},\]
so it suffices to show that the former event has probability converging to 1.
Now \(\Theta_-\) is a closed and bounded (compact) set as the intersection of the compact set \(\Theta\) and the closed set \(\{\|\theta-\theta_0\|_\infty < \epsilon\}^c\). Recall that \(\bar{\ell}\) is continuous, so there exists \(\theta_- \in \Theta_-\) with \(\bar{\ell}(\theta_-) = \sup_{\theta\in\Theta_-} \bar{\ell}(\theta)\). Let us write \(\delta := \bar{\ell}(\theta_0) - \bar{\ell}(\theta_-) > 0\) (recall Theorem 4 which shows \(\theta_0\) is the unique maximiser of \(\bar{\ell}\)).
Also,
\[\sup_{\theta\in\Theta_-} \bar{\ell}_n(\theta) \leq \sup_{\theta\in\Theta_-} \bar{\ell}(\theta) + \sup_{\theta\in\Theta} |\bar{\ell}_n(\theta) - \bar{\ell}(\theta)|.\]
Now
\[\bar{\ell}_n(\theta_0) - \sup_{\theta\in\Theta_-} \bar{\ell}_n(\theta) = \bar{\ell}_n(\theta_0) - \bar{\ell}(\theta_0) + \underbrace{\bar{\ell}(\theta_0) - \bar{\ell}(\theta_-)}_{=\delta} + \bar{\ell}(\theta_-) - \underbrace{\sup_{\theta\in\Theta_-} \bar{\ell}_n(\theta)}_{\geq -\sup_{\theta\in\Theta} |\bar{\ell}_n(\theta)-\bar{\ell}(\theta)|}.\]
But by ULLN, \(\mathbb{P}(\sup_{\theta\in\Theta} |\bar{\ell}_n(\theta) - \bar{\ell}(\theta)| < \delta/2) \to 1\), so \(\mathbb{P}(\bar{\ell}_n(\theta_0) > \sup_{\theta\in\Theta_-} \bar{\ell}_n(\theta)) \to 1\) as required.

Q: Theorem. Suppose the appropriate regularity conditions hold. Then any sequence of MLEs \(\hat{\theta}_n\) satisfies \[\sqrt{n}(\hat{\theta}_n - \theta_0) \to^d N_p(0, I_1 (\theta_0)^{-1})\]
A: As \(\theta_0 \in \text{int}\,\Theta\), there exists some \(\epsilon > 0\) such that \(\{\theta : \|\theta-\theta_0\|_\infty \leq \epsilon\} \in \text{int}\,\Theta\). We already know \(\hat{\theta}_n \stackrel{p}{\to} \theta_0\) (Theorem 12), so writing \(A_n := \{\hat{\theta}_n \in \text{int}\,\Theta\} \supseteq \{\|\hat{\theta}_n - \theta_0\|_\infty \leq \epsilon\}\), we have \(\mathbb{P}(A_n) \to 1\). We henceforth work on this sequence of events,\(^9\) noting that it will not affect the distributional convergence result\(^{10}\).
Now fix \(j \in \{1,\ldots,p\}\) and define \(q(t) := S_n(\hat{\theta}_n + (1-t)\theta_0)_j\). Then, by the mean value theorem,
\[q(1) - q(0) = q'(t)\]
for some \(t \in [0,1]\). Thus there exists \(\tilde{\theta}_n^{(j)} \stackrel{p}{\to} \theta_0\) with
\[S_n(\hat{\theta}_n)_j - S_n(\theta_0)_j = \sum_{k=1}^p (\hat{\theta}_n - \theta_0)_k \sum_{i=1}^n \left.\frac{\partial^2}{\partial\theta_k\partial\theta_j} \log f(X_i,\theta)\right|_{\theta=\tilde{\theta}_n^{(j)}}.\]
Then, defining \(\tilde{J}_n \in \mathbb{R}^{p\times p}\) with \((\tilde{J}_n)_{jk} := (J_n(\tilde{\theta}^{(j)}))_{jk}\), we have
\[\frac{1}{n}S_n(\hat{\theta}_n) - \frac{1}{n}S_n(\theta_0) = -\tilde{J}_n(\hat{\theta}_n - \theta_0).\]
Now on \(A_n\), \(S_n(\hat{\theta}_n) = 0\). Also \(\tilde{J}_n \stackrel{p}{\to} I_1(\theta_0)\) (see Lemma 14 below), writing \(B_n := \{\tilde{J}_n \text{ is invertible}\}\), we have \(\mathbb{P}(B_n) \to 1\) (see Example Sheet 1, Question 11(i)). We now work on \(A_n \cap B_n\) (and note that \(\mathbb{P}(A_n \cap B_n) \to 1\)). We have
\[\sqrt{n}(\hat{\theta}_n - \theta_0) = \tilde{J}_n^{-1}\frac{1}{\sqrt{n}}S_n(\theta_0).\]
By the CMT, \(\tilde{J}_n^{-1} \stackrel{p}{\to} I_1(\theta_0)^{-1}\) and by the CLT, \(\frac{1}{\sqrt{n}}S_n(\theta_0) \stackrel{d}{\to} N_p(0,I(\theta_0))\). Thus by Slutsky, we get the result.

Q: Definition. The observed information matrix \(J_n(\theta) \in \mathbb{R}^{p \times p}\) is defined to be \[[...]\]
A: (J_n(\theta))_{jk} := -\frac{\partial}{\partial \theta_k} (\frac{1}{n}S_n(\theta))_j = -\frac{1}{n}\sum_{i=1}^n \frac{\partial^2}{\partial \theta_j \partial \theta_k} \log f(X_i, \theta)

Q: Lemma. Suppose that regularity conditions hold. Suppose that \(\tilde{\theta}_n^{(j)} \to^p \theta_0\) for \(j = 1, \dots ,p\). Then the matrix \(\tilde{J}_n \in \mathbb{R}^{p \times p}\) with \((\tilde{J}_n)_{jk} := (J_n(\tilde{\theta}^{(j)}))_{jk}\) satisfies \(\tilde{J}_n \to^p I_1 (\theta_0)\)
A: Fix \(j,k \in \{1,\ldots,p\}\) and define \(g(X_i,\theta) := -\frac{\partial^2}{\partial\theta_j\partial\theta_k} \log f(X_i,\theta)\). The conditions ensure that \(\theta \mapsto \mathbb{E}_{\theta_0}g(X,\theta) =: \bar{g}(\theta)\) is continuous at \(\theta_0\) and that we have a ULLN of the form
\[\sup_{\theta\in\Theta} \left|\frac{1}{n}\sum_{i=1}^n g(X_i,\theta) - \bar{g}(\theta)\right| \stackrel{p}{\to} 0.\]
Now
\[(\tilde{J}_n)_{jk} - I_1(\theta_0)_{jk} = \left(\frac{1}{n}\sum_{i=1}^n g(X_i,\tilde{\theta}^{(j)}_n) - \bar{g}(\tilde{\theta}^{(j)}_n)\right) + (\bar{g}(\tilde{\theta}^{(j)}_n) - \bar{g}(\theta_0)).\]
The first term converges to 0 in probability by the ULLN, and the second term converges to 0 in probability by the CMT. Thus \((\tilde{J}_n)_{jk} - I_1(\theta_0)_{jk} \stackrel{p}{\to} 0\) by Slutsky.

Q: Theorem. Wald confidence reagion. Under regularity conditions \[W_n(\theta_0) := n(\hat{\theta}_n - \theta_0)^T J_n (\hat{\theta}_n)(\hat{\theta}_n - \theta_0) \to^d \chi_p^2\]
A: We have
\[W_n(\theta_0) = n(\hat{\theta}_n - \theta_0)^T I_1(\theta_0)(\hat{\theta}_n - \theta_0) + \sqrt{n}(\hat{\theta}_n - \theta_0)^T \{(\tilde{J}_n(\hat{\theta}_n) - I_1(\theta_0))\}\sqrt{n}(\hat{\theta}_n - \theta_0).\]
But we know \((\tilde{J}_n(\hat{\theta}_n) - I_1(\theta_0)) \stackrel{p}{\to} 0\), so by Slutsky, \(\{(\tilde{J}_n(\hat{\theta}_n) - I_1(\theta_0))\}\sqrt{n}(\hat{\theta}_n - \theta_0) \stackrel{p}{\to} 0\), and hence by Slutsky again, the second term above converges to 0 in probability.
For the first term, note that \(I_1(\theta_0)^{1/2}\sqrt{n}(\hat{\theta}_n - \theta_0) \stackrel{d}{\to} N_p(0,I) \) by the CMT, so by the CMT again, \(\{I_1(\theta_0)^{1/2}\sqrt{n}(\hat{\theta}_n - \theta_0)\}^2 \stackrel{d}{\to} \chi_p^2\), so the result follows by a final application of Slutsky.

Q: Theorem. Wilk's(\(\Theta_0 = \{\theta_0\}\)). Under regularity conditions we have \[\Lambda_n \to^d \chi_p^2\]where \(\Lambda_n(\Theta, \Theta_0) := 2\log \left (\frac{\sup_{\theta \in \Theta}L_n(\theta)}{\sup_{\theta \in \Theta_0}L_n(\theta)} \right )\)
A: Let \(\hat{\theta}_n\) be the MLE. We have that
\[\Lambda_n = 2\{\ell_n(\hat{\theta}_n) - \ell_n(\theta_0)\}.\]
We now Taylor expand \(\ell_n(\theta_0)\) about \(\hat{\theta}_n\) using a mean-value form of the remainder. As in the proof of Theorem 13, we work on the events \(A_n := \{\hat{\theta}_n \in \text{int}\ \Theta\}\), which have probability converging to 1. We have
\[\ell_n(\theta_0) = \ell_n(\hat{\theta}_n) + (\theta_0 - \hat{\theta}_n)^T S_n(\hat{\theta}_n) - \frac{n}{2}(\theta_0 - \hat{\theta}_n)^T J_n(\tilde{\theta}_n)(\theta_0 - \hat{\theta}_n),\]
where \(\tilde{\theta}_n\) is on the closed line segment between \(\theta_0\) and \(\hat{\theta}_n\), so in particular \(\tilde{\theta}_n \stackrel{p}{\to} \theta_0\). Thus
\[\Lambda_n = \sqrt{n}(\theta_0 - \hat{\theta}_n)^T I_1(\theta_0)(\theta_0 - \hat{\theta}_n) \stackrel{d}{\to} \chi_p^2,\]
where we have used the CMT, Lem. 14, and that \(\sqrt{n}(\hat{\theta}_n - \theta_0) \stackrel{d}{\to} N_p(0,I_1(\theta_0)^{-1})\).

Q: What is the score test and how can we use it?
A: We know that \[\frac{1}{\sqrt{n}}S_n(\theta_0) \to^d N_p(0,I_1(\theta_0))\]so by the CMT \[\lambda_n := \frac{1}{n}S_n(\theta_0)^TI_1(\theta_0)^{-1} S_n(\theta_0) \to^d \chi_p^2\]

Q: Proposition. Let \(\hat{\theta}\) be an MLE. Then \(\phi(\hat{\theta})\) maximises \(L^*(\psi)\) over \(\psi \in \Phi(\Theta) := \{\psi = \phi(\theta) \text{ for some } \theta \in \Theta\}\).
A: Suppose for a contradiction there exists \(\psi \in \Phi(\Theta)\) with \(L^*(\psi) > L^*(\phi(\hat{\theta})) + \epsilon\) for some \(\epsilon > 0\). Then there exists \(\tilde{\theta}\) such that \(L(\tilde{\theta}) > L^*(\psi) - \epsilon > L^*(\phi(\hat{\theta})) > L(\hat{\theta})\), contradicting the optimality of \(\hat{\theta}\).

Q: Theorem. Delta method. Let \(\phi : \Theta \rightarrow \mathbb{R}^m\) be continuously differentiable at \(\theta_0\). Let \(\hat{\theta}_n\) be a sequence of random vectors (estimators—not necessarily MLEs) such that \[ r_n (\hat{\theta}_n - \theta_0) \xrightarrow{d} Z \] where \(Z\) is some random vector and \(r_n \to \infty\) is some deterministic scalar sequence (e.g. \(r_n = \sqrt{n}\)). Then \[ r_n (\phi(\hat{\theta}_n) - \phi(\theta_0)) \xrightarrow{d} \begin{pmatrix} \nabla_\theta \phi_1(\theta_0)^\top \\ \vdots \\ \nabla_\theta \phi_m(\theta_0)^\top \end{pmatrix} Z. \]
A: By the mean value theorem applied to each component, \[ r_n (\phi(\hat{\theta}_n) - \phi(\theta_0)) = \begin{pmatrix} \nabla_\theta \phi_1(\tilde{\theta}_n^{(1)})^\top \\ \vdots \\ \nabla_\theta \phi_m(\tilde{\theta}_n^{(m)})^\top \end{pmatrix} r_n (\hat{\theta}_n - \theta_0), \] where we define this matrix as \( D_n \):
for some \(\tilde{\theta}_n^{(k)}\), \(k = 1, \dots, m\) in the closed line segment between \(\theta_0\) and \(\hat{\theta}_n\). Now \(\hat{\theta}_n \xrightarrow{p} \theta_0\) (see Example Sheet 1, Question 8(b)), so \(\tilde{\theta}_n^{(k)} \xrightarrow{p} \theta_0\) also. But then by the continuous mapping theorem (CMT), \(\nabla_\theta \phi_k(\tilde{\theta}_n^{(k)}) \xrightarrow{p} \nabla_\theta \phi_k(\theta_0)\) for each \(k\), so by Slutsky, \[ D_n r_n (\hat{\theta}_n - \theta_0) \xrightarrow{d} \begin{pmatrix} \nabla_\theta \phi_1(\theta_0)^\top \\ \vdots \\ \nabla_\theta \phi_m(\theta_0)^\top \end{pmatrix} Z. \]

Q: If we have a prior density \(\pi\), what is the posterior density given by?
A: \[\theta | X \sim \Pi(\theta | x) := \frac{f(x,\theta)\pi(\theta)}{\int_{\Theta} f(x,\theta')\pi(\theta') d\theta '}\]

Q: Definition. In a statistical model \(\{f(\cdot, \theta) : \theta \in \Theta\}\), if the prior \(\pi(\theta)\) and the posterior \(\Pi(\theta|X)\)belong to the same family of distributions, the prior is called [...]
A: a conjugate pair.

Q: Definition. The prior \(\pi(\theta) \) proportional to[...] is called the Jeffreys prior.
A: \(\sqrt{\det I(\theta)}\)

Q: How can we perform inference about \(\theta\) using the posterior?
A: Posterior mean: \[\overline{\theta} = \overline{\theta}(X) = \mathbb{E}(\theta | X) = \int_{\theta \in \Theta} \theta \Pi (\theta | X) d\theta\]Uncertainty quantification: Any subset \(\hat{C} = \hat{C}(X)\subseteq \Theta \)\[\int_{\hat{C}} \Pi (\theta | X) d\theta = \mathbb{P}(\theta \in \hat{C} | X) = 1- \alpha\]is a \((1-\alpha)-\)level credible set for \(\theta\)
Hypothesis testing: Can use the Bayes factor \[\frac{\mathbb{P}(\theta \in \Theta_0 | X)}{\mathbb{P}(\theta \in \Theta_1 | X)} = \frac{\int_{\Theta_0} f(X,\theta)\pi(\theta) d\theta}{\int_{\Theta_1} f(X,\theta)\pi(\theta) d\theta}\]

Q: What is the Bernstein-von Mises theorem?
A: \[\int |\Pi (\theta | X_1 , \dots , X_n) - \hat{\phi}_n(\theta)|d\theta \to^{a.s} 0 \]where \(\hat{\phi}_n\) is the random \(N_p(\hat{\theta}_n, I_n(\theta_0)^{-1})\) density.
The posterior distributino converges to a multivariate normal distribution centered at the MLE with covariance given by the Fisher information

Q: Definition. Given a statistical model \(\{f(\cdot, \theta): \theta \in \Theta\}\) and data \(X \in \mathcal{X}\), we can cast many statistical tasks as decision problems, where our goal is to find an appropriate decision rule [...]
A: \(\delta : \mathcal{X \to A}\) where \(\cal{A}\) is a set of actions.
Back Extra: Hypothesis testing: \(\cal{A} = \{0,1\}\) with \(\delta\) a test function
Estimation: \(\cal{A} = \Theta\) and \(\delta(X) = \hat{\theta}(X)\) is an estimator.

Q: Definition. To measure the quality of an action, we can use a loss function \[[...]\]
A: L: \mathcal{A} \times \Theta \to [0,\infty)
Back Extra: Hypothesis testing: For testing simple hypotheses we may take \(L(a,\theta) = \mathbb{1}_{\{a \neq \theta\}}\)
Estimation: \(L(a,\theta) = (a-\theta)^2\) in one dimension

Q: Definition. To assess the performance of a decision rule, we can consider its average loss or risk \[[...]\]
A: R(\delta, \theta) := \mathbb{E}_{\theta}[L(\delta(X),\theta] = \int_{\mathcal{X} } L(\delta(x), \theta)f(x,\theta) dx
Back Extra: Hypothesis testing: \(R(\delta, \theta) = \mathbb{P}_{\theta}(\delta(X) \neq \theta)\) is either the probability of a type I error or type II error, depending on the value of \(\theta\)
Estimation: The quadratic risk also known as the mean squared error \[R(\hat{\theta}, \theta) = \mathbb{E}_{\theta}(\hat{\theta} - \theta)^2\]

Q: Definition. Given a prior density \(\pi\) on \(\Theta\) and model \(\{f(\cdot, \theta): \theta \in \Theta\}\), the \(\pi-\)Bayes risk of a decision rule \(\delta\) is \[[...]\]where in the final equality both \(\theta\) and \(X\) are random, \(X|\theta\sim f(\cdot, \theta)\) and the marginal density of \(\theta\) is \(\pi\).
A: R_{\pi}(\delta):= \int_{\Theta} R(\delta, \theta)\pi (\theta) d\theta = \mathbb{E}(L(\delta(X) , \theta))
Back Extra: Any minimiser of the \(\pi\)-Bayes risk is called a \(\pi-\)Bayes rule

Q: Proposition. Suppose \(\hat{\theta} = \hat{\theta}(X)\) is an unbiased estimator of \(\theta\), so \(\mathbb{E}_{\theta}(\hat{\theta}) = \mathbb{E}(\hat{\theta}|\theta) = \theta\). If \(\hat{\theta}\) is also \(\pi\)-Bayes, for some prior \(\pi\) in the quadratic risk, then \[\mathbb{P}(\hat{\theta} = \theta) = \int \mathbb{1}_{\{\hat{\theta}(x) =\theta'\}} f(x,\theta')\pi(\theta') d\theta' dx = 1\]
A: It suffices to show that \[\mathbb{E}(\hat{\theta} - \theta)^2 = \mathbb{E}(\hat{\theta}^2) - 2 \mathbb{E}(\theta \hat{\theta}) + \mathbb{E}(\theta^2) = 0\]But \[\mathbb{E}(\theta \hat{\theta}) = \mathbb{E}\{\theta \mathbb{E}(\hat{\theta}| \theta)\} = \mathbb{E}(\theta^2)\]and also \(\hat{\theta} = \mathbb{E}(\theta | X)\) almost surely, so \[\mathbb{E}(\theta \hat{\theta}) = \mathbb{E}\{\theta \mathbb{E}(\theta | X)\} = \mathbb{E}[\mathbb{E}\{\theta \mathbb{E}(\theta | X) | X\} ] = \mathbb{E}(\hat{\theta}^2)\]We therefore see that it holds.

Q: Definition. The minimax risk is [...]
A: the infimum(over all possible decision rules) of the maximal risk over the parameter space \(\Theta\): \[\inf_{\delta} \sup_{\theta \in \Theta} R(\delta, \theta)\]
Back Extra: A decision rule \(\delta\) that has maximal rusk \(\sup_{\theta \in \Theta} R(\delta, \theta)\) attaining the minimax risk is said to be minimax.

Q: Theorem. Let \(\pi\) be a prior on \(\Theta\) and suppose \(\delta_{\pi}\) is a decision rule such that \[R_{\pi}(\delta_{\pi}) = \sup_{\theta \in \Theta} R(\delta_{\pi}, \theta)\](In particular this will occur if \(\delta_{\pi}\) has constant risk in \(\theta\)). If \(\delta_{\pi}\) is a (unique) \(\pi\)-Bayes rule, then it is a (unique) minimax rule.
A: We gave that for any decision rule \(\delta\), \[\sup_{\theta \in \Theta} R(\delta, \theta) \ge \int_{\Theta} R(\delta, \theta)\pi(\theta)d\theta = R_{\pi}(\delta) \ge R_{\pi}(\delta_{\pi}) = \sup_{\theta \in \Theta}R(\delta_{\pi}, \theta)\]so \(\inf_{\delta} \sup_{\theta \in \Theta} R(\delta, \theta) = \sup_{\theta \in \Theta} R(\delta_{\pi}, \theta)\). If \(\delta_{\pi}\) is a unique \(\pi-\)Bayes rule, then the second inequality above would be strict for \(\delta \neq \delta_{\pi}\), so no such \(\delta\) can have maximal risk equal to the minimax risk.

Q: Definition. A prior \(\pi\) is least favourable if [...]
A: given a \(\pi\)-Bayes estimator \(\delta_{\pi}\), for any other prior \(\lambda\) and \(\lambda-\)Bayes estimator \(\delta_{\lambda}\), we have \(R_{\lambda}(\delta_{\lambda}) \le R_{\pi}(\delta_{\pi})\)

Q: Definition. A decision rule \(\delta\) is inadmissable if there exists another decision rule \(\delta '\) that dominates \(\delta\) in the sense that \[[...]\]
A: R(\delta ', \theta ) \le R(\delta, \theta) \forall \theta \in \Theta \; \; \; \text{ and }\;\;\; R(\delta', \theta) < R(\delta, \theta) \text{ for some } \theta \in \Theta
Back Extra: Otherwise the rule is admissable.

Q: Proposition. If for a prior \(\pi\) the \(\pi\)-Bayes rule is unique, then it is admissible.
A: Let \(\delta_\pi\) be \(\pi\)-Bayes and suppose decision rule \(\delta\) satisfies \(R(\delta, \theta) \leq R(\delta_\pi, \theta)\) for all \(\theta \in \Theta\). Then \[ R_\pi(\delta) = \int_{\Theta} R(\delta, \theta) \pi(\theta) \, d\theta \leq \int_{\Theta} R(\delta_\pi, \theta) \pi(\theta) \, d\theta = R_\pi(\delta_\pi) \] so \(\delta'\) is \(\pi\)-Bayes and hence \(\delta' = \delta\) by uniqueness.

Q: Proposition. If decision rule \(\delta\) is admissible and has constant risk, it is minimax.
A: If \(\delta\) is not minimax, then there exists \(\delta'\) with \[ \sup_{\theta} R(\delta', \theta) < \sup_{\theta} R(\delta, \theta) = \inf_{\theta} R(\delta, \theta), \] contradicting admissibility of \(\delta\).

Q: Theorem. Consider the model \(X_1,\dots, X_n \sim^{i.i.d}N(\theta,1)\) where \(\theta \in \Theta = \mathbb{R}\). Then \(\hat{\theta} = \hat{\theta}(X_1,\dots , X_n) = \overline{X}\) is admissable and minimax for estimating \(\theta\) in quadratic risk.
A: The MLE has constant risk \( R(\hat{\theta}, \theta) = \mathbb{E}_{\theta} ( \overline{X} - \theta )^2 = 1/n \). Thus it suffices to show \(\hat{\theta}\) is admissible. Now for any estimator \(\tilde{\theta}\), we have the variance–bias decomposition \[ R(\tilde{\theta}, \theta) = b(\theta)^2 + \operatorname{Var}_{\theta} (\tilde{\theta}) \geq b(\theta)^2 + \frac{1}{n} (1 + b'(\theta))^2 \] using the Cramér–Rao lower bound for the final inequality (noting that the Gaussian model satisfies the regularity conditions). Suppose now that \( R(\tilde{\theta}, \theta) \leq R(\hat{\theta}, \theta) \), so \[ b(\theta)^2 + \frac{1}{n} (1 + b'(\theta))^2 \leq \frac{1}{n}. \tag{3.2} \] We see that \( b \) is bounded from above and below, so by the mean value theorem, there exist sequences \( \theta_n^+ \to \infty \) and \( \theta_n^- \to -\infty \) with \( b'(\theta_n^+) \to 0 \) and \( b'(\theta_n^-) \to 0 \). But from (3.2) then \( b(\theta_n^+) \to 0 \) and \( b(\theta_n^-) \to 0 \). Since from (3.2) we know that \( b \) is nondecreasing \(( b'(\theta) \leq 0 )\), this implies that \( b(\theta) = 0 \) for all \( \theta \in \mathbb{R} \), so \( R(\tilde{\theta}, \theta) = 1/n = R(\hat{\theta}, \theta) \) and \( \hat{\theta} \) is admissible.

Q: Definition. The James-Stein estimator is given by \[[...]\]
A: \hat{\theta}_{JS} = \left ( 1 - \frac{p-2}{||X||^2} \right ) X

Q: Lemma. Stein's. Let \( Z \sim N(\mu, 1) \) and let \( g : \mathbb{R} \to \mathbb{R} \) be a bounded, differentiable function such that \( \mathbb{E}[g'(Z)] < \infty \). Then \[ \mathbb{E}[(Z - \mu) g(Z)] = \mathbb{E}[g'(Z)]. \]
A: \[ \mathbb{E}[(Z - \mu) g(Z)] = \frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} g(z)(z - \mu) \exp \left( -\frac{1}{2} (z - \mu)^2 \right) \, dz \] \[ = -\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} g(z) \left( \frac{d}{dz} \exp \left( -\frac{1}{2} (z - \mu)^2 \right) \right) dz \] \[ = -\frac{1}{\sqrt{2 \pi}} \left[ g(z) \exp \left( -\frac{1}{2} (z - \mu)^2 \right) \right]_{-\infty}^{\infty} + \frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} g'(z) \exp \left( -\frac{1}{2} (z - \mu)^2 \right) \, dz \] \[ = \mathbb{E}[g'(Z)], \] using the boundedness of \( g \) for the final equality.

Q: Theorem. Let \( X \sim N_p(\theta, I) \) for \( p \geq 3\). The risk of the James--Stein estimator satisfies \( R(\hat{\theta}_{\text{JS}}, \theta) < p \) for all \( \theta \), so in particular the MLE is inadmissible.
A: In the below, we drop the subscript \( \theta \) in the expectations for simplicity. We have \[ R(\hat{\theta}_{JS}, \theta) = \mathbb{E}\|\hat{\theta}_{JS} - \theta\|^2 = \mathbb{E}\left\|X - \theta - \frac{p-2}{\|X\|^2} X\right\|^2 \] \[ = p + (p-2)^2 \mathbb{E}\|X\|^{-2} - 2(p-2) \sum_{j=1}^p \mathbb{E}\left(\frac{X_j (X_j - \theta_j)}{\|X\|^2}\right)\] Consider the final term. We have, by the tower property, \[ \mathbb{E}\left(\frac{X_j (X_j - \theta_j)}{\|X\|^2}\right) = \mathbb{E}\left[\mathbb{E}\left(\frac{X_j (X_j - \theta_j)}{\|X\|^2} \,\middle|\, X_{-j}\right)\right], \] where \( X_{-j} = (X_1, \ldots, X_{j-1}, X_{j+1}, \ldots, X_p) \). We can write the RHS as \( \mathbb{E}[(X_j - \theta_j) g_j(X_j) \mid X_{-j}] \), where \[ g_j(x) = \frac{x}{x^2 + \|X_{-j}\|^2}, \] which is bounded provided \( \|X_{-j}\|^2 \neq 0 \), which occurs almost surely. Also \[ g_j'(x) = \frac{\|X_{-j}\|^2 - x^2}{(x^2 + \|X_{-j}\|^2)^2}. \] This is similarly bounded, so applying Stein’s lemma (conditionally) we get \[ \mathbb{E}[(X_j - \theta_j) g_j(X_j) \mid X_{-j}] = \mathbb{E}[g_j'(X_j) \mid X_{-j}] \] \[ = \mathbb{E}\left(\frac{\|X_{-j}\|^2 - 2X_j^2}{\|X\|^4} \,\middle|\, X_{-j}\right). \] Thus \[ \sum_{j=1}^p \mathbb{E}\left(\frac{X_j (X_j - \theta_j)}{\|X\|^2}\right) = \sum_{j=1}^p \mathbb{E}\left(\frac{\|X\|^2 - 2X_j^2}{\|X\|^4}\right) = (p-2) \mathbb{E}\|X\|^{-2}. \] Returning to (3.3), we thus have \[ R(\hat{\theta}_{JS}, \theta) = p - (p-2) \mathbb{E}\|X\|^{-2} < p. \]

Q: What is a classification problem?
A: We have an input \(X \in \chi\) and a binary outcome \(Y \in \{0,1\}\)

Q: Proposition. A Bayes classifier is given by \[ \delta_\pi(x) = \begin{cases} 1 & \text{if } \frac{f_1(x)\pi_1}{f_0(x)\pi_0} > 1, \\ 0 & \text{otherwise}. \end{cases} \] If \[ \mathbb{P}\left( \frac{f_1(X)\pi_1}{f_0(X)\pi_0} = 1 \right) = 0, \] then any Bayes classifier \( \delta \) satisfies \[ \mathbb{P}(\delta(X) = \delta_\pi(X)) = 1. \]
A: Example sheet

Q: Definition. The correlation is given by \[[...]\]
A: \rho_{jk} = \text{Corr}(X^{(j)}, X^{(k)}) = \frac{\text{Cov}(X^{(j)}, X^{(k)})}{\sqrt{\text{Var} (X^{(j)}) \text{Var}(X^{(k)})} } = \frac{\Sigma_{jk} }{\sqrt{\Sigma_{jj}\Sigma_{kk} } }

Q: Definition. Given random vectors \( X, Y \) and \( Z \), we say \( X \) is conditionally independent of \( Y \) given \( Z \), and write \( X \perp\!\!\!\perp Y \mid Z \), if \[ [...]\]
A: f_{XY|Z}(x, y \mid z) = f_{X|Z}(x \mid z) f_{Y|Z}(y \mid z) \quad \text{whenever } f_Z(z) > 0.

Q: Proposition. Suppose \[ (X, W) \sim N_{p+q} \left( \begin{pmatrix} \mu_1 \\ \mu_2 \end{pmatrix}, \begin{pmatrix} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{pmatrix} =:\Sigma \right), \] where \(\Sigma\) is positive definite. Then \[ X \mid W = w \sim N_p(\mu_1 + \Sigma_{12}\Sigma_{22}^{-1}(w - \mu_2), \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}). \]
A: Our idea is to write \(X = MW + (X - MW)\) for a matrix \(M \in \mathbb{R}^{p \times q}\) such that \(X - MW \perp\!\!\!\perp W\). Since jointly Gaussian random vectors are independent if and only if they are uncorrelated, this is equivalent to asking for \[ 0 = \mathrm{Cov}(W, X - MW) = \Sigma_{21} - \Sigma_{22}M^\top, \] which occurs when we take \(M^\top = \Sigma_{22}^{-1}\Sigma_{21}\). Because \(X - MW \perp\!\!\!\perp W\), the distribution of \(X - MW\) conditional on \(W = w\) is equal to its unconditional distribution. As a linear transformation of the Gaussian random vector \((X, W)\), \(X - MW\) is Gaussian and hence its distribution is characterized by its mean and variance, which we now compute: \[ \mathbb{E}(X - MW) = \mu_1 - \Sigma_{12}\Sigma_{22}^{-1}\mu_2, \] \[ \mathrm{Cov}(X - MW) = \Sigma_{11} + \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{22}\Sigma_{22}^{-1}\Sigma_{21} - 2\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21} = \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}. \] On the other hand, \(MW\) is simply equal to \(Mw\) conditional on \(W = w\). Putting things together gives the result.

Q: Proposition. In the setting above, let \( Q \in \mathbb{R}^{n \times n} \) be the orthogonal projection matrix onto the column space of \( W := (1, X, Z) \in \mathbb{R}^{n \times (p+2)} \), assumed to have full column rank \( p+2 \). Then the \( F \)-statistic for testing the significance of \( X \) in a normal linear model of \( Y \) on \( (1, X, Z) \) is
\[\frac{\| (Q - P)Y \|^2}{\frac{1}{n - p - 2} \| (I - Q)Y \|^2} = (n - p - 2) \frac{\hat{\rho}^2_{X, Y | Z}}{1 - \hat{\rho}^2_{X, Y | Z}}.\]
In particular, under the null \( H_0 : X \perp Y \mid Z \),
\[(n - p - 2) \frac{\hat{\rho}^2_{X, Y | Z}}{1 - \hat{\rho}^2_{X, Y | Z}} \sim F_{1, n - p - 2}.\]
A: Recall (from IB Statistics) that \( Q - P \) is an orthogonal projection with rank 1, so \( Q - P = vv^\top / \| v \|^2 \) for some vector \( v \in \mathbb{R}^n \) with \( v \neq 0 \). Also, since
\[I - P = (I - Q) + (Q - P),\]
and both \( Q \) and \( P \) have columns in the column space of \( W \), \( (I - Q)(Q - P) = 0 \), so
\[\| (I - P)Y \|^2 = \| (I - Q)Y \|^2 + \| (Q - P)Y \|^2.\]
Now \( v \) is the only eigenvector of \( Q - P \) (up to a multiplicative constant), and \( (Q - P)X = (I - P)X \), so \( (Q - P)(I - P)X = (I - P)X \). Thus
\[Q - P = \frac{(I - P)XX^\top (I - P)}{\| (I - P)X \|^2},\]
so
\[\| (Q - P)Y \|^2 = \frac{\big( X^\top (I - P)Y \big)^2}{\| (I - P)X \|^2} = \| (I - P)Y \|^2 \hat{\rho}^2_{X, Y | Z}.\]
Thus
\[\frac{\| (Q - P)Y \|^2}{\| (I - Q)Y \|^2} = \frac{\hat{\rho}^2_{X, Y | Z}}{1 - \hat{\rho}^2_{X, Y | Z}},\]
which gives the result.

Q: How do we carry out principal component analysis?
A: 1. Form the sample covariance matrix\[\hat{\Sigma} = \frac{1}{n} \sum_{i=1}^n (X_i - \overline{X}) (X_i - \overline{X})^T\]2. Find k unit norm eigenvectors corresponding to the k largest eigenvalues
3. Write \(\hat{V} \in \mathbb{R}^{p \times k}\) for the matrix with the jth column as the jth eigenvector and set \(U_i = \hat{V}^T X_i\) so then \(U_1 , \dots , U_n\) forms the compressed data

Q: Definition. Let \(\hat{\theta}_n^{(-i)} := T_{n-1}(X_1, \dots , X_{i-1}, X_{i+1}, \dots, X_n)\). The jackknife bias estimate is defined as \[[...]\]The jackknife bias-corrected estimate of \(\theta\) is then \[[...]\]
A: \tilde{B}_n := (n-1)\left ( \frac{1}{n} \sum_{i=1}^n \hat{\theta}_n^{(-i)} - \hat{\theta}_n \right ); \tilde{\theta}_n := \hat{\theta}_n - \tilde{B}_n

Q: Theorem. Suppose the bias \( B_n := \mathbb{E}_{\theta_0}(\hat{\theta}_n) - \theta_0 \) satisfies \[ B_n = \frac{a}{n} + \frac{b}{n^2} + O(n^{-3}) \] for some \( a, b \in \mathbb{R} \). Then \[ \mathbb{E}_{\theta_0}(\tilde{\theta}_n) = \theta_0 + O(n^{-2}). \]
A: Observe that\[ \tilde{\theta}_n = n \hat{\theta}_n - \frac{n-1}{n} \sum_{i=1}^n \hat{\theta}_n^{(-i)}. \]Thus,\[ \mathbb{E}_{\theta_0}(\tilde{\theta}_n) = n (\theta_0 + B_n) - \frac{n-1}{n} \sum_{i=1}^n (\theta_0 + B_{n-1}). \]Expanding \( B_n \) and \( B_{n-1} \), we get\[ \mathbb{E}_{\theta_0}(\tilde{\theta}_n) = n \left( \theta_0 + \frac{a}{n} + \frac{b}{n^2} + O(n^{-3}) \right) - \frac{n-1}{n} \left( n (\theta_0 + \frac{a}{n} + \frac{b}{n^2} + O(n^{-3})) \right). \]Simplifying, this becomes:\[ \mathbb{E}_{\theta_0}(\tilde{\theta}_n) = \theta_0 + \left(a + \frac{b}{n}\right) - \left(a + \frac{b}{n-1}\right) + O(n^{-2}). \]Since \( \frac{1}{n-1} = \frac{1}{n} + O(n^{-2}) \), the difference simplifies to:\[ \mathbb{E}_{\theta_0}(\tilde{\theta}_n) = \theta_0 + \frac{b}{n} - \frac{b}{n-1} + O(n^{-2}), \]which gives:\[ \mathbb{E}_{\theta_0}(\tilde{\theta}_n) = \theta_0 + O(n^{-2}), \]as required

Q: Definition. Given i.i.d data \(X_1 , \dots , X_n\), the empirical distribution \(\mathbb{P}_n\) is [...]A sample \(X_1^* , \dots , X_n^* | X_1 , \dots , X_n \sim^{i.i.d} \mathbb{P}_n\) is known as a bootstrap sample. Thus the starred random variables is a random resample with replacement.
A: the (random) discrete distribution that places a mass of 1/n at each observation \(X_i\) so for a set A, \[\mathbb{P}_n(A) := \frac{1}{n} \sum_{i=1}^n \mathbb{1}_A(X_i)\]

Q: Definition. Given a distribution function \(F: \mathbb{R} \to [0,1]\), the quantile function \(F^{-1}:(0,1) \to \mathbb{R}\) is \[[...]\]
A: F^{-1}(p) := \inf \{t: F(t) \ge p\}

Q: How does the accept-reject algorithm work?
A: If \(f(x) \le Mh(x)\) for all \(x \in \chi\) we have:
1) Generate \(X \sim h\) and independently \(U \sim U[0,1]\)
2. If \(U \le \frac{f(X)}{Mh(X)}\), output \(Y=X\); otherwise return to step 1
Then \(Y \sim f\)

Q: Definition. A pdf f on \(\chi\) is invariant or stationary for K if \[[...]\]for all measurable \(B \subseteq \chi\)
A: \int_{\chi}K(t,B)f(t)dt = \int_B f(t)dt

Q: Theorem. Glivenko–Cantelli. Let \(\hat{F}_n(t)\) be the empirical distribution function.
\[\sup_{t \in \mathbb{R}} \left| \hat{F}_n(t) - F(t) \right| \overset{\text{a.s.}}{\to} 0.\]
A: We only consider the case where \(F\) is continuous for simplicity. Fix \(m \in \mathbb{N}\) and pick
\[-\infty = t_0 < t_1 < \cdots < t_{m-1} < t_m = \infty\]
such that \(F(t_j) - F(t_{j-1}) = 1/m\) for \(j = 1, \ldots, m\). Now for all \(t \in \mathbb{R}\), there exists \(j \in \{1, \ldots, m\}\) such that \(t \in [t_{j-1}, t_j]\), so
\[\hat{F}_n(t) - F(t) \leq \hat{F}_n(t_j) - F(t_{j-1}) = \hat{F}_n(t_j) - F(t_j) + \frac{1}{m},\]
\[\hat{F}_n(t) - F(t) \geq \hat{F}_n(t_{j-1}) - F(t_j) = \hat{F}_n(t_{j-1}) - F(t_{j-1}) - \frac{1}{m}.\]
Thus,
\[\sup_t \left| \hat{F}_n(t) - F(t) \right| \leq \max_{j = 0, \ldots, m} \left| \hat{F}_n(t_j) - F(t_j) \right| + \frac{1}{m} \overset{\text{a.s.}}{\to} 0 \quad \text{by SLLN.}\]
We therefore have that writing \(\Omega_m := \left\{\limsup_{n \to \infty} \sup_t \left| \hat{F}_n(t) - F(t) \right| \leq 1/m\right\}\), we have \(\mathbb{P}(\Omega_m) = 1\). But
\[\Omega_\infty := \bigcap_{m=1}^\infty \Omega_m = \left\{\lim_{n \to \infty} \sup_t \left| \hat{F}_n(t) - F(t) \right| = 0\right\},\]
and
\[\mathbb{P}(\Omega_\infty^c) = \mathbb{P}\left(\bigcup_{m=1}^\infty \Omega_m^c\right) \leq \sum_{m=1}^\infty \mathbb{P}(\Omega_m^c) = 0.\]
\end{proof}

Q: Definition. The empirical distribution function \(\hat{F}_n\) bsaed on a sample \(X_1 , \dots , X_n\) is given by \[[...]\]
A: \hat{F}_n(t) := \mathbb{P}_n((-\infty, t]) = \frac{1}{n} \sum_{i=1}^n \mathbb{1}_{(-\infty, t]}(X_i)

Q: What is the Cramer-Rao lower bound?
A: For a function \(\phi : \Theta \to \mathbb{R}\), when estimating \(\phi(\theta) \) with an estimator \(\hat{\phi}\). THen for any \(\theta \in \text{int}\Theta\) where \(I(\theta) \) is invertible \[\text{Var}_{\theta}(\hat{\phi}) \geq \nabla_{\theta}\mathbb{E}_{\theta}(\hat{\phi})^T I(\theta)^{-1} \nabla_{\theta}\mathbb{E}_{\theta} (\hat{\phi})\]

Q: For any sequence of MLEs \(\hat{\theta}_n\) we have \[\sqrt{n}(\hat{\theta}_n - \theta_0) \to^d [...]\]
A: N_p(0, I_1 (\theta_0)^{-1})

Q: What is the Wald test?
A: \[W_n(\theta_0) := n(\hat{\theta}_n - \theta_0)^T J_n (\hat{\theta}_n)(\hat{\theta}_n - \theta_0) \to^d \chi_p^2\]

Q: What is Wilk's theorem?
A: When \(\Lambda_n(\Theta, \Theta_0) := 2\log \left (\frac{\sup_{\theta \in \Theta}L_n(\theta)}{\sup_{\theta \in \Theta_0}L_n(\theta)} \right )\) and \(\Theta_0 = \{\theta_0\}\) we have \[\Lambda_n \to^d \chi_p^2\]

Q: What is the Delta method given we have \(\hat{\theta}_n\) a sequence of random vectors(estimators) such that \(r_n(\hat{\theta}_n - \theta_0) \to^d Z\) where \(Z \) is a random vector and \(r_n \to \infty\) is some deterministic scalar sequence and \(\phi:\Theta \to \mathbb{R}^m\) continuously differentiable at \(\theta_0\).
A: \[ r_n (\phi(\hat{\theta}_n) - \phi(\theta_0)) \xrightarrow{d} \begin{pmatrix} \nabla_\theta \phi_1(\theta_0)^\top \\ \vdots \\ \nabla_\theta \phi_m(\theta_0)^\top \end{pmatrix} Z. \]

Q: What is Stein's Lemma?
A: Let \(Z \sim N(\mu, 1)\) and \(g: \mathbb{R} \to \mathbb{R}\) be a bounded, differentiable function such that \(\mathbb{E}[g'(Z)] < \infty\). Then \[ \mathbb{E}[(Z - \mu) g(Z)] = \mathbb{E}[g'(Z)]. \]

Q: What is the Glivenko-Cantelli theorem?
A: Let \(\hat{F}_n(t)\) be the empirical distribution function. \[\sup_{t \in \mathbb{R}} \left| \hat{F}_n(t) - F(t) \right| \overset{\text{a.s.}}{\to} 0.\]

Q: Theorem. Kolmogorov-Smirnov. If F is a continuous distribution function, then \[[...]\]where K has a Kolmogorov distribution for all F.
A: \sqrt{n} \sup_{t \in \mathbb{R} } \left| \hat{F}_n(t) - F(t) \right| \overset{\text{d} } {\to}K

Q: What are the steps of the Gibbs sampling algorithm for a bivariate case?
A: 1. Initialize \( X_0 = x \).
2. Iteratively perform the following steps for \( m = 1, 2, \ldots \) \[ Y_m \sim f_{Y|X}(\cdot \mid X = X_{m-1}) \] \[ X_m \sim f_{X|Y}(\cdot \mid Y = Y_m) \] This generates a Markov chain \( (Y_1, X_1), (Y_2, X_2), \ldots \), where the joint density \( f \) is invariant.

Q: What are the steps of the Metropolis-Hastings algorithm?
A: 1. Given an initial \( X_0 = x \), iteratively perform the following steps for \( m = 1, 2, \ldots \) \[ S_m \sim q(\cdot \mid X_m) \] \[ A_m \sim \text{Bernoulli}(a(X_m, S_m)) \] where \[ a(X_m, S_m) = \min\left( \frac{f(S_m) q(X_m \mid S_m)}{f(X_m) q(S_m \mid X_m)}, 1 \right). \] 2. Update: \[ X_{m+1} := A_m S_m + (1 - A_m) X_m \] This algorithm only requires evaluation of the ratio \( \frac{f(s)}{f(t)} \) and not \( f(t) \) itself

Q: What is the sample version of partial correlation?
A: \[\hat{\rho}_{X,Y|Z} = \frac{X^T(I-P)Y}{||(I-P)X|| ||(I-P)Y||}\]