Title: Part IA / Probability
Tags: Limit_Theorems Inequalities_Of_Random_Variables Continuous_Random_Variables Discrete_Distributions Independence_Of_Events Probability_Spaces Branching_Processes Discrete_Random_Variables Gaussian_Vectors Moment_Generating_Functions Combination_Of_Random_Variables Multivariate_Density_Functions Inclusion-Exclusion Probability_Generating_Functions Variance_And_Covariance Exponential_Function Random_Walks Simulation_of_Random_Variables

Q: Definition. Suppose \(\Omega\) is a set, and \(\mathcal{F}\) is a collection of its subsets. We call \(\mathcal{F}\) a \(\sigma\)-algebra if:
A: i) \(\Omega \in \mathcal{F}\)
ii) if \(A \in \mathcal{F}\), then \(A^C \in \mathcal{F}\)
iii)for any countable collection \((A_n)_{n\geq 1}\)with \(A_n \in \mathcal{F}\) for all n, we must also have that \(\cup_n A_n \in \mathcal{F}\)

Q: Definition. Suppose that \(\mathcal{F}\) is a \(\sigma\)-algebra on \(\Omega\). A function \(\mathbb{P}:\mathcal{F} \to [0,1]\) is called a probability measure if:
A: i) \(\mathbb{P}(\Omega) = 1\)
ii)for any countable disjoint collection of sets \((A_n)_{n\geq 1}\)in \(\mathcal{F}(A_n \in \mathcal{F} \text{ for all n})\), then \(\mathbb{P}(\cup_{n \geq 1}A_n) = \sum_{n\geq 1} \mathbb{P}(A_n)\)(this is called countable additivity)
We say that \(\mathbb{P}(A)\) is 'the probability of A'. We call \((\Omega, \mathcal{F}, \mathbb{P})\) a probability space, where \(\Omega\) is the sample space, \(\mathcal{F}\) is the \(\sigma\)-algebra, and \(\mathbb{P}\) is the probability measure

Q: Theorem. Stirling's Formula. \(n! \sim n^n\sqrt(2\pi n)e^{-n}\) as \(n \to \infty\)
First prove weaker statement \(\log(n!) \sim n\log n\)
A: Note that \(\log \lfloor x\rfloor \leq \log x \leq \log \lfloor x+1 \rfloor\).
Integrate this from 1 to n and let \(l_n = \log(n!)\)
We get that \(l_{n-1} \leq n\log n - n + 1 \leq l_n\)
Rearrage to bound \(l_n \) on both sides
Divide through by \(n\log n\) to get \(\frac{l_n}{n\log n} \to 1\) as \(n \to \infty\)

Q: Theorem. Countable subadditivity. Let \((\Omega, \mathcal{F}, \mathbb{P})\) be a probability space and let \((A_n)_{n \geq 1}\) be a not necessarily disjoint sequence of events in \(\mathcal{F}\). Then \[\mathbb{P}\left ( \bigcup_{n = 1}^{\infty} A_n\right ) \leq \sum_{n=1}^{\infty}\mathbb{P}(A_n)\]
A: Define a new sequence \(B_1 = A_1, B_n = A_n \setminus (A_1 \cup A_2 \cup \dots \cup A_{n-1})\). So they are a disjoint sequence of events. Note further that the union of all \(B_n\) is the same as the union of all \(A_n\). So \[\mathbb{P}\left ( \bigcup_{n=1}^{\infty}A_n \right ) = \mathbb{P}\left ( \bigcup_{n=1}^{\infty} B_n\right )\]So by the countable additivity theorem: \[\mathbb{P}\left ( \bigcup_{n=1}^{\infty} B_n \right )= \sum_{n=1}^{\infty} \mathbb{P}(B_n)\]But \(B_n \subseteq A_n\). So \(\mathbb{P}(B_n) \leq \mathbb{P}(A_n)\). The result follows.

Q: Theorem. Let \((\Omega, \mathcal{F}, \mathbb{P})\) be a probability space. Let \((A_n)_{n \geq 1}\) be an increasing sequence of events such that \(A_n \subseteq A_{n+1}\). We want to show that \[\lim_{n \to \infty} \mathbb{P}(A_n) = \mathbb{P}\left ( \bigcup_n A_n\right)\]
A: Let \(B_1 = A_1, B_n = A_n \setminus (A_1 \cup A_2 \cup \dots \cup A_{n-1})\). Then \[\mathbb{P}(A_n) = \mathbb{P}\left ( \bigcup_{k=1}^n B_k\right ) = \sum_{k=1}^n \mathbb{P}(B_k) \to \sum_{k=1}^{\infty} \mathbb{P}(B_k) = \mathbb{P} \left ( \bigcup_n B_n\right ) = \mathbb{P}\left (\bigcup_n A_n\right)\]

Q: Theorem. Inclusion-Exclusion formula \[\mathbb{P} \left ( \bigcup_{i=1}^n A_i \right ) = \sum_{k=1}^n (-1)^{k+1} \sum_{1 \leq i_1 \leq i_2 \leq i_k \leq n} \mathbb{P}(A_{i_1} \cap \dots \cap A_{i_k})\]
A: Proof by induction with n=2 case as base
\(\mathbb{P} \left ( \bigcup_{i=1}^n A_i\right) = \mathbb{P} \left( \left ( \bigcup_{i=1}^{n-1} A_i \right ) \cup A_n ) \right ) = \mathbb{P} \left ( \bigcup_{i=1}^{n-1} A_i\right) + \mathbb{P} (A_n) - \mathbb{P} \left ( \bigcup_{i=1}^{n-1} (A_i \cap A_n)\right)\)
Use inductive hypothesis to reach result.

Q: Proposition. Bonferroni inequalities. Tells you about inclusion-exclusion formula as an estimate at the rth term
A: \(\text{r odd} \implies \mathbb{P} \left ( \bigcup_{i=1}^n A_i\right) \leq \sum_{k=1}^r (-1)^{k+1} \sum_{1 \leq i_1 < \dots < i_k \leq n} \mathbb{P}(A_{i_1} \cap \dots \cap A_{i_k})\)
\(\text{r even} \implies \mathbb{P} \left ( \bigcup_{i=1}^n A_i\right) \geq \sum_{k=1}^r (-1)^{k+1} \sum_{1 \leq i_1 < \dots < i_k \leq n} \mathbb{P}(A_{i_1} \cap \dots \cap A_{i_k})\)
Proof using induction on n and using same trick as for proof of inclusion-exclusion

Q: Definition. Let \((\Omega, \mathcal{F}, \mathbb{P})\) be a probability space. Let \(A,B\in \mathcal{F}\). A countable collection of events \((A_n)\) is said to be independent if for all distinct \(i_1, \dots , i_k\), we have
A: \[\mathbb{P}(A_{i_1} \cap \dots \cap A_{i_k}) = \prod_{j=1}^k \mathbb{P}(A_{i_j})\]

Q: Definition. Let \((\Omega, \mathcal{F}, \mathbb{P})\) be a probability space. Let \(B \in \mathcal{F}\) with \(\mathbb{P}(B) > 0\). We define the conditional probability of A given B, written \(\mathbb{P}(A | B)\) as
A: \[\mathbb{P}(A | B) = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)}\]

Q: Claim. Suppose that \((A_n)\) is a disjoint sequence in \(\mathcal{F}\). Then \[\mathbb{P} \left ( \bigcup A_n | B \right ) = \sum_n \mathbb{P}(A_n | B)\]This is the countable additivity property for conditional probability
A: Rearrange using formulas and note that the sequence of events are disjoint so \[\mathbb{P}\left ( \bigcup (A_n \cap B) \right ) = \sum_n (A_n \cap B)\]

Q: Claim. Law of total probability. Suppose \((B_n)\) is a disjoint collection of events in \(\mathcal{F}\), such that \(\cup B = \Omega\), and for all n, we have \(\mathbb{P}(B_n) > 0\). If \(A \in \mathcal{F}\) then \[\mathbb{P}(A) = \sum_n \mathbb{P}(A | B_n) \cdot \mathbb{P}(B_n)\]
A: \[\mathbb{P}(A) = \mathbb{P}(A \cap \Omega) = \mathbb{P}\left ( \bigcup (A \cap B_n) \right ) = \sum_n \mathbb{P}(A \cap B_n)\]and the result follows

Q: Claim. Baye's Formula. Suppose \((B_n)\) is a disjoint sequence of events with \(\cup B_n = \Omega\) and \(\mathbb{P}(B_n) > 0\) for all n. Then \[\mathbb{P}(B_n | A) = \frac{\mathbb{P}(A | B_n) \mathbb{P}(B_n)}{\sum_k \mathbb{P}(A | B_k) \mathbb{P}(B_k)}\]
A: \[\mathbb{P}(B_n | A) = \frac{\mathbb{P}(B_n \cap A)}{\mathbb{P}(A)} = \frac{\mathbb{P}(A|B_n) \mathbb{P}(B_n)}{\mathbb{P}(A)}\]The result follows from the Law of Total Probability.

Q: Definition. Bernoulli distribution. Model the outcome of a test such as the toss of a coin with this.
A: Let \(\Omega = \{0,1\}\). Then \(p = p_1 \implies p_0 = 1 - p\)

Q: Definition. Binomial Distribution. Models a sequence of N independent Bernoulli distributions of parameter p. We then count the amount of successes.
A: So \(\Omega = \{0,1,\dots,N\}\). \[\mathbb{P}(\{k\}) = p_k = {N \choose k}p^k(1-p)^{N-k}\]

Q: Definition. Multinomial distribution. Generalisation of the binomial distribution with parameters \(N \in \mathbb{Z}^+, p_1,p_2,\dots \in [0,1]\) where \(\sum_{i=1}^k p_i = 1\). Models a sequence of N independent trials in which a number from 1 to N is selected, where the probability of selecting i is \(p_i\).
A: \(\Omega = \{(n_1,n_2,\dots, n_k) \in \mathbb{N}^k:\sum_{i=1}^k n_i = N\}\). In other words ordered partitions of N.\[\mathbb{P}(n_1 \text{ outcomes had value 1, }\dots , n_k \text{ outcomes had value k}) = \mathbb{P}((n_1, n_2, \dots, n_k)) = {N \choose n_1,\dots,n_k}p_1^{n_1}\dots p_k^{n_k}\]

Q: Definition. Geometric Distribution. Consider a Bernoulli distribution of parameter p, this models running this trials many times independently until the first success.
A: Then \(\Omega = \{1,2, \dots\} = \mathbb{Z}^+\). Then \(p_k = (1-p)^{k-1}p\). We can compute the infinite geometric series to get 1.

Q: Definition. Poisson distribution. Models the number of occurences of an event in a given inteval of time. Has one parameter \(\lambda \in \mathbb{R}\)
A: \[p_k = e^{- \lambda}\frac{\lambda^k}{k!}\]We can again do the infinite sum to see that it equals 1.
Suppose customers arrive into a shop during the time interval [0,1]. We will subdivide this into N intervals \(\left [\frac{i-1}{N}, \frac{i}{N}\right ]\). In each interval, a single customer arrives with probability p, independent of other time intervals. In this example \[\mathbb{P}(\text{k customers arrive}) = {N \choose k}p^k(1-p)^{N-k}\]Let \(\lambda = pN\), this is roughly like our average number of people coming into the shop during the time interval.
Taking the limit as \(N \to \infty\) will give us the Poisson distribution

Q: Definition. Consider the probability space \((\Omega, \mathcal{F},\mathbb{P})\). A random variable X is a function \(X : \Omega \to \mathbb{R}\) satisfying
\[[...]\]for any given x
A: \{\omega \in \Omega:X(\omega) \leq x\} \in \mathcal{F}

Q: Definition. A random variable X is called discrete if it takes values in a countable set. Suppose X takes values in the countable set S. For every \(x \in S\) we write \[p_x = \mathbb{P}(X = x) = \mathbb{P}(\{\omega : X(\omega) = x\})\]
A: We call \(p_x\) the probability mass function of X, or the distribution of X

Q: Definition. Let \((\Omega, \mathcal{F}, \mathbb{P})\) be a probability space such that \(\Omega\) is countable. Let \(X : \Omega \to \mathbb{R}\) be a random variable, which is necesarily discrete. We say that X is non-negative if \(X \geq 0\). We define the expectation of X to be
A: \[\mathbb{E}[X] = \sum_{\omega}X(\omega) \cdot \mathbb{P}(\{\omega\}) = \sum_{x \in \Omega_x}xp_x\]where \(\Omega_x = \{X(\omega): \omega \in \Omega\}\)

Q: Definition. Expectation under function application. If \(g : \mathbb{R} \to \mathbb{R}\), we can define \(g(X)\) to be the random variable given by \(g(W)(\omega) = g(X(\omega))\). Then \[\mathbb{E}[g(X)] = \sum_{x \in \Omega_X}g(x) \cdot \mathbb{P}(X = x)\]
A: Prove that \(\{Y = y\} = \{X \in g^{-1}(\{y\})\}\)
Then rearrange the expectation formula in the right way to achieve result.
Remember that \[\sum_{y \in \Omega_Y}y \cdot \mathbb{P}(X \in g^{-1}(\{y\})) = \sum_{y \in \Omega_Y}y \cdot \sum_{x \in g^{-1}(\{y\})} \mathbb{P}(X = x)\]

Q: Definition. Variance of a random variable X.
A: \[\text{Var}(X) = \mathbb{E}\left [ (X - \mathbb{E}[X])^2\right]\]\[\text{Var}(X) = \mathbb{E}[X^2] - \mathbb{E}[X]^2\]The second follows from the first.

Q: Definition. Let X and Y be random variables. Their covariance measures how dependent X and Y are and is defined by
A: \[\text{Cov}(X,Y) = \mathbb{E}[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])]\]

Q: Theorem. Markov's Inequality. Let \(X \geq 0\) be a non-negative random variable. Then for all \(a > 0\), \[\mathbb{P}(X \geq a) \leq \frac{\mathbb{E}[X]}{a}\]
A: Observe that \(X \geq a \cdot 1(X \geq a)\), check for \(X < a\) and \(X \geq a\).
Then we take expectations to get the result.

Q: Theorem. Chebyshev's Inequality. Let X be a random variable with finite expectation. Then for all \(a > 0\), \[\mathbb{P}(|X - \mathbb{E}[X]| \geq a) \leq \frac{\text{Var}(X)}{a^2}\]
A: Note that \(\mathbb{P}(|X - \mathbb{E}[X]| \geq a) = \mathbb{P}(|X - \mathbb{E}[X]|^2 \geq a^2)\).
Then apply Markov's inequality.

Q: Theorem. Cauchy-Schwarz Inequality. If X and Y are random variables, then \[|\mathbb{E}[XY]| \leq \sqrt{\mathbb{E}[X^2] \cdot \mathbb{E}[Y^2]}\]
A: Take the expectation of both sides of \(0 \leq (X - tY)^2\). Take the value at the minimum and the result follows.

Q: Definition. A function \(f : \mathbb{R} \to \mathbb{R}\) is called convex if \(\forall x,y \in \mathbb{R}\) and for all \(t \in [0,1]\), \[[...]\]
A: f(tx + (1-t)y) \leq tf(x) + (1-t)f(y)
Back Extra: The linear interpolation of two points x and y is always greater than the function applied to the linear interpolation of the input points.

Q: Theorem. Jensen's Inequality. Let X be a random variable, and let f be a convex function. Then \[\mathbb{E}[f(X)] \geq f(\mathbb{E}[X])\]
A: Set \(m = \mathbb{E}[X]\). Then use the property of convex functions that \(\forall m \in \mathbb{R}, \exists a,b \in \mathbb{R}\) such that \(f(m) = am+b\) and \((\forall x)f(x) \geq ax+b \).
So we create the function and then apply it to X so \(f(X) \geq aX + b\). Now take the expectation to get the result

Q: Lemma. Let \(f : \mathbb{R} \to \mathbb{R}\) be a convex function. Then f is the supremum of all the lines lying below it. More formally, \(\forall m \in \mathbb{R}, \exists a,b \in \mathbb{R}\) such that \(f(m) = am+b\) and \(f(x) \geq ax+b\) for all x.
A: Write \(m = tx + (1-t)y\) so we can apply the definition of convexity, \(f(m) \leq tf(x) + (1-t)f(y)\).
Then \(tf(m) + (1-t)f(m) \leq tf(x) + (1-t)f(y)\), rearrange to get \(\frac{f(m) - f(x)}{m - x} \leq \frac{f(y) - f(m)}{y -m}\).
So pick \(a = \sup_{x<m} \frac{f(m) - f(x)}{m-x}\) such that \(\frac{f(m) - f(x)}{m-x} \leq a \leq \frac{f(y) - f(m)}{y -m}\) for all \(x < m <y\). Rearrange this to give \[f(x) \geq a(x-m) + f(m) = ax + (f(m) -am)\]

Q: Theorem. AM-GM using Jensen's Inequality.
A: Let f be a convex function. Suppose \(x_1, x_2, \dots , x_n \in \mathbb{R}\). Then from Jensen's inequality, \[\frac{1}{n}\sum_{k=1}^n f(x_k) \geq f\left (\frac{1}{n}\sum_{n=1}^k x_k\right )\]Define the random variable X to take values \(x_1, x_2, \dots x_n\) with equal probability, so the LHS is \(\mathbb{E}[f(X)]\) and the RHS is \(f(\mathbb{E}[X])\). Now let \(f(x) = - \log x\). This gives us the AM-GM inequality when plugged in.

Q: Theorem. Law of Total Expectation. Suppose \(X \geq 0\). Let \((\Omega_n)\)be a partition of \(\Omega\) into disjoint events, so \(\Omega = \bigcup_n \Omega_n\). Then \[\mathbb{E}[X] = \sum_n \mathbb{E}[X | \Omega_n]\cdot\mathbb{P}(\Omega_n)\]
A: Write \(X = X \cdot 1(\Omega)\). Then \(\mathbb{E}[X] = \mathbb{E} \left [ \sum_n X \cdot 1(\Omega_n)\right]\).
Then the result follows from this by countable additivity of expectation.

Q: Definition. Let \(X_1, \dots , X_n\) be discrete random variables. Their joint distribution is defined as \[[...]\]
for all \(x_i \in \Omega_i\)
A: \mathbb{P}(X_1 = x_1, \dots, X_n = x_n)
Back Extra: The conditional distribution of X given Y=y where \(y \in \Omega_y\) is defined to be \[\mathbb{P}(X = x | Y = y) = \frac{\mathbb{P}(X=x, Y=y)}{\mathbb{P}(Y=y)}\]

Q: Proposition. Convolution of distributions. Let X and Y be independent, discrete random variables. We would like to find \(\mathbb{P}(X + Y =z)\). This is \[[...]\]
A: \mathbb{P}(X+Y=z) = \sum_y \mathbb{P}(X=z-y)\cdot \mathbb{P}(Y=y) = \sum_x \mathbb{P}(X=x) \cdot \mathbb{P}(Y = z-x)
Back Extra: This relies on the fact that \(\mathbb{P}(X+Y=z) = \sum_y \mathbb{P}(X+Y=z, Y=y)\)

Q: Definition. A random process, also known as a stochastic process, is a sequence of random variables \(X_n\) for \(n \in \mathbb{N}\). A random walk is a random process that can be expressed as
A: \[X_n = x + Y_1 + Y_2 + \dots + Y_n\]Where the \(Y_i\) are independent and identically distributed, and x is a deterministic number.

Q: Definition. Probability generating function. Let X be a random variable with values in the positive integers. Let \(p_r = \mathbb{P}(X = r)\) be the probability mass function. Then the probability generating function is defined to be
A: \[p(z) = \sum_{r=0}^{\infty} p_rz^r = \mathbb{E}[z^X] \text{ for } |z| \leq 1\]

Q: Theorem. The probability generating function of X uniquely determines the distribution of X.
A: Suppose \((p_r)\) and \((q_r)\) are two probability mass functions with \[\sum_{r=0}^{\infty} p_rz^r = \sum_{r=0}^{\infty} q_rz^r\]Show that the coefficients are the same by induction.

Q: Theorem. \[\mathbb{E}[X_n] = \mathbb{E}[X_1]^n\]
A: Proof by induction. \(\mathbb{E}[X_{n+1} | X_n = m] = \mathbb{E}[Y_{1,n} + \dots + Y_{X_n,n} | X_n = m] = m\mathbb{E}[X_1]\)

Q: Theorem. Let \(G(z) = \mathbb{E}[z^{X_1}]\) be the probability generating function of \(X_1\), and \(G_n(z) = \mathbb{E}[z^{X_n}]\) be the probability generating function of \(X_n\). Then \[G_{n+1}(z) = G(G_n(z)) = G(G(\dots G(z) \dots)) = G_n(G(z))\]
A: \(G_{n+1}(z) = \mathbb{E}[z^{X_{n+1}}] = \mathbb{E}[\mathbb{E}[z^{X_{n+1}} | X_n]]\)
\(\mathbb{E} \left[ z^{X_{n+1}} | X_n = m \right] = \mathbb{E}[z^{Y_{1,n}+\dots+Y_{m,n}}| X_n = m] = \mathbb{E}[z^{X_1}]^m = G(z)^m\)
The result follows

Q: Claim. \(q_{n+1} = G(q_n)\) and \(q = G(q)\)
A: \(q_{n+1} = \mathbb{P}(X_{n+1} = 0) = G_{n+1}(0) = G(G_n(0)) = G(q_n)\)
Since G is continous, taking the limit as \(n \to \infty\)and using that \(q_n \to q\) gives \(G(q) = q\)

Q: Theorem. The extinction probability q is the minimal non-negative solution to \(G(t) = t\). Further, supposing that \(\mathbb{P}(X_1 = 1) > 1\), we have that \(q>1\) if and only if \(\mathbb{E}[X_1] > 1\)
A: Proof. Aint no way i need to memorise that shit

Q: Claim. Let \(X \geq 0\). Then \[\mathbb{E}[X] = \int_0^{\infty} \mathbb{P}(X \geq x) \mathrm{d}x\]
A: \(\mathbb{E}[X] = \int_0^{\infty}xf(x) dx =\int_0^{\infty}\left (\int_0^xdy \right)f(x) dx =\int_0^x dy \int_y^{\infty}f(x) dx \).
Since \(\int_y^{\infty} f(x)dx = 1 - \int_{-\infty}^y f(x) dx\) the result follows

Q: Let \((\Omega, \mathcal{F}, \mathbb{P})\) be a probability space. Then let X be a random variable. We define the probability distribution function as
A: \[F(x) = \mathbb{P}(X \leq x)\]

Q: Theorem. Suppose that X is a continous random variable with density f. Let f be a monotonic continous function, such that \(g^{-1}\) is differentiable. Then \(g(X)\) is a continous random variable with density \(fg^{-1}(x) \left |\frac{d}{dx}g^{-1}(x)\right |\)
A: Suppose that g is strictly increasing. We have \(\mathbb{P}(g(X) \leq x) = \mathbb{P}(X \leq g^{-1}(x)) = F(g^{-1}(x))\).
Then we differentiate to get the result. If g is strictly decreasing we get that \(\mathbb{P}(g(X) \leq x) = 1 - F(g^{-1}(x))\) and due to the modulus signs in the result we get the same thing.

Q: Definition. Suppose X is a continous random variable. Then the median of X, denoted by m, is the number satisfying
A: \[\mathbb{P}(X \leq m) = \mathbb{P}(X \geq m) = \frac{1}{2}\]

Q: Definition. Let f,g be density functions. Then the convolution of f and g is
A: \[(f*g)(y) = \int_{-\infty}^{\infty} f_Y(y-x)f_X(x)dx\]

Q: Definition. Conisder a continous random variable X with density f. Then the moment generating function of X is defined as \[[...]\]whenever this integral is finite. Note that \(m(0) = 1\)
A: m(\theta) = \mathbb{E}[e^{\theta X}] = \int_{-\infty}^{\infty} e^{\theta X}f(x)dx

Q: Theorem. Suppose the moment generating function is defined on an open interval of values of \(\theta\). Then \[\frac{d^r}{d \theta^r}m(\theta) \bigg|_{\theta = 0} = [...]\]
A: \mathbb{E}[X^r]

Q: Theorem. Let \(X = (X_1, X_2, \dots, X_n)\)be a random variable with values in \(\mathbb{R}^n\). Then the moment generating function is \(m(\theta) = \mathbb{E}[e^{\theta_1X_1 + \dots + \theta_nX_n}]\). If the moment generating function is finite for a range of values of \(\theta\), it uniquely determines the distributino of X. Also, \[\frac{\partial^rm}{\partial \theta_i^r} \bigg |_{\theta = 0} = [...]\]and \[\frac{\partial^{r+s}m}{\partial \theta_i^r \partial \theta_j^s} \bigg |_{\theta = 0} = [...]\]Further, \[m(\theta) = [...]\]if and only if \(X_1, \dots, X_n\) are independent
A: \mathbb{E}[X_i^r]; \mathbb{E}[X_i^r X_j^s]; \prod_{i=1}^n\mathbb{E}[e^{\theta_iX_i}]

Q: Definition. Let \((X_n:n \in \mathbb{N})\) be a sequence of random variables and let X be another random variable. We say that \(X_n\) converges to X in distribution,written \(X_n \to^d X\) if
A: \[F_{X_n}(x) \to F_X(x)\]

Q: Theorem. Continuity property for moment generating functions. Let X be a continous random variable with \(m(\theta) < \infty\) for some \(\theta \neq 0\). Suppose that \(m_n(\theta) \to m(\theta)\) for all \(\theta \in \mathbb{R}\), where \(m_n(\theta) = \mathbb{E}[e^{\theta X_n}]\), and \(m(\theta) = \mathbb{E}[e^{\theta X}]\). Then
A: \[X_n \xrightarrow{d} X\]

Q: Theorem. Let \((X_n : n \in \mathbb{N})\) be a seqence of independent and identically distributed random variables, with \(\mu = \mathbb{E}[X_1] < \infty\). Let \(S_n = X_1 + \dots + X_n\). Then for all \(\varepsilon > 0\), \[\mathbb{P} \left ( \left | \frac{S_n}{n} - \mu\right | > \varepsilon\right ) \to 0\]as \(n \to \infty\)
A: By Chebyshev's inequality, \(\mathbb{P}(|\frac{S_n}{n} - \mu| > \varepsilon) = \mathbb{P}(|S_n - n\mu) > \varepsilon n) \leq \frac{\text{Var}(S_n)}{\varepsilon^2 n^2} = \frac{n\sigma^2}{\varepsilon^2 n^2 } \to 0\)

Q: Definition. A sequence \((X_n)\) converges to X in probability, written \(X_n \xrightarrow{\mathbb{P}} X\) as \(n \to \infty\) if for all \(\varepsilon > 0\)
A: \[\mathbb{P}(|X_n - X| > \epsilon) \to 0; n\to \infty\]

Q: Definition. A sequence \((X_n)\) converges to X almost surely if
A: \[\mathbb{P}\left (\lim_{n \to \infty} X_n = X \right) = 1\]

Q: Theorem. Let \((X_n)_{n \in \mathbb{N}}\) be an independent and identically distributed sequence of random variables, with \(\mu = \mathbb{E}[X_1]\) finite. Let \(S_n = X_1 + \dots + X_n\). Then \[\frac{S_n}{n} \to \mu \text{ as } n \to \infty \text{ almost surely}\] In other words, \[[...]\]
A: \mathbb{P}\left ( \lim_{n \to \infty} \frac{S_n}{n} \to \mu \right ) = 1

Theorem. Let \(X_n\) be an independently and identically distributed sequence of random varialbes, with \(\mu = \mathbb{E}[X_1]\) finite. Let \(S_n = X_1 + \dots + X_n\). Then for all \(x \in \mathbb{R}\)\[\mathbb{P}\left ( \frac{S_n - n\mu}{\sqrt{n}\sigma} \leq x\right ) \to \Phi(x) = \int_{\infty}^x \frac{e^{-\frac{y^2}{2}}}{\sqrt{2\pi}} dy\]In other words
Back Extra: \[\frac{S_n -n\mu}{\sigma \sqrt{n}} \xrightarrow{d} Z \text{ where Z is the standard normal distribution}\]

Q: Definition. Let \(X = (X_1, \dots, X_n)^T\) with values in \(\mathbb{R}^n\). Then we define that X is a Gaussian vector if
\[[...]\]
A: \forall u \in \mathbb{R}^n, u^T X = \sum_{i=1}^n u_iX_i = \mu + \sigma Z
Back Extra: So any linear combination of the \(X_i\) is Gaussian.

Q: Definition. Expectation and variance of Gaussian vector. We define the mean of a Gaussian vector X as \[[...]\]We define the variance as \[[...]\]Hence the components of V are \[[...]\]
A: \mu = \mathbb{E}[X] = (\mathbb{E}[X_1], \dots, \mathbb{E}[X_n])^T ; \mu_i = \mathbb{E}[X_i]; Var(X) = \mathbb{E}[(X-\mu)(X - \mu)^T]; V_{ij} = \text{Cov}(X_i,X_j)

Q: Definition. Moment generating function of Gaussian vector. We define it by \[[...]\]
A: m(\lambda) = \mathbb{E}[e^{\lambda^T}X]
Back Extra: Where \(\lambda \in \mathbb{R}^n\).

Q: Lemma. If V is diagonal, then the \(X_i\) are independent.
A: Since V is diagonal with diagonal entries \(\lambda_i\), we have \[(x-\mu)^TV^{-1}(x-\mu) = \sum_{i=1}^n \frac{(x_i-\mu_i)^2}{\lambda_i}\]Hence, \[f_X(x) = \frac{1}{\sqrt{(2\pi)^n \det V}}\exp\left(-\sum_{i=1}^n\frac{(x_i - \mu_i)^2}{2\lambda_i} \right)\]So \(f_X\) factorises into a product. Hence the \(X_i\) are independent.

Q: Definition. Uniform Distribution. For \(a,b \in \mathbb{R}\) with a<b, the density function is [...]
A: \[f(x) = (b-a)^{-1}1_{[a,b]}(x)\]

Q: Definition. Exponential Function. For \(\lambda \in (0, \infty)\), the density function on \([0,\infty)\) is [...]
A: \[f(x) = \lambda e^{-\lambda x}\]
Back Extra: Parameter \(\lambda\), written \(E(\lambda)\)

Q: Definition. Gamma distribution. For \(\alpha, \lambda \in (0,\infty)\), the density function on \([0,\infty)\) is [...]
A: \[f(x) = \lambda^{\alpha}x^{\alpha - 1} e^{- \lambda x}/ \Gamma(\alpha)\]
Back Extra: Gamma distribution of parameters \(\alpha, \lambda\) written \(\Gamma(\alpha, \lambda)\). Here \[\Gamma(\alpha) = \int_0^{\infty} x^{\alpha -1}e^{-x}dx\]When \(\alpha\) is an integer, we can integrate by parts \(\alpha -1 \) times to see that \(\Gamma(\alpha) = (\alpha -1)!\)

Q: Proposition. Memoryless Property of the Exponential Distribution. Let T be an exponential random variable of parameter \(\lambda\), then \[\mathbb{P}(T > t) = \int_t^{\infty} \lambda e^{-\lambda \tau} d\tau = e^{\lambda t}\]so for all \(s,t \geq 0\), \[[...]\]
A: \mathbb{P}(T > s+t | T > s) = \mathbb{P}(T > t)
Back Extra: This is called the memoryless property becuase, thinking of T as a random time, conditional on T exceeding a given time s, the time T-s still to wait is also exponential of parameter \(\lambda.\)

Q: Theorem. Let X be a random variable with values in some domain D in \(\mathbb{R}^d\) and having a density function \(f_X\) on D. Suppose that \(\phi\) maps D bijectively to another domain \(\phi(D)\) in \(\mathbb{R}^d\) and suppose that \(\phi\) has a continuous derivative on D with a non-zero determinant for all x. Set \(y = \phi(x)\) and consider the new random variable \(Y = \phi(X)\). Then Y has a density function \(f_Y\) on \(\phi(D)\), given by \[[...]\]
A: f_Y(y) = f_X(x) |J|
Back Extra: where J is the Jacobian.

Q: Definition. Inverse Transform Method. Suppose we wish to simulate a random variable X which takes values in an open interval I, where it has a positive density function f. Then for any distribution function F, if we define \(G : (0,1) \to \mathbb{R}\) by \[G(u) = \inf \{x \in \mathbb{R}: u \leq F(x)\}\]then, for \(u \in (0,1) \) and \(x \in \mathbb{R}\), we have \(G(u) \leq x\) if and only if \(u \leq F(x)\). Then X = G(U) has distribution function F where U is distributed uniformly on [0,1].
A: asdffds

Q: Theorem. Uniqueness of moment generating functions.
A: Let X and Y be random variables having the same moment generating function M and suppose that \(M(\lambda) < \infty\) for some \(\lambda \neq 0\). Then X and Y have the same distribution function

Q: Theorem. Let \(X_n\) be an independently and identically distributed sequence of random varialbes, with \(\mu = \mathbb{E}[X_1]\) finite. Let \(S_n = X_1 + \dots + X_n\). Then for all \(x \in \mathbb{R}\)\[\mathbb{P}\left ( \frac{S_n - n\mu}{\sqrt{n}\sigma} \leq x\right ) \to \Phi(x) = \int_{\infty}^x \frac{e^{-\frac{y^2}{2}}}{\sqrt{2\pi}} dy\]In other words\[[...]\]
A: \frac{S_n -n\mu}{\sigma \sqrt{n} } \xrightarrow{d} Z \text{ where Z is the standard normal distribution}
Back Extra: We assume that \(M_X(\delta) < \infty\) and \(M_X(-\delta) < \infty \) for some \(\delta > 0\). It suffices to deal with the case \(\mu = 0, \sigma^2 = 1\). For then the general case follows by consdiering the random variables \(Y_n = (X_n - \mu/ \sigma)\).
Set \[R(x) = \frac{1}{2} \int_0^1 x^3 e^{tx} (1-t)^2 dt\].
By integration by parts, we see that \[e^x = 1 + x + \frac{x^2}{2} + R(x)\]Note that, for \(|\lambda| \leq \delta / 2\) and \(t \in [0,1]\), we have \(e^{t\lambda x} \leq e^{\delta |x| / 2}\), so \[|R(\lambda x)| \leq \frac{|\lambda x|^3}{3!}e^{\delta |x| / 2} \leq \left ( \frac{2|\lambda|}{\delta}\right )^3 \frac{(\delta |x|/2)^3}{3!}e^{\delta |x| / 2} \leq \left ( \frac{2 |\lambda|}{\delta}\right )^3 e^{\delta |x|}\]and so \[ |R(\lambda X)| \leq\left ( \frac{2 |\lambda|}{\delta}\right )^3 (e^{\delta X} + e^{- \delta X}). \]Hence \[ |\mathbb{E}(R(\lambda X))| \leq \left ( \frac{2 |\lambda|}{\delta}\right )^3 (M_X(\delta) + M_X(-\delta)) = o(|\lambda|^2)\]as \(\lambda \to 0\). On taking expectations in the identity \[e^{\lambda X} = 1 + \lambda X + \frac{\lambda^2 X^2}{2} + R(\lambda X)\]
we obtain \[M_X(\lambda) = 1 + \frac{\lambda^2}{2} + \mathbb{E}(R(\lambda X))\]Hence, for all \(\lambda \in \mathbb{R}\), as \(n \to \infty\), \[M_{S_n/ \sqrt{n}} (\lambda) = \mathbb{E}(e^{\lambda(X_1 + \dots + X_n)/\sqrt{n}}) = M_X(\lambda / \sqrt{n})^n = \left (1 + \frac{\lambda^2}{2n}(1 + 0(1))\right )^n \to e^{\lambda^2 / 2} = M_Z(\lambda)\]Where \(Z \sim N(0,1)\). The result follows from the continuity theorem for moment generating functions.