Title: Part IB / Distributions
Tags: Distributions

Distribution. \( X \sim U\{1, \dots , n\} \)\[\text{pmf} \; f(x) = 1/n\]
\[\text{Range of X}:\{1, \dots n\} \]
\[\text{Parameter Range}: n \in \mathbb{N}\]
\[\mathbb{E}(X) = \frac{1}{2} (n+1)\]
\[\text{Var}(X) =  \frac{1}{12} (n^2 - 1)\]
\[\mathbb{E}(z^X) =  \frac{1}{n} \sum_{i=1}^n z^i \]

Distribution. \(X \sim \text{Bin}(n,p)\)\[\text{pmf} \; f(x) = {n \choose x} p^x (1-p)^{n-x} \]
\[\text{Range of X}:\{0, \dots , n\} \]
\[\text{Parameter Range}:  n \in \mathbb{N}, p \in [0,1]\]
\[\mathbb{E}(X) =  np \]
\[\text{Var}(X) =  np(1-p)\]
\[\mathbb{E}(z^X) = \{pz + (1-p)\}^n \]
Back Extra: \(\text{Bin}(1,p)\) distribution is also called the \(\text{Bernoillu} (p)\) distribution. The sum of n iid Bernoulli distributions is a Binomial distribution. This models the number of successes in n independent trials, each with probability p of success.

Distribution. \(X \sim \text{Poi}(\lambda)\)\[\text{pmf} \; f(x) = e^{-\lambda} \frac{\lambda^x}{x!} \]
\[\text{Range of X}:\{0, 1, \dots\} \]
\[\text{Parameter Range}:  \lambda \in [0,\infty)\]
\[\mathbb{E}(X) =  \lambda\]
\[\text{Var}(X) =  \lambda\]
\[\mathbb{E}(z^X) =  e^{\lambda(z-1)} \]

Distribution. \(X \sim \text{NegBin} (k,p)\)\[\text{pmf} \; f(x) =  {x -1 \choose k-1} p^k (1-p)^{x-k} \]
\[\text{Range of X}:\{k, k+1, \dots\} \]
\[\text{Parameter Range}: k \in \mathbb{N}, p \in [0,1]\]
\[\mathbb{E}(X) = k/p\]
\[\text{Var}(X) = k(1-p)/p^2\]
\[\mathbb{E}(z^X) =  \frac{(pz)^k}{(1 - (1-p)z)^k} \]
Back Extra: \(\text{NegBin}(1,p)\) distributino is also called the \(\text{Geometric}(p)\) distribution. The sum of k iid geometric distributions is a negative binomial distribution. This models the number of independent trials required to attain k successes, each with probability p of success.

Distribution. \(X \sim \text{Multi}(n, p_1, \dots , p_k)\)  \[\text{pmf} \; f(x) = \frac{n!}{n_1! \dots n_k!} \]
\[\text{Range of X}:(n_1. \dots n_k) \in \{0,1, \dots , n\}^k : \sum_i n_i = n\]
\[\text{Parameter Range}:  p_1, \dots , p_k \in [0,1] : \sum_i p_i = 1, n \in \mathbb{N}\]
\[\mathbb{E}(X) =  (np_1 , \dots np_k)\]
\[\text{Var}(X) = \text{Cov}(X_i, X_j) = \begin{cases} np_i(1-p_i) \; i=j \\ -np_ip_j \; i \neq j\end{cases} \]
\[\mathbb{E}(z^X) =  \mathbb{E}(z_1^{X_1} \dots z_k^{X_k}) = (\sum_{i=1}^k p_iz_i) ^n \]
Back Extra: Models the number of balls that appear in each of k buckets, when n balls are placed independently in the buckets and a ball falls in the ith bucket with probability \(p_i\)

Distribution. \(X \sim U[a,b]\)\[\text{pdf} \; f(x) = \frac{1}{b-a} \]
\[\text{Range of X}: [a,b]\]
\[\text{Parameter Range}:  (a,b) \in\mathbb{R}^2, a < b \]
\[\mathbb{E}(X) =  \frac{1}{2} (a+b)\]
\[\text{Var}(X) =  \frac{1}{12} (b-a)^2\]
\[\mathbb{E}(e^{tX}) =  \frac{e^{bt} - e^{at} }{ t(b-a)} \]

Distribution. \(X \sim N(\mu, \sigma^2)\)\[\text{pdf} \; f(x) =  \frac{1}{\sqrt{2 \pi \sigma^2} } e^{- \frac{1}{2\sigma^2} (x-\mu)^2} \]
\[\text{Range of X}: \mathbb{R} \]
\[\text{Parameter Range}:  \mu \in \mathbb{R}, \sigma \in (0,\infty) \]
\[\mathbb{E}(X) =  \mu\]
\[\text{Var}(X) =  \sigma^2\]
\[\mathbb{E}(e^{tX}) =  e^{t\mu + \sigma^2 t^2/2} \]

Distribution. \(X \sim \Gamma (\alpha, \lambda)\)\[\text{pdf} \; f(x) =  \frac{\lambda^{\alpha} x^{\alpha - 1} e^{-\lambda x} }{\Gamma(\alpha)} \]
\[\text{Range of X}:(0,\infty)\]
\[\text{Parameter Range}:  \alpha \in (0,\infty), \lambda \in (0 ,\infty)\]
\[\mathbb{E}(X) =  \frac{\alpha}{\lambda} \]
\[\text{Var}(X) =  \frac{\alpha}{\lambda^2} \]
\[\mathbb{E}(e^{tX}) = \begin{cases} \left ( \frac{\lambda}{\lambda - t}\right )^\alpha \text{ if } t<\lambda \\ \infty \text{ if } t \geq \lambda \end{cases} \]
Back Extra: The \(\Gamma(1, \lambda)\) distribution is the \(\text{Exp}(\lambda)\) distribution. The sum of n iid exponential distributions is a \(\Gamma(n,\lambda)\) distribution.
For \(n \in \mathbb{N}\), then \(\Gamma(n/2, 1/2)\) distribution is the same as the \(\chi_n^2\) distribution. If \(X_i\) are iid standard normal distributions, then \(\sum_{i=1}^n X_i^2 \sim \chi_n^2 \)
Also recall that the Gamma function is defined for \(z \in \mathbb{C}\) with \(\text{Re}(z) > 0\), by \(\Gamma(z) := \int_0^{\infty} t^{z-1} e^{-t} dt\). If \(n \in \mathbb{N}\), then \(\Gamma(n) = (n-1)!\). The function \(B(a,b) = \frac{\Gamma(a) \Gamma(b)}{\Gamma(a+b)}\) is called the beta function.

Distribution. \(X \sim \text{Beta}(a,b) \)\[\text{pmf} \; f(x) =  \frac{\Gamma(a+b)}{ \Gamma(a) \Gamma(b) } x^{a - 1} (1-x)^{b-1} \]
\[\text{Range of X}:(0,1)\]
\[\text{Parameter Range}:  a \in (0,\infty), b \in (0,\infty)\]
\[\mathbb{E}(X) =  \frac{a}{a+b} \]
\[\text{Var}(X) =  \frac{ab}{(a+b)^2 (a+b+1)} \]

Distribution. \(X \sim \text{Cauchy}\)\[\text{pdf} \; f(x) =  \frac{1}{\pi (1+x^2)} \]
\[\text{Range of X}: \mathbb{R} \]
\[\mathbb{E}(X) = \text{Does not exist} \]
\[\text{Var}(X) =  \infty\]
\[\mathbb{E}(z^X) =  \begin{cases} 1 \text{ if } t = 0 \\ \infty \text{ if } t \neq 0\end{cases} \]

Distribution. \(X \sim N_d (\mu, \Sigma)\)\[\text{pdf} \; f(x) = \frac{e^{-\frac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu)} }{(2\pi)^{d/2} (\det \Sigma)^{1/2} } \]
\[\text{Range of X}: \mathbb{R}^d\]
\[\text{Parameter Range}:  \mu \in \mathbb{R}^d, \Sigma \text{ pos. def.}\]
\[\mathbb{E}(X) =  \mu\]
\[\text{Var}(X): \text{Cov} (X_i, X_j) = \Sigma_{ij} \]
\[\mathbb{E}(e^{t^T X}) =  e^{t^T \mu + t^T \Sigma t / 2} \]
Back Extra: We can also define the degenerate normal distribution: say \(X \sim N(\mu, 0)\) if \(\mathbb{P}(X = \mu) = 1\). Then we say \(X = (X_1, \dots , X_d)^T \sim N_d(\mu, \Sigma)\) if every linear combination \(t_1 X_1 + \dots + t_d X_d\) has a (possibly degenerate) univariate normal distribution. The more general definition includes situations like the followsing: let \(X_1 \sim N(0,1)\) and let \(X := (X_1, X_1)^T\). Then \(X \sim N_2(0, \Sigma)\) where \(\Sigma = \left ( \begin{matrix} 1 & 1 \\ 1 & 1 \end{matrix} \right )\) . In this case, \(\det \Sigma = 0\)