Professor, Department of Statistics
2026-04-07
\[\mathbf X =\begin{pmatrix} X_{11} & X_{12} & \cdots & X_{1j} & \cdots & X_{1p}\\ X_{21} & X_{22} & \cdots & X_{2j} & \cdots & X_{2p}\\ \cdots & \cdots & \cdots & \cdots & \cdots & \cdots\\ X_{i1} & X_{i2} & \cdots & \color{red}{X_{ij}} & \cdots & X_{ip}\\ \cdots & \cdots & \cdots & \cdots & \cdots & \cdots\\ X_{n1} & X_{n2} & \cdots & X_{nj} & \cdots & X_{np}\\ \end{pmatrix}\]
\[\mathbf X =\begin{pmatrix} X_{11} & X_{12} & \cdots & X_{1j} & \cdots & X_{1p}\\ X_{21} & X_{22} & \cdots & X_{2j} & \cdots & X_{2p}\\ \cdots & \cdots & \cdots & \cdots & \cdots & \cdots\\ \color{red}{X_{i1}} & \color{red}{X_{i2}} & \color{red}{\cdots} & \color{red}{X_{ij}} & \color{red}{\cdots} & \color{red}{X_{ip}}\\ \cdots & \cdots & \cdots & \cdots & \cdots & \cdots\\ X_{n1} & X_{n2} & \cdots & X_{nj} & \cdots & X_{np}\\ \end{pmatrix}\]
Recall that each row is a random vector from a p-dimensional multivariate distribution.
\(\mathbf X\) has \(n\) rows and \(p\) columns.
The \(n\) rows are independent and identically distributed random vectors.
We say the \(n\) rows are a random sample from a p-dimensional multivariate distribution.
A random variable is a numerical quantity that takes on different values with certain probabilities.
e.g., a normal distributed random variable takes values between \(-\infty\) to \(\infty\).
It represents the outcome of a random event or experiment.
e.g., the BMI of a randomly chosen adult living in Canada
Random variables can be discrete or continuous.
Let \((X_1, \cdots, X_n)\) be a simple random sample from a distribution with mean \(\mu\) and variance \(\sigma^2\). The notation we will use is \[X_1, \cdots, X_n \overset{iid}\sim (\mu, \sigma^2)\]
Summary Statistics and their Expectations:
The proof of unbiasedness follows from the linearity of the expected value operator: \[\begin{aligned} E(\bar{X}) &= E \left(\frac{1}{n} \sum_{i=1}^{n} X_i \right) = \frac{1}{n} \sum_{i=1}^{n} E(X_i) = \frac{1}{n} \sum_{i=1}^{n} \mu = \mu \end{aligned}\]
The unbiasedness of the sample mean is a fundamental property of statistical estimation.
\[\begin{aligned} Var(\bar{X}) &= Var \left(\frac{1}{n} \sum_{i=1}^{n} X_i \right) = \frac{1}{n^2} Var \left(\sum_{i=1}^{n} X_i \right)\\ &= \frac{1}{n^2} \sum_{i=1}^{n} Var(X_i) = \frac{1}{n^2} \sum_{i=1}^{n} \sigma^2 = \frac{\sigma^2}{n} \end{aligned}\]
\[\boldsymbol \mu=E(\mathbf{X})\overset{def}=\begin{pmatrix}\mu_1\\ \vdots \\\mu_p \end{pmatrix}, \] where \(\mu_i=E(X_i), i=1, \cdots, p\).
\[\begin{aligned} \Sigma &\overset{def} = E[(\mathbf X - \boldsymbol \mu)(\mathbf X - \boldsymbol \mu)^T]\\ &= \begin{bmatrix} Var(X_1) & Cov(X_1, X_2) & \cdots & Cov(X_1, X_p) \\ Cov(X_2, X_1) & Var(X_2) & \cdots & Cov(X_2, X_p) \\ \vdots & \vdots & \ddots & \vdots \\ Cov(X_p, X_1) & Cov(X_n, X_2) & \cdots & Var(X_p) \end{bmatrix} \end{aligned}\]
A correlation matrix is a table showing correlation coefficients between different variables.
The correlation coefficient measures the strength and direction of the linear relationship between two variables.
\[ \text{Corr}(X_i,X_j)=\frac{\text{Cov}(X_i,X_j)}{\sqrt{Var(X_i)}\sqrt{Var(X_j)}}=\frac{\sigma_{ij}}{\sigma_i \sigma_j} \]
\[ \mathbf{R} = \begin{bmatrix} 1 & \text{Corr}(X_1,X_2) & \cdots & \text{Corr}(X_1,X_p) \\ \text{Corr}(X_2,X_1) & 1 & \cdots & \text{Corr}(X_2,X_p) \\ \vdots & \vdots & \ddots & \vdots \\ \text{Corr}(X_p,X_1) & \text{Corr}(X_p,X_2) & \cdots & 1 \end{bmatrix} \]
The diagonal \(\rho_{ii}\) of the correlation matrix shows the correlation of each variable with itself, which is always equal to
The matrix is symmetric since the correlation between X and Y is the same as the correlation between Y and X: \(\rho_{ij}=\rho_{ji}\).
Correlation matrix can help identify variables that are correlated.
\[ \begin{aligned} \mathbf{Cov}(\mathbf{X},\mathbf{Y}) & \overset{def}= E[(\mathbf X-\boldsymbol \mu_X)(\mathbf Y-\boldsymbol \mu_Y)^T]\\ &= \begin{bmatrix} \text{Cov}(X_1,Y_1) & \cdots & \text{Cov}(X_1,Y_q) \\ \vdots & \ddots & \vdots \\ \text{Cov}(X_p,Y_1) & \cdots & \text{Cov}(X_p,Y_q) \end{bmatrix} \end{aligned}\]
E.g.,
\[\mathbf X_{2\times 1}=\begin{pmatrix}X_1 & X_2\end{pmatrix}^T, \mathbf Y_{3\times 1}=\begin{pmatrix}Y_1 & Y_2 & Y_3\end{pmatrix}^T\]
\[ \begin{aligned} Cov(\mathbf X,\mathbf Y)=& E[(X-E(X))(Y-E(Y))^T]\\ =&E[\begin{pmatrix}X_1 -\mu_{x1} \\ X_2- \mu_{x2}\end{pmatrix} (Y_1-\mu_{y1}, Y_2 -\mu_{y2}, Y_3-\mu_{y3})]\\ =&\begin{bmatrix} E[(X_1 -\mu_{x1})(Y_1 -\mu_{y1})] &E[(X_1 -\mu_{x1})(Y_2 -\mu_{y2})] &E[(X_1 -\mu_{x1})(Y_3 -\mu_{y3})] \\ E[(X_2 -\mu_{x2})(Y_1 -\mu_{y1})] &E[(X_2 -\mu_{x2})(Y_2 -\mu_{y2})] &E[(X_3 -\mu_{x3})(Y_3 -\mu_{y3})]\end{bmatrix}\\ =&\begin{bmatrix} Cov(X_1, Y_1) & Cov(X_1, Y_2) & Cov(X_1, Y_3)\\ Cov(X_2, Y_1) & Cov(X_2, Y_2) & Cov(X_2, Y_3) \end{bmatrix} \end{aligned}\]
Note: \(\mathbf{Cov}(\mathbf{X},\mathbf{Y}) = [\mathbf{Cov}(\mathbf{Y},\mathbf{X})]^T\)
Consider a random sample from a distribution with mean vector \(\boldsymbol \mu_{p\times 1}\) and covariance \(\boldsymbol \Sigma_{p\times p}\)
A random sample of random vectors is a collection of \(n\) independent and identically distributed random vectors, denoted as \(\mathbf{X}_1, \mathbf{X}_2, \dots, \mathbf{X}_n\).
The random sample of random vectors is denoted by \[\mathbf X_{n\times p} = \begin{pmatrix}\mathbf X_1^T \\ \vdots \\ \mathbf X_n^T\end{pmatrix}\]
Each random vector \(\mathbf{X}_i\) is of dimension \(p\) and can be represented as: \[ \mathbf{X}_i = (X_{i1}, X_{i2}, \dots, X_{ip})^T \]
\[ E[\bar{\mathbf{X}}] = E\left[\frac{1}{n}\sum_{i=1}^n \mathbf{X}_i\right] = \frac{1}{n}\sum_{i=1}^n E[\mathbf{X}_i] = \frac{1}{n}\sum_{i=1}^n \boldsymbol{\mu} = \boldsymbol{\mu} \]
\[Cov(\bar{\mathbf{X}}) = Cov(\frac{1}{n}\sum_{i=1}^n \mathbf{X}_i)= \frac{1}{n^2}\sum_{i=1}^n Cov(\mathbf{X}_i)=\frac{1}{n}\boldsymbol \Sigma\]
\[ \mathbf{S} = \frac{1}{n-1}\sum_{i=1}^n (\mathbf{X}_i - \bar{\mathbf{X}})(\mathbf{X}_i - \bar{\mathbf{X}})^T \] - Next, we show that the sample covariance matrix \(\mathbf{S}\) is an unbiased estimator of \(\boldsymbol{\Sigma}\):
\[ \mathbb{E}[\mathbf{S}] = \boldsymbol{\Sigma} \]
-Proof. By the definition of Cov, we have \[\begin{aligned} \boldsymbol \Sigma &=E[(\mathbf X_i-\boldsymbol \mu)(\mathbf X_i-\boldsymbol \mu)^T] \\ &= E[\mathbf X_i \mathbf X_i^T - \boldsymbol \mu\mathbf X_i^T - \mathbf X_i \boldsymbol \mu^T + \boldsymbol \mu \boldsymbol \mu^T]\\ &= E[\mathbf X_i \mathbf X_i^T] - \boldsymbol \mu E[\mathbf X_i^T] - E[\mathbf X_i] \boldsymbol \mu^T + \boldsymbol \mu \boldsymbol \mu^T\\ &= E[\mathbf X_i \mathbf X_i^T] - \boldsymbol \mu \boldsymbol \mu^T - \boldsymbol \mu \boldsymbol \mu^T + \boldsymbol \mu \boldsymbol \mu^T\\ &= E[\mathbf X_i \mathbf X_i^T] - \boldsymbol \mu \boldsymbol \mu^T \end{aligned}\] As a result, \(E[\mathbf X_i \mathbf X_i^T]= \boldsymbol \mu \boldsymbol \mu^T + \boldsymbol \Sigma\). - Similarly, we have Lemma 2: \[E(\bar {\mathbf X} \bar {\mathbf X}^T)=\boldsymbol \mu \boldsymbol \mu^T + Cov(\bar {\mathbf X})=\boldsymbol \mu \boldsymbol \mu^T + \frac{1}{n}\boldsymbol \Sigma\]
Proof:Expand the product:
\[ \begin{aligned} \mathbf{S} &= \frac{1}{n-1}\sum_{i=1}^n (\mathbf{X}_i \mathbf{X}_i^T - \mathbf{X}_i \bar{\mathbf{X}}^T - \bar{\mathbf{X}} \mathbf{X}_i^T + \bar{\mathbf{X}} \bar{\mathbf{X}}^T) \\ &=\frac{1}{n-1}[\sum_{i=1}^n \mathbf{X}_i \mathbf{X}_i^T- n\bar{\mathbf X}\bar{\mathbf X}^T -n\bar{\mathbf X}\bar{\mathbf X}^T + n\bar{\mathbf X}\bar{\mathbf X}^T]\\ &= \frac{1}{n-1}\sum_{i=1}^n \mathbf{X}_i \mathbf{X}_i^T - \frac{n}{n-1}\bar{\mathbf{X}} \bar{\mathbf{X}}^T \end{aligned} \]
\[\begin{aligned} (n-1)S &= \sum (X_i- \bar X) (X_i- \bar X)^T\\ &= \begin{pmatrix}X_1 - \bar X & \cdots X_n -\bar X\end{pmatrix} \begin{pmatrix}(X_1 - \bar X)^T \\ \vdots \\(X_n - \bar X)^T\end{pmatrix} \end{aligned} \]
Note that \[\begin{pmatrix}(X_1 - \bar X)^T \\ \vdots \\(X_n - \bar X)^T\end{pmatrix}= \begin{pmatrix}X_1 - \bar X \\ \vdots \\ X_n - \bar X\end{pmatrix}^T=\mathbf C\mathbf X\] where \(\mathbf C\) is the centering matrix defined in assignment 1, i.e., \(\mathbf C = \mathbf I_n - \frac{1}{n} \mathbf 1 \mathbf 1^T\). In addition, it can be verified that \(\mathbf C^T\mathbf C=\mathbf C\).
Therefore, \[(n-1)S=(C\mathbf X)^T C\mathbf X= \mathbf X^T C \mathbf X\]
Let \(\mathbf{X}\) be a \(p\)-dimensional random vector with mean vector \(\boldsymbol{\mu}\) and covariance matrix \(\boldsymbol{\Sigma}\).
Consider a linear combination of the form: \[ Y = \mathbf{a}^T\mathbf{X}\]
where \(\mathbf{a}\) is a \(p\)-dimensional constant vector.
E.g., \(\mathbf X = (X_1, X_2, X_3)^T\), \(a=(1/3, 1/3, 1/3)^T\). Then \[Y=a^TX=\frac{1}{3}(X_1 + X_2 + X_3)\]
The mean of \(Y\) can be expressed as: \[ \begin{aligned} E(Y) &= E(\mathbf{a}^T\mathbf{X}) \\ &= \mathbf{a}^T E(\mathbf{X}) \\ &= \mathbf{a}^T \boldsymbol{\mu} \end{aligned} \]
Intuitively, the mean of \(Y\) is a weighted average of the components of \(\mathbf{X}\), with weights given by the corresponding components of \(\mathbf{a}\).
The variance of \(Y\) depends on the covariance structure of \(\mathbf{X}\), as well as the weights given by \(\mathbf{a}\).
We call forms like \(\mathbf a^T \boldsymbol \Sigma \mathbf a\) as quadratic forms.
Note, we can also write the variance of \(Y\) as:
\[ \begin{aligned} \mathbf a^T \boldsymbol \Sigma a &&\\ =& \sum_i\sum_j \sigma_{ij} a_i a_j&\\ =& \sigma_{11}a_1^2 + \sigma_{12}a_1 a_2 + \cdots + \sigma_{1p}a_1 a_p + \sigma_{21}a_2 a_1 + \cdots + \sigma_{p1}a_p a_1 + \cdots&\\ =& \sigma_{11}a_1^2 + \sigma_{22}a_2^2 + \cdots + \sigma_{pp}a_p^2 + 2(\sigma_{12}a_1 a_2+ \sigma_{13}a_1 a_3 + \cdots + \sigma_{p-1,p}a_{p-1} a_p)&\\ \end{aligned}\]
\[ b=\begin{pmatrix} 1/4 \\ 1/4 \\ 1/4 \\ 1/4 \end{pmatrix} \]
\[\begin{aligned} Y&=\mathbf Xb= \begin{pmatrix} X_1^T \\ \vdots \\ X_n^T \end{pmatrix} b = \begin{pmatrix} X_1^Tb \\ \vdots \\ X_n^Tb \end{pmatrix} =\begin{pmatrix} \frac{x_{11} +x_{12} + x_{13} + x_{14}}{4} \\ \vdots \\ \frac{x_{n1} +x_{n2} + x_{n3} + x_{n4}}{4} \end{pmatrix} \end{aligned} \]
