STAT200C Assignment 2

Author

Zhaoxia Yu

This material is for the use of the students enrolled in the course. It is protected by United States copyright law [Title 17, U.S. Code]. It should not be distributed without written permission from the instructor Zhaoxia Yu. In particular, please do not upload it to online “resource” websites such as www.coursehero.com.

Part I

  1. Show that

\[ \|Y - \hat{Y}_H\|^2 - \|Y - \hat{Y}\|^2 = \sigma^2 \hat{\lambda}_H^T \left(\operatorname{var}[\hat{\lambda}_H]\right)^{-1} \hat{\lambda}_H, \]

where the notations and assumptions were provided in a STAT200C lecture.

  1. Let \(\hat{\beta}_H\) be the LSE subject to the linear restriction \(A\beta = c\) in the linear model \(Y = X\beta + \epsilon\) with \(\epsilon \sim (0, \sigma^2 I)\). Suppose that both \(A\) and \(X\) have column full rank.

    1. Give the mean and covariance matrix of \(\hat{\beta}_H\).

    2. Show that

\[ \operatorname{Cov}(\hat{\beta}) - \operatorname{Cov}(\hat{\beta}_H) \]

is positive semidefinite, which implies that

\[ \operatorname{Var}(\hat{\beta}_i) \geq \operatorname{Var}(\hat{\beta}_{H_i}), \qquad i = 1, \ldots, p. \]

  1. Adding covariates. Let \(\hat{\beta}_{old}\) be the LSE from \(Y = X\beta + \epsilon\). Let

\[ \begin{pmatrix} \hat{\beta} \\ \hat{\gamma} \end{pmatrix} \]

be the LSE from \(Y = X\beta + Z\gamma + \epsilon\). Here both \(X\) and \(Z\) have full ranks, and the columns of \(X\) and the columns of \(Z\) are linearly independent. Use the inverse of a partitioned matrix formula to show that

\[ \hat{\gamma} = (Z^T R Z)^{-1} Z^T RY, \qquad \text{where } R = I - P_X, \]

and

\[ \hat{\beta} = \hat{\beta}_{old} - (X^T X)^{-1} X^T Z \hat{\gamma}. \]

Hint: Let \(W = (X, Z)\). The following inverse of a partitioned matrix can be used to find \((W^T W)^{-1}\):

\[ \begin{pmatrix} A_{11} & A_{12} \\ A_{21} & A_{22} \end{pmatrix}^{-1} = \begin{pmatrix} A^{11} & A^{12} \\ A^{21} & A^{22} \end{pmatrix} = \begin{pmatrix} A_{11}^{-1} + B_{12} B_{22}^{-1} B_{21} & - B_{12} B_{22}^{-1} \\ - B_{22}^{-1} B_{21} & B_{22}^{-1} \end{pmatrix}, \]

where

\[ B_{22} = A_{22} - A_{21} A_{11}^{-1} A_{12}, \qquad B_{12} = A_{11}^{-1} A_{12}, \qquad B_{21} = A_{21} A_{11}^{-1}. \]

Part II

Part II of Assignment 2: due on Monday in week 10. Use the add-one-covariate-at-a-time method we discussed in class to write a function for computing the LSE.

  • Please remember that you cannot use any function or package that gives you the inverse of a matrix.

  • Include an intercept term in the model.

  • You can assume that the design matrix is full rank, meaning all columns are linearly independent.

  • Please include your code and an application to a real or simulated data set.