$$\DeclareMathOperator{\Var}{Var} \DeclareMathOperator{\Var}{Var} % DeclareMathOperator provide more space , see https://tex.stackexchange.com/questions/67506/newcommand-vs-declaremathoperator \DeclareMathOperator{\var}{var} \DeclareMathOperator{\Cov}{Cov} \DeclareMathOperator{\cov}{cov} \DeclareMathOperator{\E}{E} \DeclareMathOperator{\bSigma}{\boldsymbol{\Sigma}} \newcommand{\bs}{\boldsymbol} \newcommand{\red}{\color{red}} \newcommand{\green}{\color{green}} \newcommand{\black}{\color{black}} \newcommand{\grey}{\color{grey}} \newcommand{\purple}{\color{purple}} \newcommand{\blue}{\color{blue}}$$

5  Appendix

5.1 Calculate loglikelihood base on least square

In linear regression with fixed \(x\), the least square estimator and maximum likelihood estimator are equivalent. Therefore it is very easy to derive the loglikelihood of a given linear regression model from its least square estimation. Assume we have \[y_i=\beta_0+\beta_1x_{1i}+\epsilon_i\]

\[\begin{align*} L&=\prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}}\exp-\frac{(y_i-\beta_0-\beta_1x_i)^2}{2\sigma^2}\\ \ell&=\sum_{i=1}^{n}-\frac{1}{2}\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^n(e_i^2)\\ &=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^n(e_i^2). \end{align*}\]

The least square estimator and the normal maximum likelihood estimator of \(\sigma^2\) is \[\hat{\sigma}^2=s^2=\frac{1}{n-2}\sum_{i=1}^n(e_i^2),\]

thus we have \[\begin{align*} \ell&=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\frac{1}{n-2})-\frac{n}{2}\log\left(\sum_{i=1}^n(e_i^2)\right)-\frac{n-2}{2}\\ &=-\frac{n}{2}\log(2\pi)+\frac{n}{2}\log(n-2)-\frac{n}{2}\log\left(\sum_{i=1}^n(e_i^2)\right)-\frac{n}{2}+1\\ &=-\frac{n}{2}\left[\log(2\pi)-\log(n-2)+\log\left(\sum_{i=1}^n(e_i^2)\right)+1-\frac{2}{n}\right] \end{align*}\]

  1. View(getAnywhere(lrtest.default)$objs[[1]]), line 3
  2. View(getAnywhere(logLik.lm)$objs[[1]]), line 22
  3. View(getAnywhere(lrtest.default)$objs[[1]]), line 94, 102

5.2 Sum of squares decomposition for \(F\) test

Assume we have \(p\) predictors, \(n\) observations, \(\boldsymbol{y}'=[y_1,\cdots,y_n]\), \(\boldsymbol{\beta}'=[\beta_0,\beta_1,\cdots,\beta_p]\),

\[\begin{align*} \boldsymbol{X}&=\begin{bmatrix} 1 & x_{11} & x_{12} & \cdots & x_{1p}\\ 1 & x_{21} & x_{22} & \cdots & x_{2p}\\ \vdots &&&& \\ 1 & x_{n1} & x_{n2} & \cdots & x_{np} \end{bmatrix}. \end{align*}\] Because \[\begin{align*} \boldsymbol{X}\hat{\boldsymbol{\beta}}&=\boldsymbol{y}\nonumber\\ \boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol{\beta}}&=\boldsymbol{X}'\boldsymbol{y}\nonumber\\ \boldsymbol{X}'(\boldsymbol{X}\hat{\boldsymbol{\beta}}-\boldsymbol{y})&=0\nonumber\\ \boldsymbol{X}'\boldsymbol{e}&=0\nonumber\\ \begin{bmatrix} 1 & 1 & \cdots & 1\\ x_{11} & x_{21} & \cdots & x_{n1} \\ x_{12} & x_{22} & \cdots & x_{n2} \\ \vdots &&& \\ x_{1p} & x_{2p} & \cdots & x_{np} \\ \end{bmatrix}\times\begin{bmatrix} e_1 \\ e_2 \\ \vdots\\ e_n \end{bmatrix} &= \boldsymbol{0}\nonumber\\ \begin{bmatrix} \sum_{i=1}^{n}e_i \\ \sum_{i=1}^{n}x_{i1}e_i \\ \sum_{i=1}^{n}x_{i2}e_i \\ \vdots\\ \sum_{i=1}^{n}x_{ip}e_i \\ \end{bmatrix}&=\boldsymbol{0}. \end{align*}\] thus we have \[\begin{align*} 2\sum_{i=1}^{n}(y_i-\hat{y}_i)(\hat{y}_i-\bar{y})&=2\sum_{i=1}^{n}e_i(\hat{y}_i-\bar{y})\nonumber\\ &=\sum_{i=1}^{n}\hat{y}_ie_i-\sum_{i=1}^{n}\bar{y}e_i \nonumber \\ &=\hat{\beta}_0\sum_{i=1}^{n}e_i+\hat{\beta}_1\sum_{i=1}^{n}x_{i1}e_i+\cdots+\hat{\beta}_1\sum_{i=1}^{n}x_{ip}e_i-\bar{y}\sum_{i=1}^{n}e_i\nonumber\\ &=0. \end{align*}\]

Reference: Multiple Linear Regression (MLR) Handouts by Yibi Huang