---

Here, we assume that $x_i$'s are observed values of a random variable $X$. Therefore, we can summarize our model as \begin{align} Y = \beta_0+\beta_1 X +\epsilon, \end{align} where $\epsilon$ is a $N(0,\sigma^2)$ random variable independent of $X$. First, we take expectation from both sides to obtain \begin{align} %\label{eq:reg-E} EY &= \beta_0+\beta_1 EX +E[\epsilon]\\ &=\beta_0+\beta_1 EX \end{align} Thus, \begin{align} %\label{eq:reg-E} \beta_0=EY-\beta_1 EX. \end{align} Next, we look at $\textrm{Cov}(X,Y)$, \begin{align} \textrm{Cov}(X,Y) &= \textrm{Cov}(X,\beta_0+\beta_1 X +\epsilon)\\ &=\beta_0 \textrm{Cov}(X,1)+\beta_1\textrm{Cov}(X,X)+\textrm{Cov}(X, \epsilon)\\ &=0+\beta_1 \textrm{Cov}(X,X)+0 \quad (\textrm{since $X$ and $\epsilon$ are independent})\\ &=\beta_1 \textrm{Var}(X). \end{align} Therefore, we obtain \begin{align} \beta_1=\frac{\textrm{Cov}(X,Y)}{\textrm{Var}(X)}, \quad \beta_0=EY-\beta_1 EX. \end{align} Now, we can find $\beta_0$ and $\beta_1$ if we know $EX$, $EY$, $\frac{\textrm{Cov}(X,Y)}{\textrm{Var}(X)}$. Here, we have the observed pairs $(x_1,y_1)$, $(x_2,y_2)$, $\cdots$, $(x_n,y_n)$, so we may estimate these quantities. More specifically, we define \begin{align} &\overline{x}=\frac{x_1+x_2+...+x_n}{n},\\ &\overline{y}=\frac{y_1+y_2+...+y_n}{n},\\ &s_{xx}=\sum_{i=1}^n (x_i-\overline{x})^2,\\ &s_{xy}=\sum_{i=1}^{n} (x_i-\overline{x})(y_i-\overline{y}). \end{align} We can then estimate $\beta_0$ and $\beta_1$ as \begin{align} &\hat{\beta_1}=\frac{s_{xy}}{s_{xx}},\\ &\hat{\beta_0}=\overline{y}-\hat{\beta_1} \overline{x}. \end{align} The above formulas give us the regression line \begin{align} \hat{y} = \hat{\beta_0}+\hat{\beta_1} x. \end{align} For each $x_i$, the fitted value $\hat{y}_i$ is obtained by \begin{align} \hat{y}_i = \hat{\beta_0}+\hat{\beta_1} x_i. \end{align} Here, $\hat{y}_i$ is the predicted value of $y_i$ using the regression formula. The errors in this prediction are given by \begin{align} e_i=y_i-\hat{y}_i, \end{align} which are called the residuals.

---

Example
Consider the following observed values of $(x_i,y_i)$: \begin{equation} (1,3) \quad (2,4) \quad (3,8) \quad (4,9) \end{equation}

1. Find the estimated regression line \begin{align} \hat{y} = \hat{\beta_0}+\hat{\beta_1} x, \end{align} based on the observed data.
2. For each $x_i$, compute the fitted value of $y_i$ using \begin{align} \hat{y}_i = \hat{\beta_0}+\hat{\beta_1} x_i. \end{align}
3. Compute the residuals, $e_i=y_i-\hat{y}_i$ and note that \begin{align} \sum_{i=1}^{4} e_i =0. \end{align}

---

We can use MATLAB or other software packages to do regression analysis. For example, the following MATLAB code can be used to obtain the estimated regression line in Example 8.31.

Coefficient of Determination ($R$-Squared):

Let's look again at the above model for regression. We wrote \begin{align} Y = \beta_0+\beta_1 X +\epsilon, \end{align} where $\epsilon$ is a $N(0,\sigma^2)$ random variable independent of $X$. Note that, here, $X$ is the only variable that we observe, so we estimate $Y$ using $X$. That is, we can write \begin{align} \hat{Y} = \beta_0+\beta_1 X. \end{align} The error in our estimate is \begin{align} Y-\hat{Y}=\epsilon. \end{align} Note that the randomness in $Y$ comes from two sources: $X$ and $\epsilon$. More specifically, if we look at $\textrm{Var}(Y)$, we can write \begin{align} \textrm{Var}(Y) &= \beta_1^2 \textrm{Var}(X) +\textrm{Var}(\epsilon) \quad (\textrm{since $X$ and $\epsilon$ are assumed to be independent}). \end{align} The above equation can be interpreted as follows. The total variation in $Y$ can be divided into two parts. The first part, $\beta_1^2 \textrm{Var}(X)$, is due to variation in $X$. The second part, $\textrm{Var}(\epsilon)$, is the variance of error. In other words, $\textrm{Var}(\epsilon)$ is the variance left in $Y$ after we know $X$. If the variance of error, $\textrm{Var}(\epsilon)$, is small, then $Y$ is close to $\hat{Y}$, so our regression model will be successful in estimating $Y$. From the above discussion, we can define \begin{align} \rho^2=\frac{\beta_1^2 \textrm{Var}(X)}{\textrm{Var}(Y)} \end{align} as the portion of variance of $Y$ that is explained by variation in $X$. From the above discussion, we can also conclude that $0 \leq \rho^2 \leq 1$. More specifically, if $\rho^2$ is close to $1$, $Y$ can be estimated very well as a linear function of $X$. On the other hand if $\rho^2$ is small, then the variance of error is large and $Y$ cannot be accurately estimated as a linear function of $X$. Since $\beta_1=\frac{\textrm{Cov}(X,Y)}{\textrm{Var}(X)}$, we can write \begin{align} \label{eq:rho-reg} \rho^2=\frac{\beta_1^2 \textrm{Var}(X)}{\textrm{Var}(Y)}=\frac{\left[\textrm{Cov}(X,Y)\right]^2}{\textrm{Var}(X) \textrm{Var}(Y)} \hspace{30pt} (8.6) \end{align} The above equation should look familiar to you. Here, $\rho$ is the correlation coefficient that we have seen before. Here, we are basically saying that if $X$ and $Y$ are highly correlated (i.e., $\rho(X,Y)$ is large), then $Y$ can be well approximated by a linear function of $X$, i.e., $Y \approx \hat{Y}=\beta_0+\beta_1 X$.

We conclude that $\rho^2$ is an indicator showing the strength of our regression model in estimating (predicting) $Y$ from $X$. In practice, we often do not have $\rho$ but we have the observed pairs $(x_1,y_1)$, $(x_2,y_2)$, $\cdots$, $(x_n,y_n)$. We can estimate $\rho^2$ from the observed data. We show it by $r^2$ and call it $R$-squared or coefficient of determination.

Two sets of data pairs are shown in Figure 8.12. In both data sets, the values of the $y_i$'s (the heights of the data points) have considerable variation. The data points shown in (a) are very close to the regression line. Therefore, most of the variation in $y$ is explained by the regression formula. That is, here, the $\hat{y_i}$'s are relatively close to the $y_i$'s, so $r^2$ is close to $1$. On the other hand, for the data shown in (b), a lot of variation in $y$ is left unexplained by the regression model. Therefore, $r^2$ for this data set is much smaller than $r^2$ for the data set in (a).

---

Example
For the data in Example 8.31, find the coefficient of determination.

---