Consider the research question: "Is a regression model containing at least one predictor useful in predicting the size of the infarct?" Are you convinced that testing the following hypotheses helps answer the research question?

• \(H_{0}\colon \beta_{1} = \beta_{2} = \beta_{3} = 0\)
• \(H_{A}\colon\) At least one \(\beta_{i} \ne 0\) (for i = 1, 2, 3)

In this case, the researchers are interested in testing that all three slope parameters are zero. We'll soon see that the null hypothesis is tested using the analysis of variance F-test.

Consider the research question: "Is the size of the infarct significantly (linearly) related to the area of the region at risk?" Are you convinced that testing the following hypotheses helps answer the research question?

• \(H_{0} \colon \beta_{1} = 0\)
• \(H_{A} \colon \beta_{1} ≠ 0\)

In this case, the researchers are interested in testing that just one of the three slope parameters is zero. You already know how to do this, don't you? Wouldn't this just involve performing a t-test for \(\beta_{1}\)? We'll soon learn how to think about the t-test for a single slope parameter in the multiple regression framework.

Consider the researcher's primary research question: "Is the size of the infarct area significantly (linearly) related to the type of treatment after controlling for the size of the region at risk for infarction?" Are you convinced that testing the following hypotheses helps answer the research question?

• \(H_{0} \colon \beta_{2} = \beta_{3} = 0\)
• \(H_{A} \colon\) At least one \(\beta_{i} ≠ 0\) (for i = 2, 3)

In this case, the researchers are interested in testing whether a subset (more than one, but not all) of the slope parameters are simultaneously zero. We will learn a general linear F-test for testing such a hypothesis.

Unfortunately, we can't just jump right into the hypothesis tests. We first have to take two side trips — the first one to learn what is called "the general linear F-test."

The "full model", which is also sometimes referred to as the "unrestricted model," is the model thought to be most appropriate for the data. For simple linear regression, the full model is:

\(y_i=(\beta_0+\beta_1x_{i1})+\epsilon_i\)

Here's a plot of a hypothesized full model for a set of data that we worked with previously in this course (student heights and grade point averages):

And, here's another plot of a hypothesized full model that we previously encountered (state latitudes and skin cancer mortalities):

In each plot, the solid line represents what the hypothesized population regression line might look like for the full model. The question we have to answer in each case is "does the full model describe the data well?" Here, we might think that the full model does well in summarizing the trend in the second plot but not the first.

The "reduced model," which is sometimes also referred to as the "restricted model," is the model described by the null hypothesis \(H_{0}\). For simple linear regression, a common null hypothesis is \(H_{0} : \beta_{1} = 0\). In this case, the reduced model is obtained by "zeroing out" the slope \(\beta_{1}\) that appears in the full model. That is, the reduced model is:

\(y_i=\beta_0+\epsilon_i\)

This reduced model suggests that each response \(y_{i}\) is a function only of some overall mean, \(\beta_{0}\), and some error \(\epsilon_{i}\).

Let's take another look at the plot of student grade point average against height, but this time with a line representing what the hypothesized population regression line might look like for the reduced model:

Not bad — there (fortunately?!) doesn't appear to be a relationship between height and grade point average. And, it appears as if the reduced model might be appropriate in describing the lack of a relationship between heights and grade point averages. What does the reduced model do for the skin cancer mortality example?

It doesn't appear as if the reduced model would do a very good job of summarizing the trend in the population.

How do we decide if the reduced model or the full model does a better job of describing the trend in the data when it can't be determined by simply looking at a plot? What we need to do is to quantify how much error remains after fitting each of the two models to our data. That is, we take the general linear test approach:

• "Fit the full model" to the data.
  • Obtain the least squares estimates of \(\beta_{0}\) and \(\beta_{1}\).
  • Determine the error sum of squares, which we denote as "SSE(F)."
• "Fit the reduced model" to the data.
  • Obtain the least squares estimate of \(\beta_{0}\).
  • Determine the error sum of squares, which we denote as "SSE(R)."

Recall that, in general, the error sum of squares is obtained by summing the squared distances between the observed and fitted (estimated) responses:

\(\sum(\text{observed } - \text{ fitted})^2\)

Therefore, since \(y_i\) is the observed response and \(\hat{y}_i\) is the fitted response for the full model:

\(SSE(F)=\sum(y_i-\hat{y}_i)^2\)

And, since \(y_i\) is the observed response and \(\bar{y}\) is the fitted response for the reduced model:

\(SSE(R)=\sum(y_i-\bar{y})^2\)

Let's get a better feel for the general linear F-test approach by applying it to two different datasets. First, let's look at the Height and GPA data. The following plot of grade point averages against heights contains two estimated regression lines — the solid line is the estimated line for the full model, and the dashed line is the estimated line for the reduced model:

As you can see, the estimated lines are almost identical. Calculating the error sum of squares for each model, we obtain:

\(SSE(F)=\sum(y_i-\hat{y}_i)^2=9.7055\)

\(SSE(R)=\sum(y_i-\bar{y})^2=9.7331\)

The two quantities are almost identical. Adding height to the reduced model to obtain the full model reduces the amount of error by only 0.0276 (from 9.7331 to 9.7055). That is, adding height to the model does very little in reducing the variability in grade point averages. In this case, there appears to be no advantage in using the larger full model over the simpler reduced model.

Look what happens when we fit the full and reduced models to the skin cancer mortality and latitude dataset:

Here, there is quite a big difference between the estimated equation for the full model (solid line) and the estimated equation for the reduced model (dashed line). The error sums of squares quantify the substantial difference in the two estimated equations:

\(SSE(F)=\sum(y_i-\hat{y}_i)^2=17173\)

\(SSE(R)=\sum(y_i-\bar{y})^2=53637\)

Adding latitude to the reduced model to obtain the full model reduces the amount of error by 36464 (from 53637 to 17173). That is, adding latitude to the model substantially reduces the variability in skin cancer mortality. In this case, there appears to be a big advantage in using the larger full model over the simpler reduced model.

Where are we going with this general linear test approach? In short:

• The general linear test involves a comparison between SSE(R) and SSE(F).
• SSE(R) can never be smaller than SSE(F). It is always larger than (or possibly the same as) SSE(F).
  • If SSE(F) is close to SSE(R), then the variation around the estimated full model regression function is almost as large as the variation around the estimated reduced model regression function. If that's the case, it makes sense to use the simpler reduced model.
  • On the other hand, if SSE(F) and SSE(R) differ greatly, then the additional parameter(s) in the full model substantially reduce the variation around the estimated regression function. In this case, it makes sense to go with the larger full model.

How different does SSE(R) have to be from SSE(F) in order to justify using the larger full model? The general linear F-statistic:

\(F^*=\left( \dfrac{SSE(R)-SSE(F)}{df_R-df_F}\right)\div\left( \dfrac{SSE(F)}{df_F}\right)\)

helps answer this question. The F-statistic intuitively makes sense — it is a function of SSE(R)-SSE(F), the difference in the error between the two models. The degrees of freedom — denoted \(df_{R}\) and \(df_{F}\) — are those associated with the reduced and full model error sum of squares, respectively.

We use the general linear F-statistic to decide whether or not:

• to reject the null hypothesis \(H_{0}\colon\) The reduced model
• in favor of the alternative hypothesis \(H_{A}\colon\) The full model

In general, we reject \(H_{0}\) if F* is large — or equivalently if its associated P-value is small.

For simple linear regression, it turns out that the general linear F-test is just the same ANOVA F-test that we learned before. As noted earlier for the simple linear regression case, the full model is:

\(y_i=(\beta_0+\beta_1x_{i1})+\epsilon_i\)

and the reduced model is:

\(y_i=\beta_0+\epsilon_i\)

Therefore, the appropriate null and alternative hypotheses are specified either as:

• \(H_{0} \colon y_i = \beta_{0} + \epsilon_{i}\)
• \(H_{A} \colon y_i = \beta_{0} + \beta_{1} x_{i} + \epsilon_{i}\)

or as:

• \(H_{0} \colon \beta_{1} = 0 \)
• \(H_{A} \colon \beta_{1} ≠ 0 \)

The degrees of freedom associated with the error sum of squares for the reduced model is n-1, and:

\(SSE(R)=\sum(y_i-\bar{y})^2=SSTO\)

The degrees of freedom associated with the error sum of squares for the full model is n-2, and:

\(SSE(F)=\sum(y_i-\hat{y}_i)^2=SSE\)

Now, we can see how the general linear F-statistic just reduces algebraically to the ANOVA F-test that we know:

\(F^*=\left( \dfrac{SSE(R)-SSE(F)}{df_R-df_F}\right)\div\left( \dfrac{SSE(F)}{df_F}\right)\)

Can be rewritten by substituting...

\(\begin{aligned} &&df_{R} = n - 1\\  &&df_{F} = n - 2\\ &&SSE(R)=SSTO\\&&SSE(F)=SSE\end{aligned}\)

\(F^*=\left( \dfrac{SSTO-SSE}{(n-1)-(n-2)}\right)\div\left( \dfrac{SSE}{(n-2)}\right)=\frac{MSR}{MSE}\)

That is, the general linear F-statistic reduces to the ANOVA F-statistic:

\(F^*=\dfrac{MSR}{MSE}\)

For the student height and grade point average example:

\( F^*=\dfrac{MSR}{MSE}=\dfrac{0.0276/1}{9.7055/33}=\dfrac{0.0276}{0.2941}=0.094\)

For the skin cancer mortality example:

\( F^*=\dfrac{MSR}{MSE}=\dfrac{36464/1}{17173/47}=\dfrac{36464}{365.4}=99.8\)

The P-value is calculated as usual. The P-value answers the question: "what is the probability that we’d get an F* statistic as large as we did if the null hypothesis were true?" The P-value is determined by comparing F* to an F distribution with 1 numerator degree of freedom and n-2 denominator degrees of freedom. For the student height and grade point average example, the P-value is 0.761 (so we fail to reject \(H_{0}\) and we favor the reduced model), while for the skin cancer mortality example, the P-value is 0.000 (so we reject \(H_{0}\) and we favor the full model).

The numerator of the general linear F-statistic — that is, \(SSE(R)-SSE(F)\) is what is referred to as a "sequential sum of squares" or "extra sum of squares."

Definition

What is a "sequential sum of squares?"

It can be viewed in either of two ways:

• It is the reduction in the error sum of squares (SSE) when one or more predictor variables are added to the model.
• Or, it is the increase in the regression sum of squares (SSR) when one or more predictor variables are added to the model.

In essence, when we add a predictor to a model, we hope to explain some of the variability in the response, and thereby reduce some of the error. A sequential sum of squares quantifies how much variability we explain (increase in regression sum of squares) or alternatively how much error we reduce (reduction in the error sum of squares).

Notation

The amount of error that remains upon fitting a multiple regression model naturally depends on which predictors are in the model. That is, the error sum of squares (SSE) and, hence, the regression sum of squares (SSR) depend on what predictors are in the model. Therefore, we need a way of keeping track of the predictors in the model for each calculated SSE and SSR value.

We'll just note what predictors are in the model by listing them in parentheses after any SSE or SSR. For example:

• \(SSE({x}_{1})\) denotes the error sum of squares when \(x_{1}\) is the only predictor in the model.
• \(SSR({x}_{1}, {x}_{2})\) denotes the regression sum of squares when \(x_{1}\) and \(x_{2}\) are both in the model.

And, we'll use notation like \(SSR(x_{2} | x_{1})\) to denote a sequential sum of squares. \(SSR(x_{2} | x_{1})\) denotes the sequential sum of squares obtained by adding \(x_{2}\) to a model already containing only the predictor \(x_{1}\). The vertical bar "|" is read as "given" — that is, "\(x_{2}\) | \(x_{1}\)" is read as "\(x_{2}\) given \(x_{1}\)." In general, the variables appearing to the right of the bar "|" are the predictors in the original model, and the variables appearing to the left of the bar "|" are the predictors newly added to the model.

Here are a few more examples of the notation:

• The sequential sum of squares obtained by adding \(x_{1}\) to the model already containing only the predictor \(x_{2}\) is denoted as \(SSR({x}_{1}| {x}_{2})\).
• The sequential sum of squares obtained by adding \(x_{1}\) to the model in which \(x_{2}\) and \(x_{3}\) are predictors is denoted as \(SSR({x}_{1}| {x}_{2},{x}_{3}) \).
• The sequential sum of squares obtained by adding \(x_{1}\) and \(x_{2}\) to the model in which \(x_{3}\) is the only predictor is denoted as \(SSR({x}_{1}, {x}_{2}|{x}_{3}) \).

Let's try out the notation and the two alternative definitions of a sequential sum of squares on an example.

Perhaps, you noticed from the previous illustration that the order in which we add predictors to the model determines the sequential sums of squares ("Seq SS") we get. That is, the order is important! Therefore, we'll have to pay attention to it — we'll soon see that the desired order depends on the hypothesis test we want to conduct.

Let's revisit the Allen Cognitive Level Study data to see what happens when we reverse the order in which we enter the predictors in the model. Let's start by regressing y = ACL on \(\boldsymbol{x_{3}}\) = SDMT (using the Minitab default Adjusted or Type III sums of squares):

Noting that \(x_{3}\) is the only predictor in the model, the resulting output tells us that:

• SSR(\(x_{3}\)) = 11.68
• SSE(\(x_{3}\)) = 31.37

Now, regressing y = ACL on \(\boldsymbol{x_{3}}\) = SDMT and \(\boldsymbol{x_{1}}\) = Vocab — in that order, that is, specifying \(x_{3}\) first and \(x_{1}\) second, we obtain:

Noting that \(x_{1}\) and \(x_{3}\) are the two predictors in the model, the output tells us that:

• SSR(\(x_{1}\), \(x_{3}\)) = 11.7778
• SSE(\(x_{1}\), \(x_{3}\)) = 31.2717

How much did the error sum of squares decrease — or alternatively, the regression sum of squares increase? The sequential sum of squares \(\boldsymbol{SSR}\boldsymbol{({x}_{1}}| \boldsymbol{{x}_{3})}\) tells us how much. \(\boldsymbol{SSR}\boldsymbol{({x}_{1}}| \boldsymbol{{x}_{3})}\) is the reduction in the error sum of squares when \(\boldsymbol{x_{1}}\) is added to the model in which \(\boldsymbol{x_{3}}\) is the only predictor:

\(\boldsymbol{SSR}\boldsymbol{({x}_{1}}| \boldsymbol{{x}_{3})}=\boldsymbol{SSE}\boldsymbol{({x}_{3})}- \boldsymbol{SSE}\boldsymbol{({x}_{1}}, \boldsymbol{{x}_{3})}\)

\(\boldsymbol{SSR}\boldsymbol{({x}_{1}}| \boldsymbol{{x}_{3})}=\boldsymbol{31.37 - 31.2717 = 0.098}\)

Alternatively, \(\boldsymbol{SSR}\boldsymbol{({x}_{1}}| \boldsymbol{{x}_{3})}\) is the increase in the regression sum of squares when \(\boldsymbol{x_{1}}\) is added to the model in which \(\boldsymbol{x_{3}}\) is the only predictor:

\(\boldsymbol{SSR}\boldsymbol{({x}_{1}}| \boldsymbol{{x}_{3})}=\boldsymbol{SSR}\boldsymbol{({x}_{1}}, \boldsymbol{{x}_{3})}- \boldsymbol{SSR}\boldsymbol{({x}_{3})} \) 

\(\boldsymbol{SSR}\boldsymbol{({x}_{1}}| \boldsymbol{{x}_{3})}= \boldsymbol{11.7778 - 11.68 = 0.098}\)

Again, we obtain the same answers. Regardless of how we perform the calculation, it appears that taking into account \(\boldsymbol{x_{1}}\) = Vocab doesn't help much in explaining the variability in y = ACL after \(\boldsymbol{x_{3}}\) = SDMT has already been considered.

Once again, we don't have to calculate sequential sums of squares by hand. Minitab does it for us. If we regress y = ACL on \(x_{3}\) = SDMT and \(x_{1}\) = Vocab in that order and use Sequential (Type I) sums of squares, we obtain:

The Minitab output tells us:

• SSR(\(x_{3}\)) = 11.6799. That is, the error sum of squares is reduced — or the regression sum of squares is increased — by 11.6799 when you add \(x_{3}\) = SDMT to a model containing no predictors.
• SSR(\(x_{1}\) | \(x_{3}\)) = 0.0979. That is, the error sum of squares is reduced — or the regression sum of squares is increased — by (only!) 0.0979 when you add \(x_{1}\) = Vocab to a model already containing \(x_{3}\) = SDMT.

So far, we've only evaluated how much the error and regression sums of squares change when adding one additional predictor to the model. What happens if we simultaneously add two predictors to a model containing only one predictor? We obtain what is called a "two-degree-of-freedom sequential sum of squares." If we simultaneously add three predictors to a model containing only one predictor, we obtain a "three-degree-of-freedom sequential sum of squares," and so on.

There are two ways of obtaining these types of sequential sums of squares. We can:

• either add up the appropriate one-degree-of-freedom sequential sums of squares
• or use the definition of a sequential sum of squares

Let's try out these two methods on our Allen Cognitive Level Study example. Regressing, in order, y = ACL on \(\boldsymbol{x_{3}}\) = SDMT and \(\boldsymbol{x_{1}}\) = Vocab and \(\boldsymbol{x_{2}}\) = Abstract, and using sequential (Type I) sums of squares, we obtain:

The Minitab output tells us:

• SSR(\(x_{3}\)) = 11.6799
• SSR(\(x_{1}\) | \(x_{3}\)) = 0.0979
• SSR(\(x_{2 }\)| \(x_{1}\), \(x_{3}\)) = 0.5230

Therefore, the reduction in the error sum of squares when adding \(x_{1}\) and \(x_{2}\) to a model already containing \(x_{3}\) is:

\(\boldsymbol{SSR}\boldsymbol{({x}_{1}}, \boldsymbol{{x}_{2}}|\boldsymbol{{x}_{3})}= \boldsymbol{0.0979 + 0.5230 = 0.621}\)

Alternatively, we can calculate the sequential sum of squares SSR(\(x_{1}\), \(x_{2}\)| \(x_{3}\)) by definition of the reduction in the error sum of squares:

\(\boldsymbol{SSR}\boldsymbol{({x}_{1}}, \boldsymbol{{x}_{2}}|\boldsymbol{{x}_{3})}= \boldsymbol{SSE}\boldsymbol{({x}_{3})}- \boldsymbol{SSE}\boldsymbol{({x}_{1}}, \boldsymbol{{x}_{2}}, \boldsymbol{{x}_{3})}\)

\(\boldsymbol{SSR}\boldsymbol{({x}_{1}}, \boldsymbol{{x}_{2}}|\boldsymbol{{x}_{3})= 31.37 - 30.7487 = 0.621}\)

or the increase in the regression sum of squares:

\(\boldsymbol{SSR}\boldsymbol{({x}_{1}}, \boldsymbol{{x}_{2}}|\boldsymbol{{x}_{3})}=\boldsymbol{SSR}\boldsymbol{({x}_{1}}, \boldsymbol{{x}_{2}}, \boldsymbol{{x}_{3})}- \boldsymbol{SSR}\boldsymbol{({x}_{3})} \)

\(\boldsymbol{SSR}\boldsymbol{({x}_{1}}, \boldsymbol{{x}_{2}}|\boldsymbol{{x}_{3})}=\boldsymbol{12.3009 - 11.68 = 0.621}\)

Note that the Sequential (Type I) sums of squares in the Anova table add up to the (overall) regression sum of squares (SSR): 11.6799 + 0.0979 + 0.5230 = 12.3009 (within rounding error). By contrast, Adjusted (Type III) sums of squares do not have this property.

We've now finished our second aside. We can — finally — get back to the whole point of this lesson, namely learning how to conduct hypothesis tests for the slope parameters in a multiple regression model.

At the beginning of this lesson, we translated three different research questions pertaining to heart attacks in rabbits (Cool Hearts dataset) into three sets of hypotheses we can test using the general linear F-statistic. The research questions and their corresponding hypotheses are:

1. Hypotheses 1

   Is the regression model containing at least one predictor useful in predicting the size of the infarct?

   • \(H_{0} \colon \beta_{1} = \beta_{2} = \beta_{3} = 0\)
   • \(H_{A} \colon\) At least one \(\beta_{j} ≠ 0\) (for j = 1, 2, 3)

2. Hypotheses 2

   Is the size of the infarct significantly (linearly) related to the area of the region at risk?

   • \(H_{0} \colon \beta_{1} = 0 \)
   • \(H_{A} \colon \beta_{1} \ne 0 \)

3. Hypotheses 3

   (Primary research question) Is the size of the infarct area significantly (linearly) related to the type of treatment upon controlling for the size of the region at risk for infarction?

   • \(H_{0} \colon \beta_{2} = \beta_{3} = 0\)
   • \(H_{A} \colon \) At least one \(\beta_{j} ≠ 0\) (for j = 2, 3)

Let's test each of the hypotheses now using the general linear F-statistic:

\(F^*=\left(\dfrac{SSE(R)-SSE(F)}{df_R-df_F}\right) \div \left(\dfrac{SSE(F)}{df_F}\right)\)

To calculate the F-statistic for each test, we first determine the error sum of squares for the reduced and full models — SSE(R) and SSE(F), respectively. The number of error degrees of freedom associated with the reduced and full models — \(df_{R}\) and \(df_{F}\), respectively — is the number of observations, n, minus the number of parameters, p, in the model. That is, in general, the number of error degrees of freedom is n-p. We use statistical software, such as Minitab's F-distribution probability calculator, to determine the P-value for each test.

To answer the research question: "Is the regression model containing at least one predictor useful in predicting the size of the infarct?" To do so, we test the hypotheses:

• \(H_{0} \colon \beta_{1} = \beta_{2} = \beta_{3} = 0 \)
• \(H_{A} \colon\) At least one \(\beta_{j} \ne 0 \) (for j = 1, 2, 3)

1. The full model

   The full model is the largest possible model — that is, the model containing all of the possible predictors. In this case, the full model is:

   \(y_i=(\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\beta_3x_{i3})+\epsilon_i\)

   The error sum of squares for the full model, SSE(F), is just the usual error sum of squares, SSE, that appears in the analysis of variance table. Because there are 4 parameters in the full model, the number of error degrees of freedom associated with the full model is \(df_{F} = n - 4\).

2. The reduced model

   The reduced model is the model that the null hypothesis describes. Because the null hypothesis sets each of the slope parameters in the full model equal to 0, the reduced model is:

   \(y_i=\beta_0+\epsilon_i\)

   The reduced model suggests that none of the variations in the response y is explained by any of the predictors. Therefore, the error sum of squares for the reduced model, SSE(R), is just the total sum of squares, SSTO, that appears in the analysis of variance table. Because there is only one parameter in the reduced model, the number of error degrees of freedom associated with the reduced model is \(df_{R} = n - 1 \).

3. The test

   Upon plugging in the above quantities, the general linear F-statistic:

   \(F^*=\dfrac{SSE(R)-SSE(F)}{df_R-df_F} \div \dfrac{SSE(F)}{df_F}\)

   becomes the usual "overall F-test":

   \(F^*=\dfrac{SSR}{3} \div \dfrac{SSE}{n-4}=\dfrac{MSR}{MSE}\)

   That is, to test \(H_{0}\) : \(\beta_{1} = \beta_{2} = \beta_{3} = 0 \), we just use the overall F-test and P-value reported in the analysis of variance table:

   Analysis of Variance

   Source	DF	Adj SS	Adj MS	F- Value	P-Value
   Regression	3	0.95927	0.31976	16.43	0.000
   Area	1	0.63742	0.63742	32.75	0.000
   X2	1	0.29733	0.29733	15.28	0.001
   X3	1	0.01981	0.01981	1.02	0.322
   Error	28	0.54491	0.01946
   Total	31	1.50418

   Regression Equation

   Inf = - 0.135 + 0.613 Area - 0.2435 X2 - 0.0657 X3

   There is sufficient evidence (F = 16.43, P < 0.001) to conclude that at least one of the slope parameters is not equal to 0.

   In general, to test that all of the slope parameters in a multiple linear regression model are 0, we use the overall F-test reported in the analysis of variance table.

Now let's answer the second research question: "Is the size of the infarct significantly (linearly) related to the area of the region at risk?" To do so, we test the hypotheses:

• \(H_{0} \colon \beta_{1} = 0 \)
• \(H_{A} \colon \beta_{1} \ne 0 \)

1. The full model

   Again, the full model is the model containing all of the possible predictors:

   \(y_i=(\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\beta_3x_{i3})+\epsilon_i\)

   The error sum of squares for the full model, SSE(F), is just the usual error sum of squares, SSE. Alternatively, because the three predictors in the model are \(x_{1}\), \(x_{2}\), and \(x_{3}\), we can denote the error sum of squares as SSE(\(x_{1}\), \(x_{2}\), \(x_{3}\)). Again, because there are 4 parameters in the model, the number of error degrees of freedom associated with the full model is \(df_{F} = n - 4 \).

2. The reduced model

   Because the null hypothesis sets the first slope parameter, \(\beta_{1}\), equal to 0, the reduced model is:

   \(y_i=(\beta_0+\beta_2x_{i2}+\beta_3x_{i3})+\epsilon_i\)

   Because the two predictors in the model are \(x_{2}\) and \(x_{3}\), we denote the error sum of squares as SSE(\(x_{2}\), \(x_{3}\)). Because there are 3 parameters in the model, the number of error degrees of freedom associated with the reduced model is \(df_{R} = n - 3\).

The Test

The general linear statistic:

\(F^*=\dfrac{SSE(R)-SSE(F)}{df_R-df_F} \div \dfrac{SSE(F)}{df_F}\)

simplifies to:

\(F^*=\dfrac{SSR(x_1|x_2, x_3)}{1}\div \dfrac{SSE(x_1,x_2, x_3)}{n-4}=\dfrac{MSR(x_1|x_2, x_3)}{MSE(x_1,x_2, x_3)}\)

Getting the numbers from the Minitab output:

we determine that the value of the F-statistic is:

1. \(F^* = \dfrac{SSR(x_1 \vert x_2, x_3)}{1} \div \dfrac{SSE(x_1, x_2, x_3)}{28} = \dfrac{0.63742}{0.01946}=32.7554\)

   The P-value is the probability — if the null hypothesis were true — that we would get an F-statistic larger than 32.7554. Comparing our F-statistic to an F-distribution with 1 numerator degree of freedom and 28 denominator degrees of freedom, Minitab tells us that the probability is close to 1 that we would observe an F-statistic smaller than 32.7554:

   F distribution with 1 DF in Numerator and 28 DF in denominator

   x	P ( X ≤x )
   32.7554	1.00000

   Therefore, the probability that we would get an F-statistic larger than 32.7554 is close to 0. That is, the P-value is < 0.001. There is sufficient evidence (F = 32.8, P < 0.001) to conclude that the size of the infarct is significantly related to the size of the area at risk after the other predictors x2 and x3 have been taken into account.

   But wait a second! Have you been wondering why we couldn't just use the slope's t-statistic to test that the slope parameter, \(\beta_{1}\), is 0? We can! Notice that the P-value (P < 0.001) for the t-test (t* = 5.72):

   Coefficients

   Term	Coef	SE Coef	T-Value	P-Value	VIF
   Constant	-0.135	0.104	-1.29	0.206
   Area	0.613	0.107	5.72	0.000	1.14
   X2	-0.2435	0.0623	-3.91	0.001	1.44
   X3	-0.0657	0.0651	-1.01	0.322	1.57

   Regression Equation

   Inf = - 0.135 + 0.613 Area - 0.2435 X2 - 0.0657 X3

   is the same as the P-value we obtained for the F-test. This will always be the case when we test that only one slope parameter is 0. That's because of the well-known relationship between a t-statistic and an F-statistic that has one numerator degree of freedom:

   \(t_{(n-p)}^{2}=F_{(1, n-p)}\)

   For our example, the square of the t-statistic, 5.72, equals our F-statistic (within rounding error). That is:

   \(t^{*2}=5.72^2=32.72=F^*\)

   So what have we learned in all of this discussion about the equivalence of the F-test and the t-test? In short:

   Compare the output obtained when \(x_{1}\) = Area is entered into the model last:

   Coefficients

   Term	Coef	SE Coef	T-Value	P-Value	VIF
   Constant	-0.135	0.104	-1.29	0.206
   X2	-0.2435	0.0623	-3.91	0.001	1.44
   X3	-0.0657	0.0651	-1.01	0.322	1.57
   Area	0.613	0.107	5.72	0.000	1.14

   Regression Equation

   Inf = - 0.135 - 0.2435 X2 - 0.0657 X3 + 0.613 Area

   to the output obtained when \(x_{1}\) = Area is entered into the model first:

   Coefficients

   Term	Coef	SE Coef	T-Value	P-Value	VIF
   Constant	-0.135	0.104	-1.29	0.206
   Area	0.613	0.107	5.72	0.000	1.14
   X2	-0.2435	0.0623	-3.91	0.001	1.44
   X3	-0.0657	0.0651	-1.01	0.322	1.57

   Regression Equation

   Inf = - 0.135 + 0.613 Area - 0.2435 X2 - 0.0657 X3

   The t-statistic and P-value are the same regardless of the order in which \(x_{1}\) = Area is entered into the model. That's because — by its equivalence to the F-test — the t-test for one slope parameter adjusts for all of the other predictors included in the model.

   • We can use either the F-test or the t-test to test that only one slope parameter is 0. Because the t-test results can be read right off of the Minitab output, it makes sense that it would be the test that we'll use most often.
   • But, we have to be careful with our interpretations! The equivalence of the t-test to the F-test has taught us something new about the t-test. The t-test is a test for the marginal significance of the \(x_{1}\) predictor after the other predictors \(x_{2}\) and \(x_{3}\) have been taken into account. It does not test for the significance of the relationship between the response y and the predictor \(x_{1}\) alone.

Finally, let's answer the third — and primary — research question: "Is the size of the infarct area significantly (linearly) related to the type of treatment upon controlling for the size of the region at risk for infarction?" To do so, we test the hypotheses:

• \(H_{0} \colon \beta_{2} = \beta_{3} = 0 \)
• \(H_{A} \colon\) At least one \(\beta_{j} \ne 0 \) (for j = 2, 3)

1. The full model

   Again, the full model is the model containing all of the possible predictors:

   \(y_i=(\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\beta_3x_{i3})+\epsilon_i\)

   The error sum of squares for the full model, SSE(F), is just the usual error sum of squares, SSE. Alternatively, because the three predictors in the model are \(x_{1}\), \(x_{2}\), and \(x_{3}\), we can denote the error sum of squares as SSE(\(x_{1}\), \(x_{2}\), \(x_{3}\)). Again, because there are 4 parameters in the model, the number of error degrees of freedom associated with the full model is \(df_{F} = n - 4 \).

2. The reduced model

   Because the null hypothesis sets the second and third slope parameters, \(\beta_{2}\) and \(\beta_{3}\), equal to 0, the reduced model is:

   \(y_i=(\beta_0+\beta_1x_{i1})+\epsilon_i\)

   The ANOVA table for the reduced model is:

   Analysis of Variance

   Source	DF	Adj SS	Adj MS	F- Value	P-Value
   Regression	1	0.6249	0.62492	21.32	0.000
   Area	1	0.6249	0.62492	21.32	0.000
   Error	30	0.8793	0.02931
   Total	31	1.5042

   Because the only predictor in the model is \(x_{1}\), we denote the error sum of squares as SSE(\(x_{1}\)) = 0.8793. Because there are 2 parameters in the model, the number of error degrees of freedom associated with the reduced model is \(df_{R} = n - 2 = 32 – 2 = 30\).

3. The Test

   The general linear statistic:

   \begin{align} F^*&=\dfrac{SSE(R)-SSE(F)}{df_R-df_F} \div\dfrac{SSE(F)}{df_F}\\&=\dfrac{0.8793-0.54491}{30-28} \div\dfrac{0.54491}{28}\\&= \dfrac{0.33439}{2} \div 0.01946\\&=8.59.\end{align}

   Alternatively, we can calculate the F-statistic using a partial F-test:

   \begin{align}F^*&=\dfrac{SSR(x_2, x_3|x_1)}{2}\div \dfrac{SSE(x_1,x_2, x_3)}{n-4}\\&=\dfrac{MSR(x_2, x_3|x_1)}{MSE(x_1,x_2, x_3)}.\end{align}

   To conduct the test, we regress y = InfSize on \(x_{1}\) = Area and \(x_{2}\) and \(x_{3 }\)— in order (and with "Sequential sums of squares" selected under "Options"):

   Analysis of Variance

   Source	DF	Seq SS	Seq MS	F- Value	P-Value
   Regression	3	0.95927	0.31976	16.43	0.000
   Area	1	0.62492	0.63492	32.11	0.000
   X2	1	0.3143	0.31453	16.16	0.001
   X3	1	0.01981	0.01981	1.02	0.322
   Error	28	0.54491	0.01946
   Total	31	1.50418

   Coefficients

   Term	Coef	SE Coef	T-Value	P-Value	VIF
   Constant	-0.135	0.104	-1.29	0.206
   Area	0.613	0.107	5.72	0.000	1.14
   X2	-0.2435	0.0623	-3.91	0.001	1.44
   X3	-0.0657	0.0651	-1.01	0.322	1.57

   Regression Equation

   Inf = - 0.135 + 0.613 Area - 0.2435 X2 - 0.0657 X3

   yielding SSR(\(x_{2}\) | \(x_{1}\)) = 0.31453, SSR(\(x_{3}\) | \(x_{1}\), \(x_{2}\)) = 0.01981, and MSE = 0.54491/28 = 0.01946. Therefore, the value of the partial F-statistic is:

   \begin{align} F^*&=\dfrac{SSR(x_2, x_3|x_1)}{2}\div \dfrac{SSE(x_1,x_2, x_3)}{n-4}\\&=\dfrac{0.31453+0.01981}{2}\div\dfrac{0.54491}{28}\\&= \dfrac{0.33434}{2} \div 0.01946\\&=8.59,\end{align}

   which is identical (within round-off error) to the general F-statistic above. The P-value is the probability — if the null hypothesis were true — that we would observe a partial F-statistic more extreme than 8.59. The following Minitab output:

   F distribution with 2 DF in Numerator and 28 DF in denominator

   x	P ( X ≤ x )
   8.59	0.998767

   tells us that the probability of observing such an F-statistic that is smaller than 8.59 is 0.9988. Therefore, the probability of observing such an F-statistic that is larger than 8.59 is 1 - 0.9988 = 0.0012. The P-value is very small. There is sufficient evidence (F = 8.59, P = 0.0012) to conclude that the type of cooling is significantly related to the extent of damage that occurs — after taking into account the size of the region at risk.

For the simple linear regression model, there is only one slope parameter about which one can perform hypothesis tests. For the multiple linear regression model, there are three different hypothesis tests for slopes that one could conduct. They are:

• Hypothesis test for testing that all of the slope parameters are 0.
• Hypothesis test for testing that a subset — more than one, but not all — of the slope parameters are 0.
• Hypothesis test for testing that one slope parameter is 0.

We have learned how to perform each of the above three hypothesis tests. Along the way, we also took two detours — one to learn about the "general linear F-test" and one to learn about "sequential sums of squares." As you now know, knowledge about both is necessary for performing the three hypothesis tests.

The F-statistic and associated p-value in the ANOVA table is used for testing whether all of the slope parameters are 0. In most applications, this p-value will be small enough to reject the null hypothesis and conclude that at least one predictor is useful in the model. For example, for the rabbit heart attacks study, the F-statistic is (0.95927/(4–1)) / (0.54491/(32–4)) = 16.43 with p-value 0.000.

To test whether a subset — more than one, but not all — of the slope parameters are 0, there are two equivalent ways to calculate the F-statistic:

• Use the general linear F-test formula by fitting the full model to find SSE(F) and fitting the reduced model to find SSE(R). Then the numerator of the F-statistic is (SSE(R) – SSE(F)) / ( \(df_{R}\) – \(df_{F}\)).
• Alternatively, use the partial F-test formula by fitting only the full model but making sure the relevant predictors are fitted last and "sequential sums of squares" have been selected. Then the numerator of the F-statistic is the sum of the relevant sequential sums of squares divided by the sum of the degrees of freedom for these sequential sums of squares. The denominator of the F-statistic is the mean squared error in the ANOVA table.

For example, for the rabbit heart attacks study, the general linear F-statistic is ((0.8793 – 0.54491) / (30 – 28)) / (0.54491 / 28) = 8.59 with p-value 0.0012. Alternatively, the partial F-statistic for testing the slope parameters for predictors \(x_{2}\) and \(x_{3}\) using sequential sums of squares is ((0.31453 + 0.01981) / 2) / (0.54491 / 28) = 8.59.

To test whether one slope parameter is 0, we can use an F-test as just described. Alternatively, we can use a t-test, which will have an identical p-value since in this case, the square of the t-statistic is equal to the F-statistic. For example, for the rabbit heart attacks study, the F-statistic for testing the slope parameter for the Area predictor is (0.63742/1) / (0.54491/(32–4)) = 32.75 with p-value 0.000. Alternatively, the t-statistic for testing the slope parameter for the Area predictor is 0.613 / 0.107 = 5.72 with p-value 0.000, and \(5.72^{2} = 32.72\).

Incidentally, you may be wondering why we can't just do a series of individual t-tests to test whether a subset of the slope parameters is 0. For example, for the rabbit heart attacks study, we could have done the following:

• Fit the model of y = InfSize on \(x_{1}\) = Area and \(x_{2}\) and \(x_{3}\) and use an individual t-test for \(x_{3}\).
• If the test results indicate that we can drop \(x_{3}\) then fit the model of y = InfSize on \(x_{1}\) = Area and \(x_{2}\) and use an individual t-test for \(x_{2}\).

The problem with this approach is we're using two individual t-tests instead of one F-test, which means our chance of drawing an incorrect conclusion in our testing procedure is higher. Every time we do a hypothesis test, we can draw an incorrect conclusion by:

• rejecting a true null hypothesis, i.e., make a type I error by concluding the tested predictor(s) should be retained in the model when in truth it/they should be dropped; or
• failing to reject a false null hypothesis, i.e., make a type II error by concluding the tested predictor(s) should be dropped from the model when in truth it/they should be retained.

Thus, in general, the fewer tests we perform the better. In this case, this means that wherever possible using one F-test in place of multiple individual t-tests is preferable.

Suppose we have set up a general linear F-test. Then, we may be interested in seeing what percent of the variation in the response cannot be explained by the predictors in the reduced model (i.e., the model specified by \(H_{0}\)), but can be explained by the rest of the predictors in the full model. If we obtain a large percentage, then it is likely we would want to specify some or all of the remaining predictors to be in the final model since they explain so much variation.

The way we formally define this percentage is by what is called the partial \(\textbf{R}^{2}\) (or it is also called the coefficient of partial determination). Specifically, suppose we have three predictors: \(x_{1}\), \(x_{2}\), and \(x_{3}\). For the corresponding multiple regression model (with response y), we wish to know what percent of the variation not explained by \(x_1\) is explained by \(x_{2}\) and \(x_{3}\). In other words, given \(x_{1}\), what additional percent of the variation can be explained by \(x_{2}\) and \(x_{3}\)? Note that here the full model will include all three predictors, while the reduced model will only include \(x_{1}\).

After obtaining the relevant ANOVA tables for these two models, the partial \(R^{2}\) is as follows:

\[\begin{align*} R^{2}_{y,2,3|1}&=\frac{\textrm{SSR}(x_{2},x_{3}|x_{1})}{\textrm{SSE}(x_{1})} \\ &=\frac{\textrm{SSR}(x_{1},x_{2},x_{3})-\textrm{SSR}(x_{1})}{\textrm{SSE}(x_{1})}\\ &=\frac{\textrm{SSE}(x_{1})-\textrm{SSE}(x_{1},x_{2},x_{3})}{\textrm{SSE}(x_{1})}\\ &=\frac{\textrm{SSE(reduced)}-\textrm{SSE(full)}}{\textrm{SSE(reduced)}} \end{align*}\]

Then, this gives us the proportion of variation explained by \(x_{2}\) and \(x_{3}\) that cannot be explained by \(x_{1}\). Note that the last line of the above equation is just demonstrating that the partial \(R^{2}\) has a similar form to regular \(R^{2}\).

For the rabbit heart attacks example,

\begin{align} \textrm{SSR}(x_{2},x_{3}|x_{1})&=\textrm{SSR}(x_1,x_{2},x_{3})-\textrm{SSR}(x_{1})\\&=0.95927-0.62492\\&=0.33435\end{align}

\begin{align}\textrm{SSE}(x_{1})&=\textrm{SSTO}-\textrm{SSR}(x_{1})\\&=1.50418-0.62492\\&=0.87926\end{align}

so \(R^{2}_{y,2,3|1}=0.33435/0.87926=0.38\). In other words, \(x_{2}\) and \(x_{3}\) explain 38% of the variation in \(y\) that cannot be explained by \(x_{1}\).

More generally, consider partitioning the predictors \(x_{1},x_{2},\ldots,x_{k}\) into two groups, A and B, containing u and k - u predictors, respectively. The proportion of variation explained by the predictors in group B that cannot be explained by the predictors in group A is given by

\[\begin{align*} R^{2}_{y,B|A}&=\frac{\textrm{SSR}(B|A)}{\textrm{SSE}(A)} \\ &=\frac{\textrm{SSE}(A)-\textrm{SSE}(A,B)}{\textrm{SSE}(A)} \end{align*}\]

Formal lack of fit testing can also be performed in the multiple regression setting; however, the ability to achieve replicates can be more difficult as more predictors are added to the model. Note that the corresponding ANOVA table below is similar to that introduced for the simple linear regression setting. However, now we have p regression parameters and c unique X vectors. Further, each predictor must have the same value for at least two observations for it to be considered a replicate. For example, suppose we have 3 predictors for our model. The observations (40, 10, 12) and (40, 10, 7) are unique levels for our X vectors, whereas the observations (10, 5, 13) and (10, 5, 13) would constitute a replicate.

Formal lack of fit testing in multiple regression can be difficult due to sparse data unless we're analyzing an experiment that was designed to include replicates. However, other methods can be employed for lack of fit testing when we do not have replicates. Such methods involve data subsetting. The basic approach is to establish criteria by introducing indicator variables, which in turn create coded variables. By coding the variables, you can artificially create replicates and then you can proceed with lack of fit testing. Another approach with data subsetting is to look at central regions of the data and treat this as a reduced data set. Then compare this reduced fit to the full fit (i.e., the fit with all of the data), for which the formulas for a lack of fit test can be employed. Be forewarned that these methods should only be used as exploratory methods and they are heavily dependent on what sort of data subsetting method is used.

The next two pages cover the Minitab and R commands for the procedures in this lesson.

Below is a zip file that contains all the data sets used in this lesson:

STAT501_Lesson06.zip

• alcoholarm.txt
• allentest.txt
• coolhearts.txt
• iqsize.txt
• peru.txt
• Physical.txt
• sugarbeets.txt