9.6 - Interactions Between Quantitative Predictors

We introduced interactions in the previous lesson, where we created interaction terms between indicator variables and quantitative predictors to allow for different "slopes" for levels of a categorical predictor. We can also create interaction terms between quantitative predictors, which allow the relationship between the response and one predictor to vary with the values of another predictor. Interestingly, this provides another way to introduce curvature into a multiple linear regression model. For example, consider a model with two quantitative predictors, which we can visualize in a three-dimensional scatterplot with the response values placed vertically as usual and the predictors placed along the two horizontal axes. A multiple linear regression model with just these two predictors results in a fitted regression plane that looks like a flat piece of paper. If, however, we include an interaction between the predictors in our model, then the fitted regression plane looks like a rectangular piece of paper that has been warped (with edges that slope at different angles). This creates a three-dimensional plane that has been warped, as illustrated below for the equation \(y = 3x_1 + 5x_2 + 4x_1 x_2\).

3d plot
3d plot

Note that for fixed \(x_1\), \(y\) is a function of \(x_2\) only, and the slopes in the cross-section vary: for example, when \(x_1 = -4\), the slope is \(5 + 4(-4) = -11\), but when \(x_1 = 4\), the slope is \(5 + 4(4) = 21\). Similarly, for fixed \(x_2\), \(y\) is a function of \(x_1\) only, and the slopes in the cross-section vary: for example, when \(x_2 = -4\), the slope is \(3 + 4(-4) = -9\), but when \(x_2 = 4\), the slope is \(3 + 4(4) = 19\).

Typically, regression models that include interactions between quantitative predictors adhere to the hierarchy principle, which says that if your model includes an interaction term, \(X_1X_2\), and \(X_1X_2\) is shown to be a statistically significant predictor of Y, then your model should also include the "main effects," \(X_1\) and \(X_2\), whether or not the coefficients for these main effects are significant. Depending on the subject area, there may be circumstances where a main effect could be excluded, but this tends to be the exception.

We can use interaction terms in any multiple linear regression model. Here we consider an example with two quantitative predictors and one indicator variable for a categorical predictor. In Lesson 5 we looked at some data resulting from a study in which the researchers (Colby, et al , 1987) wanted to determine if nestling bank swallows alter the way they breathe in order to survive the poor air quality conditions of their underground burrows. In reality, the researchers studied not only the breathing behavior of nestling bank swallows, but that of adult bank swallows as well.

To refresh your memory, the researchers conducted the following randomized experiment on 120 nestling bank swallows. In an underground burrow, they varied the percentage of oxygen at four different levels (13%, 15%, 17%, and 19%) and the percentage of carbon dioxide at five different levels (0%, 3%, 4.5%, 6%, and 9%). Under each of the resulting 5×4 = 20 experimental conditions, the researchers observed the total volume of air breathed per minute for each of 6 nestling bank swallows. They replicated the same randomized experiment on 120 adult bank swallows. In this way, they obtained the following data (Swallows data) on n = 240 swallows:

  • Response (y): percentage increase in "minute ventilation", (Vent), i.e., total volume of air breathed per minute.
  • Potential predictor (\(x_{1} \colon \) percentage of oxygen (O2) in the air the swallows breathe
  • Potential predictor (\(x_{2} \colon \) percentage of carbon dioxide (CO2)in the air the baby birds breathe
  • Potential qualitative predictor (\(x_{3} \colon \) (Type) 1 if bird is an adult, 0 if bird is a nestling

Here's a plot of the resulting data for the adult swallows:

3d scatterplot of adult swallows

and a plot of the resulting data for the nestling bank swallows:

3d scatterplot of nestling swallows

As mentioned previously, the "best fitting" function through each of the above plots will be some sort of surface like a sheet of paper. If you drag the button to the right, you will see one possible estimate of the surface for the nestlings:

What we don't know is if the best fitting function — that is, the sheet of paper — through the data will be curved or not. Including interaction terms in the regression model allows the function to have some curvature, while leaving interaction terms out of the regression model forces the function to be flat.

Let's consider the research question "is there any evidence that the adults differ from the nestlings in terms of their minute ventilation as a function of oxygen and carbon dioxide?"

We could start by formulating the following multiple regression model with two quantitative predictors and one qualitative predictor:

\(y_i=(\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\beta_3x_{i3})+ \epsilon_i\)

where:

  • \(y_{i}\) is the percentage of minute ventilation for swallow i
  • \(x_{i1}\) is the percentage of oxygen for swallow i
  • \(x_{i2}\) is the percentage of carbon dioxide for swallow i
  • \(x_{i3}\) is the type of bird (0, if nestling and 1, if adult) for swallow i

and the independent error terms \(\epsilon_{i}\) follow a normal distribution with mean 0 and equal variance \(\sigma^{2}\).

We now know, however, that there is a risk in omitting an important interaction term. Therefore, let's instead formulate the following multiple regression model with three interaction terms:

\(y_i=(\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\beta_3x_{i3}+\beta_{12}x_{i1}x_{i2}+\beta_{13}x_{i1}x_{i3}+\beta_{23}x_{i2}x_{i3})+ \epsilon_i\)

where:

  • \(y_{i}\) is the percentage of minute ventilation for swallow i
  • \(x_{i1}\) is the percentage of oxygen for swallow i
  • \(x_{i2}\) is the percentage of carbon dioxide for swallow i
  • \(x_{i3}\) is the type of bird (0, if nestling and 1, if adult) for swallow i
  • \(x_{i1} x_{i2}, x_{i1} x_{i3}, \text{ and } x_{i2} x_{i3} \) are interaction terms

By setting the predictor \(x_{3}\) to equal 0 and 1 and doing a little bit of algebra we see that our formulated model yields two response functions — one for each type of bird:

Type of bird Formulated regression function
If a nestling, then \(x_{i3} = 0 \) and ...

\(\mu_Y=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\beta_{12}x_{i1}x_{i2}\)

If an adult, then \(x_{i3} = 1 \) and ...

\(\mu_Y=(\beta_0+\beta_3)+(\beta_1+\beta_{13})x_{i1}+(\beta_2+\beta_{23})x_{i2}+\beta_{12}x_{i1}x_{i2}\)

The \(\beta_12 x_{i1} x_{i2}\) interaction term appearing in both functions allows the two functions to have the same curvature. The additional \(\beta_{13}\) parameter appearing before the \(x_{i1}\) predictor in the regression function for the adults allows the adult function to be shifted from the nestling function in the \(x_{i1 }\) direction by \(\beta_{13}\) units. And, the additional \(\beta_{23}\) parameter appearing before the \(x_{i2}\) predictor in the regression function for the adults allows the adult function to be shifted from the nestling function in the \(x_{i2}\) direction by \(\beta_{23}\) units.

To add interaction terms to a model in Minitab click the "Model" tab in the Regression Dialog, then select the predictor terms you wish to create interaction terms for, then click "Add" for "Interaction through order 2." You should see the appropriate interaction terms added to the list of "Terms in the model." If we do this in Minitab to estimate the regression function above, we obtain:

Analysis of Variance
Source DF Seq SS Seq MS F-Value P-Value
Regression 6 2387540 397923 14.51 0.000
O2 1 93651 93651 3.42 0.066
CO2 1 2247696 2247696 81.9 0.000
Type 1 5910 5910 0.22 0.643
TypeO2 1 14735 14735 0.45 0.464
TypeCO2 1 2884 2884 0.11 0.746
CO2O2 1 22664 22664 0.83 0.364
Error 233 6388603 27419    
Lack-of-Fit 33 485700 14718 0.50 0.990
Pure Error 200 5902903 29515    
Total 239 8776143      
Model Summary
S R-sq R-sq(adj) R-sq(pred)
165.587 27.20% 25.33% 22.44%
Regression Equation

Vent = -18 + 1.19 O2 + 54.3 CO2 +112 Type - 7.01 TypeO2 + 2.31 TypeCO2 - 1.45 CO2O2

Note! that the P-values for each of the interaction parameters, \(\beta_{12}\), \(\beta_{13}\) , and \(\beta_{23}\) are quite large, suggesting there is little evidence for two-way interactions between type of bird, oxygen level, and carbon dioxide level.

Again, however, we should minimize the number of hypothesis tests we perform — and thereby reduce the chance of committing a Type I error on — our data, by instead conducting a partial F-test for testing \(H_0 \colon \beta_{12} = \beta_{13} = \beta_{23} = 0\) simultaneously:

Analysis of Variance
Source DF SS MS F P
Regression 6 2387540 397923 14.51 0.000
Residual Error 233 6388603 27419    
Total 239 8776143      
Source DF Seq SS
O2 1 93651
CO2 1 2247696
Type 1 5910
TypeO2 1 14735
TypeCO2 1 2884
CO2O2 1 22664

The Minitab output allows us to determine that the partial F-statistic is:

\(F^*=\dfrac{(14735+2884+22664)/3}{27419}=0.49\)

And the following Minitab output:

F distribution with 3 DF in numerator and 233 DF in denominator
x P(X ≤ x)
0.49 0.310445

Tells us that the probability of observing an F-statistic less than 0.49, with 3 numerator and 233 denominator degrees of freedom, is 0.31. Therefore, the probability of observing an F-statistic greater than 0.49, with 3 numerator and 233 denominator degrees of freedom, is 1-0.31 or 0.69. That is, the P-value is 0.69. There is insufficient evidence at the 0.05 level to conclude that at least one of the interaction parameters is not 0.

The residual versus fits plot:

residuals vs fitted values plot

also suggests that there is something not quite right about the fit of the model containing interaction terms.

Incidentally, if we go back and re-examine the two scatter plots of the data — one for the adults:

3d scatterplot of adult swallows

and one for the nestlings:

3d scatterplot of nestling swallows

we see that it is believable that there are no interaction terms. If you tried to "draw" the "best fitting" function through each scatter plot, the two functions would probably look like two parallel planes.

So, let's go back to formulating the model with no interactions terms:

\(y_i=(\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\beta_3x_{i3})+\epsilon_i\)

where:

  • \(y_{i}\) is the percentage of minute ventilation for swallow i
  • \(x_{i1}\) is the percentage of oxygen for swallow i
  • \(x_{i2}\) is the percentage of carbon dioxide for swallow i
  • \(x_{i3}\) is the type of bird (0, if nestling and 1, if adult) for swallow i

and the independent error terms \(\epsilon_{i}\) follow a normal distribution with mean 0 and equal variance \(\sigma^{2}\).

Using Minitab to estimate the regression function, we obtain:

Analysis of Variance
Source DF Seq SS Seq MS F-Value P-Value
Regression 3 2347257 782419 28.72 0.000
O2 1 93651 93651 3.44 0.065
CO2 1 2247696 2247696 82.51 0.000
Type 1 5910 5910 0.22 0.642
Error 236 6428886 27241    
Lack-of-Fit 36 525983 14611 0.50 0.993
Pure Error 200 5902903 29515    
Total 239 8776143      
Vent = 136.8 - 8.83 O2 + 32.26 CO2 + 9.9 Type

Let's finally answer our primary research question: "is there any evidence that the adult swallows differ from the nestling swallows in terms of their minute ventilation as a function of oxygen and carbon dioxide?" To answer the question, we need only test the null hypothesis \(H_0 \colon \beta_3 = 0\). And, Minitab reports that the P-value is 0.642. We fail to reject the null hypothesis at any reasonable significance level. There is insufficient evidence to conclude that adult swallows differ from nestling swallows with respect to their minute ventilation.

Incidentally, before using the model to answer the research question, we should have evaluated the model. All is fine, however. The residuals versus fits plot for the model with no interaction terms:

residuals vs fitted values plot

shows a marked improvement over the residuals versus fits plot for the model with the interaction terms. Perhaps there is a little bit of fanning? A little bit, but perhaps not enough to worry about.

And, the normal probability plot:

normal probability plot

suggests there is no reason to worry about non-normal error terms.