An Extended Overview of ANCOVA

Designed experiments often contain treatment levels that have been set with increasing numerical values. For example, consider a chemical process which is hypothesized to vary by two factors: the Reagent type (A or B), and temperature. Researchers conducted an experiment that investigates a response at 40, 50, 60, 70, and 80 degrees (F) for each of the Reagent types.

You can find the data at QuantFactorData.csv

If temperature is considered as a categorical factor, we can proceed as usual with a 2 × 5 factorial ANOVA to evaluate the null hypotheses

\(H_0 \colon \mu_A = \mu_B\)

\(H_0 \colon \mu_{40} = \mu_{50} = \mu_{60} = \mu_{70} = \mu_{80}\)

and

\(H_0\colon\) no interaction.

Although the above hypotheses compare the response means for the process carried out at different temperatures, no conclusion can be made about the trend of the response as the temperature is increased. 

In general, the trend effects of a continuous predictor are modeled using a polynomial where its non-constant terms represent the different trends, such as linear, quadratic, and cubic effects. These non-constant terms in the polynomial are called trend terms.

The statistical significance of these trend terms can be tested in an ANCOVA by adding columns into the design matrix (X) of the General Linear Model (see Lesson 4 for the definition of a design matrix). These additional columns will represent the trend terms and their interaction effects with the categorical factor. Inclusion of the trend term columns will facilitate significance testing for the overall trend effects, whereas the columns representing the interactions can be utilized to compare differences of each trend effect among the categorical factor levels. 

In the chemical process example, if the quantitative property of temperature is used, we can carry out an ANCOVA by fitting a polynomial regression model to express the impact of temperature on the response. If a quadratic polynomial is desired, the appropriate ANCOVA design matrix can be obtained by adding two columns representing \(temp\) and \(temp^2\) to the existing column of ones representing the reagent type A (the reference category) and the dummy variable column representing the reagent type B.

The \(temp\) and \(temp^2\) terms allow us to investigate the linear and quadratic trends, respectively. Furthermore, the inclusion of columns representing the interactions between the reagent type and the two trend terms will facilitate the testing of differences between these two trends between the two reagent types. Note also that additional columns can be added appropriately to fit a polynomial of an even higher order.

Polynomial trends in the response with respect to a quantitative predictor can be evaluated by using orthogonal polynomial contrasts, a special set of linear contrasts. This is an alternative to the regression analysis illustrated in the previous section, which may be affected by multicollinearity. Note that centering to remedy multicollinearity is effective only for quadratic polynomials. Therefore this simple technique of trend analysis performed via orthogonal polynomial coding will prove to be beneficial for higher-order polynomials. Orthogonal polynomials have the property that the cross-products defined by the numerical coefficients of their terms add to zero.

The orthogonal polynomial coding can be applied only when the levels of quantitative predictor are equally spaced. The method partitions the quantitative factor in the ANOVA table into independent single degrees of freedom comparisons. The comparisons are called orthogonal polynomial contrasts or comparisons.

Orthogonal polynomials are equations such that each is associated with a power of the independent variable (e.g. \(x\), linear; \(x^2\), quadratic; \(x^3\), cubic, etc.) In other words, orthogonal polynomials are coded forms of simple polynomials. The number of possible comparisons is equal to \(k-1\), where \(k\) is the number of quantitative factor levels. For example, if \(k=3\), only two comparisons are possible allowing for testing of linear and quadratic effects.

Using orthogonal polynomials to fit the desired model to the data allows us to eliminate collinearity while obtaining the same information as with simple polynomials. 

A typical polynomial model of order \(k-1\) would be:

\(y = \beta_0 + \beta_1 x + \beta_2 x^2 + \cdots + \beta_{(k-1)} x^{(k-1)} + \epsilon\)

The simple polynomials used are \(x, x^2, \dots, x^{(k-1)}\). We can obtain orthogonal polynomials as linear combinations of these simple polynomials. If the levels of the predictor variable, \(x\) are equally spaced then one can easily use coefficient tables to determine the orthogonal polynomial coefficients that can be used to set up an orthogonal polynomial model. 

If we are to fit the \((k-1)^{th}\) order polynomial using orthogonal contrasts coefficients, the general equation can be written as 

\(y_{ij} = \alpha_0 + \alpha_1 g_{1i}(x) + \alpha_2 g_{2i}(x) + \cdots + \alpha_{(k-1)} g_{(k-1)i}(x)+\epsilon_{ij}\)

where \(g_{pi}(x)\) is a polynomial in \(x\) of degree \(p\) for the \(i^{th}\) level treatment factor. The parameter \(\alpha_p\) will depend on the coefficients \(\beta_p\). Using the properties of the function \(g_{pi}(x)\), one can show that the first five orthogonal polynomial are of the following form:

Mean: \(g_0(x)=1\)

Linear: \(g_1(x)=\lambda_1 \left( \dfrac{x-\bar{x}}{d}\right)\)

Quadratic: \(g_2(x)=\lambda_2 \left( \left(\dfrac{x-\bar{x}}{d}\right)^2-\left( \dfrac{t^2-1}{12} \right)\right)\)

Cubic: \(g_3(x)=\lambda_3 \left( \left(\dfrac{x-\bar{x}}{d}\right)^3- \left(\dfrac{x-\bar{x}}{d}\right)\left( \dfrac{3t^2-7}{20} \right)\right)\)

Quartic: \(g_4(x) = \lambda_4 \left( \left( \dfrac{x - \bar{x}}{d} \right)^4 - \left( \dfrac{x - \bar{x}}{d} \right)^2 \left( \dfrac{3t^2 - 13}{14} \right)+ \dfrac{3 \left(t^2 - 1\right) \left(t^2 - 9 \right)}{560} \right)\)

where \(t\) = number of levels of the factor, \(x\) = value of the factor level, \(\bar{x}\) = mean of the factor levels, and \(d\) = distance between factor levels.

In the next section, we will illustrate how the orthogonal polynomial contrast coefficients are generated, and the Factor SS is partitioned. This method will be required to fit polynomial regression models with terms greater than the quadratic because even after centering there will still be multicollinearity between \(x\) and \(x^3\) as well as between \(x^2\) and \(x^4\).

The versatility of the ANCOVA model has been illustrated in Lessons 9 and 10. In application, it is often used in ANOVA settings to adjust or 'control for' a covariate that may be masking real treatment differences. In regression settings, researchers may be focused on a family of regression relationships, and are interested in testing for significant differences among regression coefficients across different groups. These approaches are like two sides of the same coin.

In terms of model development, ANOVA and regression approaches converge in the general linear model as ANCOVA. Mastery of ANCOVA methodology is arguably one of the most important tools to have in an applied statistician's toolbox.