2.5 - Analysis of Variance: The Basic Idea

Break down the total variation in y ("total sum of squares (SSTO)") into two components:

  • a component that is "due to" the change in x ("regression sum of squares (SSR)")
  • a component that is just due to random error ("error sum of squares (SSE)")

If the regression sum of squares is a "large" component of the total sum of squares, it suggests that there is a linear association between the predictor x and the response y.

Here is a simple picture illustrating how the distance \(y_i-\bar{y}\) is decomposed into the sum of two distances, \(\hat{y}_i-\bar{y}\) and \(y_i-\hat{y}_i\). Drag the bar at the bottom of the image to see each of the three components of the equation represented geometrically.

Although the derivation isn't as simple as it seems, the decomposition holds for the sum of the squared distances, too:

\(\underbrace{\left(\sum\limits_{i=1}^{n}(y_i-\bar{y})^2\right)}_{\underset{\text{Total Sum of Squares}}{\text{SSTO}}} = \underbrace{\sum\limits_{i=1} ^{n} \left( \hat{y}_{i} - \overline{y} \right)^{2}}_{\underset{\text{Regression of Sums}}{\text{SSR}}} + \underbrace{\sum\limits_{i=1} ^{n} \left( y_{i} - \hat{y} \right)^{2}}_{\underset{\text{Error Sum of Squares}}{\text{SSE}}}\)

\(\text{SSTO} = \text{SSR} + \text{SSE}\)

The degrees of freedom associated with each of these sums of squares follow a similar decomposition.

  • You might recognize SSTO as being the numerator of the sample variance. Recall that the denominator of the sample variance is n-1. Therefore, n-1 is the degrees of freedom associated with SSTO.
  • Recall that the mean square error MSE is obtained by dividing SSE by n-2. Therefore, n-2 is the degrees of freedom associated with SSE.

Then, we obtain the following breakdown of the degrees of freedom:

\(\underset{\substack{\text{degrees of freedom}\\ \text{associated with SSTO}}}{\left(n-1\right)} = \underset{\substack{\text{degrees of freedom}\\ \text{associated with SSR}}}{\left(1\right)} + \underset{\substack{\text{degrees of freedom}\\ \text{associated with SSE}}}{\left(n-2\right)}\)