When a treatment (or factor) is a random effect, the model specifications as well as the relevant null and alternative hypotheses will have to be changed.

Recall the cell means model for the fixed effect case (from Lesson 4) which has the model equation

\(Y_{ij}=\mu_i+\epsilon_{ij}\)

where \(\mu_i\) are parameters for the treatment level means. For the single factor random effects model we have a very similar equation

\(Y_{ij}=\mu_i+\epsilon_{ij}\)

however, here \(\mu_i\) and \(\epsilon_{ij}\) are both independent random variables such that \(\mu_i \overset{iid}{\sim} \mathcal{N}\left(\mu, \sigma^2_{\mu}\right)\) and \(\epsilon_{ij} \overset{iid}{\sim} \mathcal{N}\left(0, \sigma_{\epsilon}^2\right)\) with \(i=1,2,...,T\), and \(j=1,2,...n_i\) where \(n_i \equiv n\) if the design is balanced.

The random effects ANOVA model is similar in appearance to the fixed effects ANOVA model. However, the treatment mean \(\mu_i\)'s are constant in the fixed-effect ANOVA model whereas in the random-effects ANOVA model the treatment mean \(\mu_i\)'s are random variables. Notice that the expected mean response in the random effects model stated above is the same at every treatment level and equals \(\mu\).

\(E\left(Y_{ij}\right) = E\left(\mu_i + \epsilon_{ij}\right)  =  E\left(\mu_i\right) + E\left(\epsilon_{ij}\right) = \mu \)

The variance of the response variable (say \(\sigma^2_{Y}\)) in the random effects model can be partitioned as

\(\sigma^2_{Y} = V\left(Y_{ij}\right) = V\left(\mu_i + \epsilon_{ij}\right) = V\left(\mu_i\right) + V\left(\epsilon_{ij}\right) = \sigma _{\mu}^{2}+\sigma_{\epsilon}^2\)

since \(\mu_i\) and \(\epsilon_{ij}\) are both independent random variables.

Similar to fixed effects ANOVA model, we can express the random effects ANOVA model using the factor effect representation, using \(\tau_i = \mu_i -\mu\). Therefore the factor effects representation of the random effects ANOVA model would be

\(Y_{ij} = \mu + \tau_i+\epsilon_{ij}\)

where \(\mu\) is a constant overall mean, \(\tau_i\) and \(\epsilon_{ij}\) are independent random variables such that \(\tau_i \overset{iid}{\sim} \mathcal{N}\left(0, \sigma^2_{\mu}\right)\) and \(\epsilon_{ij} \overset{iid}{\sim} \mathcal{N}\left(0, \sigma_{\epsilon}^2\right)\). Again, \(i=1,2,...,T\), and \(j=1,2,...n_i\) where \(n_i \equiv n\) if balanced. Here, \(\tau_i\) is the effect of the randomly selected \(i^{th}\) level.

The terms \(\sigma _{\tau}^{2}\) and \(\sigma _{\epsilon}^{2}\) are referred to as variance components. In general, as will be seen later in more complex models, there will be a variance component associated with each effect involving at least one random factor.

Variance components play an important role in analyzing random effects data. They can be used to verify the significant contribution of each random effect to the variability of the response. For the single factor random-effects model stated above, the appropriate null and alternative hypothesis for this purpose is:

\(H_0 \colon \sigma_{\mu}^{2}= 0 \text{ vs. } H_A \colon \sigma_{\mu}^{2}> 0\)

Similar to the fixed effects model, an ANOVA analysis can then be carried out to determine if \(H_0\) can be rejected.

The MS and df computations of the ANOVA table are the same for both the fixed and random-effects models. However, the computations of the F-statistics needed for hypothesis testing require some modification. More specifically, the F-statistics denominator will no longer always be the mean squared error (MSE) and will vary according to the effect of interest (listed in the Source column of the ANOVA table). For a random-effects model, the quantities known as Expected Means Squares (EMS) shown in the ANOVA table below can be used to identify the appropriate F-statistic denominator for a given source in the ANOVA table. These EMS quantities will also be useful in estimating the variance components associated with a given random effect. Note that the EMS quantities are in fact the population counterparts of the mean sums of squares (MS) estimates that we are already familiar with. In SAS the proc mixed, the method=type3 option will generate the EMS column in the ANOVA table output.

More about variance components...

Often the variance component of a specific effect in the model is expressed as a percent of the total variation in the response variable. Another common application of variance components is the calculation of the relative size of the treatment effect compared to the within-treatment level variation. This leads to a quantity called the intraclass correlation coefficient (ICC), defined as:

\(ICC=\dfrac{\sigma_{\text{among trts}}^{2}}{\sigma_{\text{among trts}}^{2}+\sigma_{\text{within trts}}^{2}}\)

For single random factor studies, \(ICC = \frac{\sigma^2_{\mu}}{\sigma^2_{\mu} + \sigma_{\epsilon}^2}\). ICC can also be thought of as the correlation between the observations within the group (i.e. \(\text{corr}\left(Y_{ij}, Y_{ij'}\right)\), where \(j \neq j'\)). Small values of ICC indicate a large spread of values at each level of the treatment, whereas large values of ICC indicate relatively little spread at each level of the treatment:

Random effects can appear in both factorial and nested designs. By inspecting the EMS quantities, we can determine the appropriate F-statistic denominator for a given source. Let us look at two-factor studies.

Factorial Design

Recall the Greenhouse example in Section 5.1. In this example, there were two crossed factors (fert and species). We treated both factors to be fixed and the SAS  proc mixed ANOVA table was as follows:

If we inspect the EMS quantities in the output, we see when both factors are fixed in the 2-factor crossed study that the correct denominator for all F-tests is Error Mean Squares. 

Now let us consider a case in which both factors A and B are random effects in the factorial design (i.e. factors A and B crossed, and both are random effects). The expected mean squares for each of the source of variations in the ANOVA model would be as follows:

The F-tests following from the EMS above would be:

Here we can see the ramifications of having random effects. In fixed-effects models, the denominator for the F-statistics in significance testing was the mean square error (MSE). In random-effects models, however, we may have to choose different denominators depending on the term we are testing.

In general, the F-statistic for testing the significance of a given effect is the ratio of two MS values, with MS of the effect as the numerator and the denominator MS is chosen such that the F-statistic equals 1 if \(H_0\) is true and is greater than 1 if \(H_a\) is true.

Following this logic, we can see that when testing for the interaction effect of 2 random factors, the correct denominator is the error mean squares. Therefore the test statistic for testing \(A \times B\) is \(\frac{MSAB}{MSE}\). However, when we are testing for the main effect of factor A, the correct denominator would be \(MSAB\). 

Recall that the EMS quantities are the population counterparts for the MS estimates which actually are sample statistics. Examination of EMS expressions can therefore be used to choose the correct denominator for the F-statistic utilized for testing significance and will be discussed in detail in Section 6.7.

Nested Design

In the case of a nested design, where factor B is nested within the levels of factor A and both are random effects, the expected mean squares for each of the source of variations in the ANOVA model would be as follows:

The F-tests follow from the EMS above:

Special Case: Fully Nested Random Effects Design

Here, we consider a special case of random effects models where each factor is nested within the levels of the next ‘order’ of a hierarchy. This Fully Nested Random Effects model is similar to Russian Babushka dolls where the smaller dolls are nested within the next larger one.

Consider 3 random factors A, B, and C that are hierarchically nested. That is C is nested in (B, A) combinations and B is nested within levels of A. Suppose there are n observations made at the lowest level.

The statistical model for this case is:

\(Y_{ijkl}=\mu+\alpha_i+\beta_{j(i)}+\gamma_{k(ij)}+\epsilon_{ijkl}\)

where \(i = 1, 2, \dots, a\), \(j = 1, 2, \dots, b\), \(k = 1, 2, \dots, c\) and \(l = 1, 2, \dots, n\).

We will also have \(\epsilon_{ijkl} \overset{iid}{\sim} \mathcal{N}\left(0, \sigma^2\right)\), \(\gamma_{k(ij)} \overset{iid}{\sim} \mathcal{N}\left(0, \sigma^2_{\gamma}\right)\), \(\beta_{i(j)} \overset{iid}{\sim} \mathcal{N}\left(0, \sigma^2_{\beta}\right)\) and \(\alpha_{i} \overset{iid}{\sim} \mathcal{N}\left(0, \sigma^2_{\alpha}\right)\). 

The dfs and expected mean squares for this design would be as follows:

In this case, each F-test we construct for the sources will be based on different denominators.

Treatment designs can be comprised of both fixed and random effects. When we have this situation, the treatment design is referred to as a mixed model. Mixed models are by far the most commonly encountered treatment designs. The three possible situations we now have are often referred to as Model I (fixed effects only), Model II (random effects only), and Model III (mixed) ANOVAs. When designating the effects of a mixed model as fixed or random, the following rule will be useful.

Below are the ANOVA layouts of two basic mixed models with 2-factors.

Factorial

In the simplest case of a balanced mixed model in a factorial design, we may have two factors, A and B, in which factor A is a fixed effect and factor B is a random effect. The statistical model is similar to what we have seen before:

\(y_{ijk} = \mu + \alpha_i + \beta_j + (\alpha\beta)_{ij} + \epsilon_{ijk}\) where \(i=1,2,\dots, a\), \(j=1,2,\dots, b\) and \(k=1,2, \dots, n\).

Here, \(\sum_{i} \alpha_i = 0, \) \(\beta_j \sim \mathcal{N}\left(0, \sigma^2_{\beta}\right), \) \((\alpha\beta)_{i,j} \sim N(0,\sigma^2_{\alpha\beta}) \) and \(\epsilon_{ijk} \sim \mathcal{N}\left(0, \sigma^2\right)\). Also, \(\beta_j, (\alpha\beta)_{ij}\) and \(\epsilon_{ij}\) are pairwise independent. 

In this case, we have the following ANOVA:

The F-tests are set up based on the EMS column above and we can see that we have to use different denominators in testing significance for the various sources in the ANOVA table:

As a reminder, the null hypothesis for a fixed effect is that the \(\alpha_i\)'s are equal, whereas the null hypothesis for the random effect is that the \(\sigma_{\beta}^{2}\) is equal to zero.

Nested

In the case of a balanced nested treatment design, where A is a fixed effect and B(A) is a random effect, the statistical model would be:

\(y_{ijk} = \mu + \alpha_i + \beta_{j(i)} + \epsilon_{ijk}\) where \(i=1,2,\dots, a\), \(j=1,2,\dots, b\) and \(k=1,2, \dots, n\).

Here, \(\sum_{i} \alpha_i = 0\), \(\beta_{j(i)} \sim \mathcal{N}\left(0, \sigma^2_{\beta(\alpha)}\right)\) and \(\epsilon_{ijk} \sim \mathcal{N}\left(0, \sigma^2\right)\). 

We have the following ANOVA for this model:

Here is the same table with the F-statistics added. Note that the denominators for the F-test are different.

As can be seen from the examples above (and in the previous sections), when significance testing in random or mixed models the denominator of the F-statistic is no longer always the MSE value and has to be aptly chosen. Recall that the F-statistic for testing the significance of a given effect is the ratio with the numerator equal to the MS value of the effect, and the denominator is also an MS value of an effect included in the ANOVA model. Furthermore, it can be said the F-statistic has a non-central distribution when \(H_a\) is true and a central F-distribution when \(H_0\) is true.

When \(H_a\) is true, the non-centrality parameter of the non-central F-distribution depends on the type of effect (fixed vs random) and equals \(\sum_{i=1}^T {\alpha_{i}^2}\) for a fixed effect and \(\sigma_{trt}^2\) for a random effect. Here \(\alpha_i=\mu_i-\mu\), where \(\mu_i\) is the \(i^{th}\) level of the fixed effect for \((i=1,2,...,T)\), \(\mu\) is the overall mean, and \(\sigma_{trt}^2\) is the variance component associated with the random effect. Also, the MS under true \(H_a\) equals to MS under true \(H_0\) plus non-centrality parameter, so that

F-statistic = \(\dfrac{\text{MS when \(H_0\) is true + non-centrality parameter}}{\text{MS when \(H_0\) is true}}\).

The above identity can be used to identify the correct denominator (also called the error term) with the aid of EMS expressions displayed in the ANOVA table. This can be summarized by the following rule.

From what we have discussed so far we see that even for the simplest multi-factor studies (i.e. those involving only two factors) there are many possibilities of treatment designs; each factor is either fixed or random as well as either crossed or nested. 

For any of these possibilities, we can carry out the hypothesis tests using the EMS expressions to identify the correct denominator for the relevant F-statistics. The possible EMS expressions are summarized in the following tables.

Random effects of an ANOVA model represent measurements arising from a larger population. In other words, the levels or groups of the random effect that are observed can be considered as a sample from an original population. Random effects can also be subject effects. Consequently, in public health, a random effect is sometimes referred to as the subject-specific effect.

As all the levels of a random effect have the same mean, its significance is measured in terms of the variance with \(H_0\colon \sigma_{\tau}^2=0 \ \text{vs.}\ H_a\colon\sigma_{\tau}^2\gt0\). Note also that any interaction effect involving at least one random effect is also a random effect. Due to the added variability incurred by each random effect, the variance of the response now will have several components which are called variance components. In the most basic case only one single factor and no fixed effects, this compound variance of the response will be  \(\sigma_{Y}^2=\sigma_{\tau}^2+\sigma_{\epsilon}^2\), where \(\sigma_{\tau}^2\) is the variance component associated with the random factor. The intra-class correlation (ICC), defined in terms of the variance components, is a useful indicator of the high or low variability within groups (or subjects).

Mixed models introduced in section 6.5 include both fixed and random effects. Throughout the lesson, we learned how EMS quantities can be used to determine the correct F-test to test the hypotheses associated with the effects. EMS quantities can be thought of as the population counterparts of the mean sums of squares (MS) which are computable for each source in the ANOVA table.