As noted previously \(\bar{\textbf{x}}\) is a function of random data, and hence \(\bar{\textbf{x}}\) is also a random vector with a mean, a variance-covariance matrix, and a distribution. We have already seen that the mean of the sample mean vector is equal to the population mean vector \(\boldsymbol{\mu}\).

Before considering the sample variance-covariance matrix for the mean vector \(\bar{\textbf{x}}\), let us revisit the univariate setting.

Univariate Setting

You should recall from introductory statistics that the population variance of the sample mean, generated from independent samples of size n, is equal to the population variance, \(\sigma^{2}\) divided by n.

\(\text{var}(\bar{x}) = \dfrac{\sigma^2}{n}\)

This, of course, is a function of the unknown population variance \(\sigma^{2}\). We can estimate this by simply substituting \(s^{2}\) in the sample variance \(\sigma^{2}\) yielding our estimate for the variance of the population mean as shown below:

\(\widehat{\text{var}}(\bar{x}) = \dfrac{s^2}{n}\)

If we were to take the square root of this quantity we would obtain the standard error of the mean. The standard error of the mean is a measure of the uncertainty of our estimate of the population mean. If the standard error is large, then we are less confident of our estimate of the mean. Conversely, if the standard error is small, then we are more confident in our estimate. What is meant by large or small depends on the application at hand. But in any case, because the standard error is a decreasing function of sample size, the larger our sample the more confident we can be of our estimate of the population mean.

Multivariate Setting

The population variance-covariance matrix replaces the variance of the \(\bar{x}\)’s generated from independent samples of size n, taking a form similar to the univariate setting as shown below:

\(\text{var}(\bar{\textbf{x}}) = \dfrac{1}{n}\Sigma\)

Again, this is a function of the unknown population variance-covariance matrix \(\Sigma\). An estimate of the variance-covariance matrix of \(\bar{\textbf{x}}\) can be obtained by substituting the sample variance-covariance matrix S for the population variance-covariance matrix \(\Sigma\), yielding the estimate as shown below:

\(\widehat{\text{var}}(\bar{\textbf{x}}) = \dfrac{1}{n}\textbf{S}\)

Let's consider the distribution of the sample mean vector, first looking at the univariate setting and comparing this to the multivariate setting.

Univariate Setting

Here we are going to make the additional assumption that \(X _ { 1 } , X _ { 2 , \dots } X _ { n } \) are independently sampled from a normal distribution with mean \(\mu\) and variance \(\sigma_{2}\). In this case, \(\bar{x}\) is normally distributed as

\(\bar{x} \sim N\left(\mu, \frac{\sigma^2}{n}\right)\)

Multivariate Setting

Similarly, for the multivariate setting, we are going to assume that the data vectors \(\boldsymbol{X _ { 1 }, X _ { 2, \dots } X _ { n }} \) are independently sampled from a multivariate normal distribution with mean vector \(\boldsymbol{\mu}\) and variance-covariance matrix \(\Sigma\). Then, in this case, the sample mean vector, \(\bar{\textbf{x}}\), is distributed as multivariate normal with mean vector \(\boldsymbol{\mu}\) and variance-covariance matrix \(\frac{1}{n}\Sigma\), the variance-covariance matrix for \(\bar{\textbf{x}}\). In statistical notation we write:

\(\bar{\textbf{x}} \sim N \left( \boldsymbol {\mu}, \frac{1}{n}\Sigma \right)\)

At this point, we will drop the assumption that the individual observations are sampled from a normal distribution and look at the laws of large numbers. These will hold regardless of the distribution of the individual observations.

Univariate Setting

In the univariate setting, we see that if the data are independently sampled, then the sample mean, \(\bar{x}\), is going to converge (in probability) to the population mean \(\mu\). What does this mean exactly? It means that as the sample size gets larger and larger the sample mean will tend to approach the true value for a population \(\mu\).

Multivariate Setting

A similar result is involved in the multivariate setting, the sample mean vector, \(\bar{\textbf{x}}\), will also converge (in probability) to the mean vector \(\boldsymbol{\mu}\) As our sample size gets larger and larger, each of the individual components of that vector, \(\bar{x}_{j}\), will converge to the corresponding mean, \(\mu_{j}\).

\(\bar{x}_j \stackrel{p}\rightarrow \mu_j\)

Just as in the univariate setting we also have a multivariate Central Limit Theorem. But first, let's review the univariate Central Limit Theorem.

Univariate Setting

If all of our individual observations, \(X _ { 1 } , X _ { 2 , \dots } X _ { n }\), are independently sampled from a population with mean \(\mu\) and variance \(\sigma_{2}\), then, the sample mean, \(\bar{x}\), is approximately normally distributed with mean \(\mu\) and variance \(\sigma_{2}\).

Multivariate Setting

A similar result is available in the multivariate setting. If our data vectors \(\boldsymbol{X _ { 1 }, X _ { 2 , \dots } X _ { n }}\) , are independently sampled from a population with mean vector \(\boldsymbol{\mu}\) and variance-covariance matrix \(\Sigma\), then the sample mean vector, \(\bar{\textbf{x}}\), is going to be approximately normally distributed with mean vector \(\boldsymbol{\mu}\) and variance-covariance matrix \(\frac{1}{n}\Sigma\).

This Central Limit Theorem is a key result that we will take advantage of later on in this course when we talk about hypothesis tests for individual mean vectors or collections of mean vectors under different treatment regimens.

Here we consider the joint estimation of a multivariate set of population means. That is, we have observed a set of p X-variables and may wish to estimate the population mean for each variable. In some instances, we may also want to estimate one or more linear combinations of population means. Our basic tool for estimating the unknown value of a population parameter is a confidence interval, an interval of values that is likely to include the unknown value of the parameter.

In this formula, \(\bar{x}_{j}\) is the sample mean, \(s_{j}\) is the sample standard deviation and n is the sample size. The multiplier value is a function of the confidence level, the sample size, and the strategy used for dealing with the multiple inference issue.

The following list covers some common strategies:

1. One-at-a-Time Confidence Intervals: This strategy essentially considers each mean separately and uses the desired confidence level (usually 95%) for every single interval.
2. Bonferroni Method: With this method, we set a family-wide error rate and then divide this family error rate by the number of intervals to be computed to determine the error rate (and hence confidence level) for each individual interval.
3. Simultaneous Confidence Region: This strategy uses properties of the multivariate normal distribution to define joint confidence intervals. The multiplier for this method is conservative because the family error rate applies to the family of all possible linear combinations of population means.

For a \(1 - \alpha\) confidence interval, the “one at a time” multiplier is the t-value such that the probability is \(1 - \alpha\) between –t and +t under a t-distribution with n - 1 degrees of freedom. Said another way, the value of t is such that the probability greater than +t is \(\alpha/2\).

Notationally, the one-at-a-time multiplier is:

\(\text{Multiplier} = t_{n-1}(\alpha/2)\)

With this notation, a confidence interval for \(\mu_{j}\) is computed as:

\(\bar{x}_j \pm t_{n-1}(\alpha/2)\frac{s_j}{\sqrt{n}}\)

When we determine confidence intervals for the population means of several variables, we are creating a family of confidence intervals. The family-wide error rate is the probability that at least one of the confidence intervals in the family will not capture the population mean. The family-wide confidence level = 1 – family-wide error rate.

Suppose that we have a family of p confidence intervals and the error rates for the individual intervals are \(\alpha _ { 1 }, \alpha _ { 2 }, \dots , \alpha _ { p }\). The Bonferroni Inequality states that the family wide-error rate is less than or equal to the sum of \(\alpha _ { 1 }, \alpha _ { 2 }, \dots , \alpha _ { p }\). That is family-wide error rate \(\leq \Sigma \alpha _ { i }\). In terms of the family-wide confidence that all intervals capture their population means, we can write this as \(1 - \Sigma \alpha _ { i } \leq\) family-wide confidence level.

Most often, we divide the desired family-wide error rate equally across the intervals that we will compute. If we are computing p confidence intervals with a desired family-wide confidence level of \(\alpha\), we use an error rate of \(\alpha / p\) (so confidence \(= 1 - (\alpha / p)\) for each individual interval. This guarantees that the family-wide confidence level will be greater than or equal to \(1 - \alpha\).

Suppose that we are calculating p intervals with a family error rate equal to \(\alpha\).

Notationally, the Bonferroni method multiplier is:

\(\text{Multiplier} = t_{n-1}(\alpha/2p)\)

A confidence interval for\(\mu_{j}\) is computed as:

\(\bar{x}_j \pm t_{n-1}(\alpha/2p)\frac{s_j}{\sqrt{n}}\)

This method is derived from the properties of the multivariate normal distribution. The multiplier applies to the family of all possible linear combinations of the population means considered, including the individual means. It is conservative (meaning that the multiplier tends to be larger than absolutely necessary). When family confidence is used, compare the value of this multiplier to the Bonferroni method multiplier and use the smaller of the two.

Notationally, the simultaneous confidence region multiplier is:

\(\text{Multiplier}=\sqrt{\frac{p(n-1)}{n-p}F_{p,n-p}(\alpha)}\)

\(F _ { p , n - p } ( \alpha )\) represents a value of F such that the probability greater than this value is α under an F-distribution with p and n - p degrees of freedom.

The following table summarizes the three different multipliers and gives notes about using Excel and SAS.

Let us consider testing the null hypothesis that there is zero correlation between two variables \(X_{j}\) and \(X_{k}\). Mathematically we write this as shown below:

\(H_0\colon \rho_{jk}=0\) against \(H_a\colon \rho_{jk} \ne 0 \)

Recall that the correlation is estimated by sample correlation \(r_{jk}\) given in the expression below:

\(r_{jk} = \dfrac{s_{jk}}{\sqrt{s^2_js^2_k}}\)

Here we have the sample covariance between the two variables divided by the square root of the product of the individual variances.

We shall assume that the pair of variables \(X_{j}\)and \(X_{k}\) are independently sampled from a bivariate normal distribution throughout this discussion; that is:

\(\left(\begin{array}{c}X_{1j}\\X_{1k} \end{array}\right)\), \(\left(\begin{array}{c}X_{2j}\\X_{2k} \end{array}\right)\), \(\dots\), \(\left(\begin{array}{c}X_{nj}\\X_{nk} \end{array}\right)\)

are independently sampled from a bivariate normal distribution.

To test the null hypothesis, we form the test statistic, t as below

\(t = r_{jk}\sqrt{\frac{n-2}{1-r^2_{jk}}}\)  \(\dot{\sim}\)  \( t_{n-2}\)

Under the null hypothesis, \(H_{o}\), this test statistic will be approximately distributed as t with n - 2 degrees of freedom.

To illustrate these concepts let's return to our example dataset, the Wechsler Adult Intelligence Scale.

Once we conclude that there is a positive or negative correlation between two variables the next thing we might want to do is compute a confidence interval for the correlation. This confidence interval will give us a range of reasonable values for the correlation itself. The sample correlation, because it is bounded between -1 and 1 is typically not normally distributed or even approximately so. If the population correlation is near zero, the distribution of sample correlations may be approximately bell-shaped in distribution around zero. However, if the population correlation is near +1 or -1, the distribution of sample correlations will be skewed. For example, if \(p_{jk}= .9\), the distribution of sample correlations will be more concentrated near .9.  Because they cannot exceed 1, they have more room to spread out to the left of .9, which causes a left-skewed shape. To adjust for this asymmetry or the skewness of distribution, we apply a transformation of the correlation coefficients. In particular, we are going to apply Fisher's transformation which is given in the expression below in Step 1 of our procedure for computing confidence intervals for the correlation coefficient.

Steps

1. Compute Fisher’s transformation

   \(z_{jk}=\frac{1}{2}\log\dfrac{1+r_{jk}}{1-r_{jk}}\)

   Here we have one-half of the natural log of 1 plus the correlation, divided by one minus the correlation.

   Note! In this course, whenever log is mentioned, unless specified otherwise, log stands for the natural log.

   For large samples, this transform correlation coefficient z is going to be approximately normally distributed with the mean equal to the same transformation of the population correlation, as shown below, and a variance of 1 over the sample size minus 3.

   \(z_{jk}\) \(\dot{\sim}\) \(N\left(\dfrac{1}{2}\log\dfrac{1+\rho_{jk}}{1-\rho_{jk}}, \dfrac{1}{n-3}\right)\)

2. Compute a (1 - \(\alpha\)) x 100% confidence interval for the Fisher transform of the population correlation.

   \(\dfrac{1}{2}\log \dfrac{1+\rho_{jk}}{1-\rho_{jk}}\)

   That is one-half log of 1 plus the correlation divided by 1 minus the correlation. In other words, this confidence interval is given by the expression below:

   \(\left(\underset{Z_l}{\underbrace{Z_{jk}-\frac{Z_{\alpha/2}}{\sqrt{n-3}}}}, \underset{Z_U}{\underbrace{Z_{jk}+\frac{Z_{\alpha/2}}{\sqrt{n-3}}}}\right)\)

   Here we take the value of Fisher's transform Z, plus and minus the critical value from the z table, divided by the square root of n - 3. The lower bound we will call the \(Z_{1}\) and the upper bound we will call the \(Z_{u}\).

3. Back transform the confidence values to obtain the desired confidence interval for \(\rho_{jk}\) This is given in the expression below:

   \(\left(\dfrac{e^{2Z_l}-1}{e^{2Z_l}+1},\dfrac{e^{2Z_U}-1}{e^{2Z_U}+1}\right)\)

   The first term we see is a function of the lower bound, the \(Z_{1}\). The second term is a function of the upper bound or \(Z_{u}\).

Let's return to the Wechsler Adult Intelligence Data to see how these procedures are carried out.

In this lesson we learned about:

• The statistical properties of the sample mean vector, including its variance-covariance matrix and its distribution;
• The multivariate central limit theorem; which states that the sample mean vector will be approximately normally distributed;
• Construction of confidence intervals one at a time and simultaneously;
• How to test the hypothesis that the population correlation between two variables is equal to zero, and how to draw conclusions regarding the results of that hypothesis test;
• How to compute confidence intervals for the correlation, and what conclusions can be drawn from such intervals.