Suppose that we have a random vector \(\mathbf{X}\).

\(\textbf{X} = \left(\begin{array}{c} X_1\\ X_2\\ \vdots \\X_p\end{array}\right)\)

with population variance-covariance matrix

\(\text{var}(\textbf{X}) = \Sigma = \left(\begin{array}{cccc}\sigma^2_1 & \sigma_{12} & \dots &\sigma_{1p}\\ \sigma_{21} & \sigma^2_2 & \dots &\sigma_{2p}\\  \vdots & \vdots & \ddots & \vdots \\ \sigma_{p1} & \sigma_{p2} & \dots & \sigma^2_p\end{array}\right)\)

Consider the linear combinations

\(\begin{array}{lll} Y_1 & = & e_{11}X_1 + e_{12}X_2 + \dots + e_{1p}X_p \\ Y_2 & = & e_{21}X_1 + e_{22}X_2 + \dots + e_{2p}X_p \\ & & \vdots \\ Y_p & = & e_{p1}X_1 + e_{p2}X_2 + \dots +e_{pp}X_p\end{array}\)

Each of these can be thought of as linear regression, predicting \(Y_{i}\) from \(X_{1}\), \(X_{2}\), ... , \(X_{p}\). There is no intercept, but \(e_{i1}\), \(e_{i2}\), ..., \(e_{ip}\) can be viewed as regression coefficients.

Note that \(Y_{i}\) is a function of our random data, and so is also random. Therefore it has a population variance

\(\text{var}(Y_i) = \sum\limits_{k=1}^{p}\sum\limits_{l=1}^{p}e_{ik}e_{il}\sigma_{kl} = \mathbf{e}'_i\Sigma\mathbf{e}_i\)

Moreover, \(Y_{i}\) and \(Y_{j}\) have population covariance

\(\text{cov}(Y_i, Y_j) = \sum\limits_{k=1}^{p}\sum\limits_{l=1}^{p}e_{ik}e_{jl}\sigma_{kl} = \mathbf{e}'_i\Sigma\mathbf{e}_j\)

Collect the coefficients \(e_{ij}\) into the vector

\(\mathbf{e}_i = \left(\begin{array}{c} e_{i1}\\ e_{i2}\\ \vdots \\ e_{ip}\end{array}\right)\)

The first principal component is the linear combination of x-variables that has maximum variance (among all linear combinations).  It accounts for as much variation in the data as possible.

Specifically, we define coefficients \( \boldsymbol { e } _ { 11, } \boldsymbol { e } _ { 12 }, \ldots, \boldsymbol { e } _ { 1 p }\) for the first component in such a way that its variance is maximized, subject to the constraint that the sum of the squared coefficients is equal to one. This constraint is required so that a unique answer may be obtained.

More formally, select \(\boldsymbol { e } _ { 11 , } \boldsymbol { e } _ { 12 } , \ldots , \boldsymbol { e } _ { 1 p }\) that maximizes

\(\text{var}(Y_1) = \sum\limits_{k=1}^{p}\sum\limits_{l=1}^{p}e_{1k}e_{1l}\sigma_{kl} = \mathbf{e}'_1\Sigma\mathbf{e}_1\)

subject to the constraint that

\(\mathbf{e}'_1\mathbf{e}_1 = \sum\limits_{j=1}^{p}e^2_{1j} = 1\)

The second principal component is the linear combination of x-variables that accounts for as much of the remaining variation as possible, with the constraint that the correlation between the first and second components is 0

Select \(\boldsymbol { e } _ { 21 , } \boldsymbol { e } _ { 22 } , \ldots , \boldsymbol { e } _ { 2 p }\) that maximizes the variance of this new component...

\(\text{var}(Y_2) = \sum\limits_{k=1}^{p}\sum\limits_{l=1}^{p}e_{2k}e_{2l}\sigma_{kl} = \mathbf{e}'_2\Sigma\mathbf{e}_2\)

subject to the constraint that the sums of squared coefficients add up to one,

\(\mathbf{e}'_2\mathbf{e}_2 = \sum\limits_{j=1}^{p}e^2_{2j} = 1\)

along with the additional constraint that these two components are uncorrelated.

\(\text{cov}(Y_1, Y_2) = \sum\limits_{k=1}^{p}\sum\limits_{l=1}^{p}e_{1k}e_{2l}\sigma_{kl} = \mathbf{e}'_1\Sigma\mathbf{e}_2 = 0\)

All subsequent principal components have this same property – they are linear combinations that account for as much of the remaining variation as possible and they are not correlated with the other principal components.

We will do this in the same way with each additional component. For instance:

We select \(\boldsymbol { e } _ { i1 , } \boldsymbol { e } _ { i2 } , \ldots , \boldsymbol { e } _ { i p }\) to maximize

\(\text{var}(Y_i) = \sum\limits_{k=1}^{p}\sum\limits_{l=1}^{p}e_{ik}e_{il}\sigma_{kl} = \mathbf{e}'_i\Sigma\mathbf{e}_i\)

subject to the constraint that the sums of squared coefficients add up to one...along with the additional constraint that this new component is uncorrelated with all the previously defined components.

\(\mathbf{e}'_i\mathbf{e}_i = \sum\limits_{j=1}^{p}e^2_{ij} = 1\)

\(\text{cov}(Y_1, Y_i) = \sum\limits_{k=1}^{p}\sum\limits_{l=1}^{p}e_{1k}e_{il}\sigma_{kl} = \mathbf{e}'_1\Sigma\mathbf{e}_i = 0\),

\(\text{cov}(Y_2, Y_i) = \sum\limits_{k=1}^{p}\sum\limits_{l=1}^{p}e_{2k}e_{il}\sigma_{kl} = \mathbf{e}'_2\Sigma\mathbf{e}_i = 0\),

\(\vdots\)

\(\text{cov}(Y_{i-1}, Y_i) = \sum\limits_{k=1}^{p}\sum\limits_{l=1}^{p}e_{i-1,k}e_{il}\sigma_{kl} = \mathbf{e}'_{i-1}\Sigma\mathbf{e}_i = 0\)

Therefore all principal components are uncorrelated from one another.

The solution involves the eigenvalues and eigenvectors of the variance-covariance matrix \(Σ\).

The variance-covariance matrix can be written as the sum over the p eigenvalues, multiplied by the product of the corresponding eigenvector times its transpose as shown in the first expression below:

\begin{align} \Sigma & =  \sum_{i=1}^{p}\lambda_i \mathbf{e}_i \mathbf{e}_i' \\ & \cong  \sum_{i=1}^{k}\lambda_i \mathbf{e}_i\mathbf{e}_i'\end{align}

The second expression is a useful approximation if \(\lambda_{k+1}, \lambda_{k+2}, \dots , \lambda_{p}\) are small. We may approximate Σ by

\(\sum\limits_{i=1}^{k}\lambda_i\mathbf{e}_i\mathbf{e}_i'\)

Again, this is more useful when we talk about factor analysis.

Earlier in the course, we defined the total variation of \(\mathbf{X}\) as the trace of the variance-covariance matrix, that is the sum of the variances of the individual variables. This is also equal to the sum of the eigenvalues as shown below:

\begin{align} trace(\Sigma) & =  \sigma^2_1 + \sigma^2_2 + \dots +\sigma^2_p \\ & =  \lambda_1 + \lambda_2 + \dots + \lambda_p\end{align}

This will give us an interpretation of the components in terms of the amount of the full variation explained by each component. The proportion of variation explained by the ith principal component is then defined to be the eigenvalue for that component divided by the sum of the eigenvalues. In other words, the ith principal component explains the following proportion of the total variation:

\(\dfrac{\lambda_i}{\lambda_1 + \lambda_2 + \dots + \lambda_p}\)

A related quantity is the proportion of variation explained by the first k principal component. This would be the sum of the first k eigenvalues divided by its total variation.

\(\dfrac{\lambda_1 + \lambda_2 + \dots + \lambda_k}{\lambda_1 + \lambda_2 + \dots + \lambda_p}\)

Naturally, if the proportion of variation explained by the first k principal components is large, then not much information is lost by considering only the first k principal components.

When we have a correlation (multicollinearity) between the x-variables, the data may more or less fall on a line or plane in a lower number of dimensions. For instance, imagine a plot of two x-variables that have a nearly perfect correlation. The data points will fall close to a straight line. That line could be used as a new (one-dimensional) axis to represent the variation among data points. As another example, suppose that we have verbal, math, and total SAT scores for a sample of students. We have three variables, but really (at most) two dimensions to the data because total= verbal+math, meaning the third variable is completely determined by the first two. The reason for saying “at most” two dimensions is that if there is a strong correlation between verbal and math, then it may be possible that there is only one true dimension to the data.

Compute the eigenvalues \(\hat{\lambda}_1, \hat{\lambda}_2, \dots, \hat{\lambda}_p\) of the sample variance-covariance matrix S, and the corresponding eigenvectors \(\hat{\mathbf{e}}_1, \hat{\mathbf{e}}_2, \dots, \hat{\mathbf{e}}_p\).

Then we define the estimated principal components using the eigenvectors as the coefficients:

\begin{align} \hat{Y}_1 & =  \hat{e}_{11}X_1 + \hat{e}_{12}X_2 + \dots + \hat{e}_{1p}X_p \\ \hat{Y}_2 & =  \hat{e}_{21}X_1 + \hat{e}_{22}X_2 + \dots + \hat{e}_{2p}X_p \\&\vdots\\ \hat{Y}_p & =  \hat{e}_{p1}X_1 + \hat{e}_{p2}X_2 + \dots + \hat{e}_{pp}X_p \\ \end{align}

Generally, we only retain the first k principal components. Here we must balance two conflicting desires:

1. To obtain the simplest possible interpretation, we want k to be as small as possible. If we can explain most of the variation just by two principal components then this would give us a simple description of the data. When k is small, the first k components explain a large portion of the overall variation. If the first few components explain a small amount of variation, we need more of them to explain a desired percentage of total variance resulting in a large k.
2. To avoid loss of information, we want the proportion of variation explained by the first k principal components to be large. Ideally as close to one as possible; i.e., we want

\(\dfrac{\hat{\lambda}_1 + \hat{\lambda}_2 + \dots + \hat{\lambda}_k}{\hat{\lambda}_1 + \hat{\lambda}_2 + \dots + \hat{\lambda}_p} \cong 1\)

The first principal component is strongly correlated with five of the original variables. The first principal component increases with increasing Arts, Health, Transportation, Housing, and Recreation scores. This suggests that these five criteria vary together. If one increases, then the remaining ones tend to increase as well. This component can be viewed as a measure of the quality of Arts, Health, Transportation, and Recreation, and the lack of quality in Housing (recall that high values for Housing are bad). Furthermore, we see that the first principal component correlates most strongly with the Arts. In fact, we could state that based on the correlation of 0.985 that this principal component is primarily a measure of the Arts. It would follow that communities with high values tend to have a lot of arts available, in terms of theaters, orchestras, etc. Whereas communities with small values would have very few of these types of opportunities.

The second principal component increases with only one of the values, decreasing Health. This component can be viewed as a measure of how unhealthy the location is in terms of available health care including doctors, hospitals, etc.

The third principal component increases with increasing Crime and Recreation. This suggests that places with high crime also tend to have better recreation facilities.

To complete the analysis we oftentimes would like to produce a scatter plot of the component scores.

In looking at the program, you will see a gplot procedure at the bottom where we plot the second component against the first component. A similar plot can also be prepared in Minitab but is not shown here.

Each dot in this plot represents one community. Looking at the red dot out by itself to the right, you may conclude that this particular dot has a very high value for the first principal component and we would expect this community to have high values for the Arts, Health, Housing, Transportation, and Recreation. Whereas if you look at the red dot at the left of the spectrum, you would expect to have low values for each of those variables.

The top dot in blue has a high value for the second component. We would not expect this community to have the best Health Care. And conversely, if you were to look at the blue dot on the bottom, the corresponding community would have high values for Health Care.

Further analyses may include:

• Scatter plots of principal component scores. In the present context, we may wish to identify the locations of each point in the plot to see if places with high levels of a given component tend to be clustered in a particular region of the country, while sites with low levels of that component are clustered in another region of the country.
• Principal components are often treated as dependent variables for regression and analysis of variance.

In the previous example, we looked at a principal components analysis applied to the raw data. In our earlier discussion, we noted that if the raw data is used, then a principal component analysis will tend to give more emphasis to those variables that have higher variances than to those variables that have lower variances. In effect, the results of the analysis will depend on the units of measurement used to measure each variable. That would imply that a principal component analysis should only be used with the raw data if all variables have the same units of measure. And even in this case, only if you wish to give those variables which have higher variances more weight in the analysis.

A unique example of this type of implementation might be in an ecological setting where you are looking at counts of different species of organisms at a number of different sample sites. Here, one may want to give more weight to the more common species that are observed. By analyzing the raw data you will tend to find that more common species will also show higher variances and will be given more emphasis. If you were to do a principal component analysis on standardized counts, all species would be weighted equally regardless of how abundant they are and hence, you may find some very rare species entering in as significant contributors in the analysis. This may or may not be desirable. These types of decisions need to be made with a scientist from the field.

• The results of the principal component analysis depend on the measurement scales.
• Variables with the highest sample variances tend to be emphasized in the first few principal components.
• Principal component analysis using the covariance function should only be considered if all of the variables have the same units of measurement.

If the variables have different units of measurement, (i.e., pounds, feet, gallons, etc), or if we wish each variable to receive equal weight in the analysis, then the variables should be standardized before conducting a principal components analysis.  To standardize a variable, subtract the mean and divide by the standard deviation:

\(Z_{ij} = \frac{X_{ij}-\bar{x}_j}{s_j}\)

where

• \(X_{ij}\) = Data for variable j in sample unit i
• \(\bar{x}_{j}\)= Sample mean for variable j
• \(s_j\) = Sample standard deviation for variable j

The principal components are first calculated by obtaining the eigenvalues for the correlation matrix:

\(\hat{\lambda}_1, \hat{\lambda}_2, \dots, \hat{\lambda}_p\)

In this matrix, we denote the eigenvalues of the sample correlation matrix R and the corresponding eigenvectors

\(\mathbf{\hat{e}}_1, \mathbf{\hat{e}}_2, \dots, \mathbf{\hat{e}}_p\)

The estimated principal components scores are calculated using formulas similar to before, but instead of using the raw data we use the standardized data:

\begin{align} \hat{Y}_1 & =  \hat{e}_{11}Z_1 + \hat{e}_{12}Z_2 + \dots + \hat{e}_{1p}Z_p \\ \hat{Y}_2 & = \hat{e}_{21}Z_1 + \hat{e}_{22}Z_2 + \dots + \hat{e}_{2p}Z_p \\&\vdots\\ \hat{Y}_p & =  \hat{e}_{p1}Z_1 + \hat{e}_{p2}Z_2 + \dots + \hat{e}_{pp}Z_p \\ \end{align}

The rest of the procedure and the interpretations follow as discussed before.

• First Principal Component Analysis - PCA1

  The first principal component is a measure of the quality of Health and the Arts, and to some extent Housing, Transportation, and Recreation. This component is associated with high ratings on all of these variables, especially Health, and Arts. They are all positively related to PCA1 because they all have positive signs.

• Second Principal Component Analysis - PCA2

  The second principal component is a measure of the severity of the crime, the quality of the economy, and the lack of quality education. PCA2 is associated with high ratings of Crime and Economy and low ratings of Education. Here we can see that PCA2 distinguishes cities with high levels of crime and good economies from cities with poor educational systems.

• Third Principal Component Analysis - PCA3

  The third principal component is a measure of the quality of the climate and the poorness of the economy. PCA3 is associated with high Climate ratings and low Economy ratings. The inclusion of economy within this component will add a bit of redundancy to our results. This component is primarily a measure of climate and to a lesser extent the economy.

• Fourth Principal Component Analysis - PCA4

  The fourth principal component is a measure of the quality of education and the economy and the poorness of the transportation network and recreational opportunities. PCA4 is associated with high Education and Economy ratings and low Transportation and Recreation ratings.

• Fifth Principal Component Analysis - PCA5

  The fifth principal component is a measure of the severity of the crime and the quality of housing. PCA5 is associated with high Crime ratings and low housing ratings.

One can interpret this component by component. One method of deciding how many components to include is to choose only those that give unambiguous results, i.e., where no variable appears in two different columns as a significant contribution.

We have to make a decision as to what is an important correlation, not necessarily from a statistical hypothesis testing perspective, but from, in this case, an urban-sociological perspective. You have to decide what is important in the context of the problem at hand. This decision may differ from discipline to discipline. In some disciplines such as sociology and ecology, the data tend to be inherently 'noisy', and in this case, you would expect 'messier' interpretations. If you are looking in a discipline such as engineering where everything has to be precise, you might put higher demands on the analysis. You would want to have very high correlations. Principal component analyses are mostly implemented in sociological and ecological types of applications as well as in marketing research.

As before, you can plot the principal components against one another and explore where the data for certain observations lies.

Sometimes the principal component scores will be used as explanatory variables in a regression. Sometimes in regression settings, you might have a very large number of potential explanatory variables and you may not have much of an idea as to which ones you might think are important. You might perform a principal components analysis first and then perform a regression predicting the variables from the principal components themselves. The nice thing about this analysis is that the regression coefficients will be independent of one another because the components are independent of one another. In this case, you actually say how much of the variation in the variable of interest is explained by each of the individual components. This is something that you can not normally do in multiple regression.

One of the problems with this analysis is that the analysis is not as 'clean' as one would like with all of the numbers involved. For example, in looking at the second and third components, the economy is considered to be significant for both of those components. As you can see, this will lead to an ambiguous interpretation in our analysis.

An alternative method of data reduction is Factor Analysis where factor rotations are used to reduce the complexity and obtain a cleaner interpretation of the data.

In this lesson we learned about:

• The definition of a principal components analysis;
• How to interpret the principal components;
• How to select the number of principal components;
• How to choose between an analysis based on the variance-covariance matrix or the correlation matrix?