12.6 - Final Notes about the Principal Component Method

Unlike the competing methods, the estimated factor loadings under the principal component method do not change as the number of factors is increased. This is not true of the remaining methods (e.g., maximum likelihood). However, the communalities and the specific variances will depend on the number of factors in the model. In general, as you increase the number of factors, the communalities increase toward one and the specific variances will decrease toward zero.

The diagonal elements of the variance-covariance matrix \(\mathbf{S}\) (or \(\mathbf{R}\)) are equal to the diagonal elements of the model:

\(\mathbf{\hat{L}\hat{L}' + \mathbf{\hat{\Psi}}}\)

The off-diagonal elements are not exactly reproduced. This is in part due to variability in the data - just random chance. Therefore, we want to select the number of factors to make the off-diagonal elements of the residual matrix small:

\(\mathbf{S - (\hat{L}\hat{L}' + \hat{\Psi})}\)

Here, we have a trade-off between two conflicting desires. For a parsimonious model, we wish to select the number of factors m to be as small as possible, but for such a model, the residuals could be large. Conversely, by selecting m to be large, we may reduce the sizes of the residuals but at the cost of producing a more complex and less interpretable model (there are more factors to interpret).

Another result to note is that the sum of the squared elements of the residual matrix is equal to the sum of the squared values of the eigenvalues left out of the matrix:

\(\sum\limits_{j=m+1}^{p}\hat{\lambda}^2_j\)

General Methods used in determining the number of Factors

Below are three common techniques used to determine the number of factors to extract:

1. Cumulative proportion of at least 0.80 (or 80% explained variance)
2. Eigenvalues of at least one
3. Scree plot is based on the "elbow" of the plot; that is, where the plot turns and begins to flatten out