If the two models have nearly the same values for the odds ratio, choose the simpler one.

However, what is ”nearly the same values”? What do we do?

This is a subjective decision, and depends on the purpose of the data.

They seem close enough, but what else can we look at...

Dissimilarity Index

The Dissimilarity Index is a summary statistic describing how close the fitted values of a model are to the observed data (e.g. observed counts). For a table of any dimension, it equals

For our example used in this lesson, see collarDIndex.sas.

For the model (MW, MS):

D = 55.2306 / 2 (715) = 0.039

What does this mean? We could interpret this values such that, we would need to move 3.9% percent of the observations to achieve a perfect fit of the model (MW, MS) to observed (sample) data.

For the model (MW, MS, SW):

D = 5.8888 / 2 (715) = 0.004

Here we would only need to move 0.4% of the observations to achieve a perfect fit of the models (MW, MS, SW) to the observed data.

Which one do we use? Possibly the model of conditional independence. I, personally, could be OK with moving about 3.9% of the data.

Properties of D

• 0 ≤ D ≤ 1
• D = the proportion of sample cases that need to move to a different cell to have the model fit perfectly.
• Small D means that there is little difference between fitted values and observed counts.
• Larger D means that there is a big difference between fitted values and observed counts.
• D is an estimate of the change, Δ, which measures the lack-of-fit of the model in the population.
  • When the model fits perfectly in the population: Δ = 0, and D overestimates the lack-of-fit (especially for small samples).
  • For large samples when the model does not fit perfectly: G² > and X² will be large, and D reveals when the lack-of-fit is important in a practical sense.
• Rule-of-thumb: D < 0.03 indicates non-important lack-of-fit.

Correlations between observed and fitted counts

A large correlation value indicates that the observed and fitted values are close.

For our example, for the conditional independence model (MW, MS) in our example, r = 0.9906, and for the homogeneous model, r = 0.9999. Is there really a difference here? If I round to two decimal places, there is no difference at all!

Information Criteria

These are different statistics that compare fitted and observed values, that is the loglikelihoods of two models, but take into account the complexity of the model and/or sample size.

The complexity typically refers to the number of parameters in the model.

Note: Unlike partial association tests, these statistics do NOT require the models to be nested.

The two most commonly used ones are:

Akiake Information Criteria (AIC)

AIC = −2 × LogL + 2 × number of parameters

Bayesian Information Criteria (BIC)

BIC = G² − (df) × logN = −2 × log(B)

where N = total number of observations, and B = posterior odds.

Note: The smaller the value, the better the model.

BIC Statistic and Bayesian Model Selection

BIC gives a way of considering the trade-off between a simple parsimonous model (practical significance) and a more complex and closer to reality model (statistical significance).

Suppose we are considering a model M₀, and comparing it to the saturated model, M₁. Which model gives a better description of the main features as reflected in the data? (e.g. which of the two is more likely to be the true model? We can use posterior odds which then figure into the BIC:

where X = data.

• When comparing a set of models, choose the one with the smallest BIC value.
• If BIC is negative choose M₀
• The models do not have to be nested.
• This procedure provides you with a consistent model in the sense that in large samples, it chooses the correct model with high probability.

The following table gives both BIC and AIC for our running example.

Which model would you choose based on BIC? How about AIC? Is it the same model?

Look for the smallest value. In this case the smallest BIC is -7.75, indicating the conditional independence model. How about AIC?

Please note that there is a lot of debate about which value, BIC or AIC is more helpful. Best is to report both when making your decision.

References:

• Raftery, A.E. (1985). A note on Bayes factors for log-linear contingency table models with vague prior information. Journal of the Royal Statistical Society, Series B.
• Raftery, A. E. (1986). Choosing models for cross-classifications. American Sociological Review, 51, 145146.
• Spiegelhalter, D.J. and Smith, A.F.M. (1982). Bayes Factors for linear and log-linear models with vague prior information. Journal of the Royal Statistical Society, Series B, 44, 377387.