11.2.4 - Measure of Agreement: Kappa

Recall the example on movie ratings from the introduction. Do the two movie critics, in this case, Siskel and Ebert, classify the same movies into the same categories; do they really agree?

In the (necessarily) square table above, the main diagonal counts represent movies where both raters agreed. Let the term \(\pi_{ij}\) denote the probability that Siskel classifies the move in category \(i\), and Ebert classifies the same movie in category \(j\). For example, \(\pi_{13}\) is the probability that Siskel rates a movie as "con", but Ebert rates it as "pro".

The term \(\sum_{i} \pi_{ii}\) then is the total probability of agreement. The extreme case that all observations are classified on the main diagonal is known as "perfect agreement".

Cohen’s kappa is a single summary index that describes strength of inter-rater agreement.

For \(I \times I\) tables, it’s equal to

\(\kappa=\dfrac{\sum\pi_{ii}-\sum\pi_{i+}\pi_{+i}}{1-\sum\pi_{i+}\pi_{+i}}\)

This statistic compares the observed agreement to the expected agreement, computed assuming the ratings are independent.

The null hypothesis that the ratings are independent is, therefore, equivalent to

\(\pi_{ii}=\pi_{i+}\pi_{+i}\quad\text{ for all }i\)

If the observed agreement is due to chance only, i.e., if the ratings are completely independent, then each diagonal element is a product of the two marginals.

Since the total probability of agreement is \(\sum_{i} \pi_{ii}\), then the probability of agreement under the null hypothesis equals to \(\sum_{i} \pi_{i+}\pi_{+i}\). Note also that \(\sum_{i} \pi_{ii} = 0\) means no agreement, and \(\sum_{i} \pi_{ii} = 1\) indicates perfect agreement. The kappa statistic is defined so that a larger value implies stronger agreement. Furthermore,

• Perfect agreement \(\kappa = 1\).
• \(\kappa = 0\), does not mean perfect disagreement but rather only agreement that would result from chance only, where the diagonal cell probabilities are simply products of the corresponding marginals.
• If the actual agreement is greater than agreement by chance, then \(\kappa\geq 0\).
• If the actual agreement is less than agreement obtained by chance, then \(\kappa\leq 0\).
• The minimum possible value of \(\kappa = −1\).
• A value of kappa higher than 0.75 can be considered (arbitrarily) as "excellent" agreement, while lower than 0.4 will indicate "poor" agreement.

Under multinomial sampling, the sampled value \(\hat{\kappa}\) has a large-sample normal distribution. Thus, we can rely on the asymptotic 95% confidence interval.

data critic;
input siskel $ ebert $ count ;
datalines;
 con con  24 
 con mixed 8 
 con pro 13
 mixed con 8 
 mixed mixed 13
 mixed pro 11
 pro con 10
 pro mixed 9
 pro pro 64
 ; run;
 
proc freq; 
weight count;
tables siskel*ebert / agree chisq;
run;

critic = matrix(c(24,8,10,8,13,9,13,11,64),nr=3,
 dimnames=list("siskel"=c("con","mixed","pro"),"ebert"=c("con","mixed","pro")))
critic

# chi-square test for independence between raters
result = chisq.test(critic)
result

# kappa coefficient for agreement
library(vcd)
kappa = Kappa(critic)

> kappa
            value     ASE     z  Pr(>|z|)
Unweighted 0.3888 0.05979 6.503 7.870e-11
Weighted   0.4269 0.06350 6.723 1.781e-11
> confint(kappa)
            
Kappa              lwr       upr
  Unweighted 0.2716461 0.5060309
  Weighted   0.3024256 0.5513224

 Issue with Cohen's Kappa

Kappa strongly depends on the marginal distributions. That is the same rating but with different proportions of cases in different categories can give very different \(\kappa\) values. This is one reason why the minimum value of \(\kappa\) depends on the marginal distribution and the minimum possible value of \(-1\) is not always attainable.

 Solution

Modeling agreement (e.g., via log-linear or other models) is typically a more informative approach.

Weighted kappa is a version of kappa used for measuring agreement on ordered variables, where certain disagreements (e.g., lowest versus highest) can be weighted as more or less important.