7.4 - Receiver Operating Characteristic Curve (ROC)

A Receiver Operating Characteristic Curve (ROC) is a standard technique for summarizing classifier performance over a range of trade-offs between true positive (TP) and false positive (FP) error rates (Sweets, 1988). ROC curve is a plot of sensitivity (the ability of the model to predict an event correctly) versus 1-specificity for the possible cut-off classification probability values \(\pi_0\).

For logistic regression we can create a \(2\times 2\) classification table of predicted values from your model for the response if \(\hat{y}=0\) or 1 versus the true value of \(y = 0\) or 1. The prediction if \(\hat{y}=1\) depends on some cut-off probability, \(\pi_0\). For example, \(\hat{y}=1\) if \(\hat{\pi}_i>\pi_0\) and \(\hat{y}=0\) if \(\hat{\pi}_i \leq \pi_0\). The most common value for \(\pi_0 = 0.5\). Then \(sensitivity=P(\hat{y}=1|y=1)\) and \(specificity=P(\hat{y}=0|y=0)\).

The ROC curve is more informative than the classification table since it summarizes the predictive power for all possible \(\pi_0\).

The position of the ROC on the graph reflects the accuracy of the diagnostic test. It covers all possible thresholds (cut-off points). The ROC of random guessing lies on the diagonal line. The ROC of a perfect diagnostic technique is a point at the upper left corner of the graph, where the TP proportion is 1.0 and the FP proportion is 0.

The Area Under the Curve (AUC), also referred to as index of accuracy (A), or concordance index, \(c\), in SAS, and it is an accepted traditional performance metric for a ROC curve. The higher the area under the curve the better prediction power the model has. \(c = 0.8 \) can be interpreted to mean that a randomly selected individual from the positive group has a test value larger than that for a randomly chosen individual from the negative group 80 percent of the time.

options nocenter nodate nonumber linesize=72;
data assay; 
input logconc y n; 
cards; 
2.68  10  31 
2.76  17  30 
2.82  12  31 
2.90   7  27 
3.02  23  26 
3.04  22  30 
3.13  29  31 
3.20  29  30 
3.21  23  30 
; 
run; 
                                                                                 
proc logistic data=assay; 
  model y/n= logconc / scale=pearson outroc=roc1; 
  output out=out1 xbeta=xb reschi=reschi; 
  run; 
                                                                                 
axis1 label=('Linear predictor');
axis2 label=('Pearson Residual');
proc gplot data=out1; 
  title 'Residual plot'; 
  plot reschi * xb / haxis=axis1 vaxis=axis2; 
run; 
symbol1 i=join v=none c=blue;
proc gplot data=roc1;
  title 'ROC plot';
  plot  _sensit_*_1mspec_=1 / vaxis=0 to 1 by .1 cframe=ligr ;
run;

                                  #### ROC curve
                                  #### sensitivity vs 1-specificity
                                  lp = result$linear.predictors
                                  p = exp(lp)/(1+exp(lp))
                                  cbind(yes,no,p)
                                  p0 = 0
                                  sens = 1
                                  spec = 0
                                  total = 100
                                  for (i in (1:total)/total)
                                  { 
                                  yy = sum(r*(p>=i))
                                  yn = sum(r*(p=i))
                                  nn = sum(n*(p<i))