Tutorial 0215: ROC Curve

Welcome

Welcome to our interactive tutorial that uses the quarto-webr extension to generate interactive code cells with Quarto and webR.

Downloading Data

We will use our “toy” dataset mnist_27 from the dslab package.

Modified kNN

We use ‘modified’ kNN, which predicts probabilities, instead of labels.

Positive Class

We need to decide the positive class. Let’s say we are interested in the digit 2. Then, we can define the positive class as 2 and the negative class as 7. We can then calculate the TPR and FPR for a given threshold.

TPR and FPR

Since we have the predicted probabilities, we can try different thresholds to see how the ROC curve changes. We should first a function that calculates TPR and FPR for a given threshold.

ROC Curve

ROC Curve Plot

ROCR Package

AUC

It seems that as the “area under the curve” (AUC) is high, it shows the . Let’s calculate it

Fin

Now we need to know how to use AUC to tune a classification model, like kNN.