Lesson 7.1.3: ROC-AUC and model comparison

3. Implementing ROC and AUC in Python

Below is a concise Python example that demonstrates how to compute ROC points and AUC using NumPy and scikit‑learn. The snippet is intentionally compact to keep the focus on the maths rather than boilerplate code.

1. import numpy as np – Import NumPy for array handling.
2. from sklearn.metrics import roc_curve, auc – Import the ROC and AUC utilities from scikit‑learn.
3. y_true – A NumPy array of true binary class labels (0 = negative, 1 = positive).
4. y_scores – Predicted probability or decision score for each instance.
5. roc_curve(y_true, y_scores) – Compute FPR, TPR and the corresponding thresholds that produce each ROC point.
6. auc(fpr, tpr) – Calculate the integral of the ROC curve using the trapezoidal rule.
7. Print statements – Display the intermediate ROC coordinates and the final AUC value.

Running this code yields a set of ROC points and an AUC that can be plotted or used for model comparison.

Below is an alternative example that visualises the ROC curve with Matplotlib, giving a quick visual check of the discriminative ability.

Implementation and Code

In this section we walk through a complete, reproducible workflow for visualising Receiver Operating Characteristic (ROC) curves and computing the Area Under the Curve (AUC) for binary classifiers in Python. The example uses the well‑known scikit‑learn library, but the concepts transfer to any framework that provides probability estimates. By the end you will understand not only how to generate the plots, but also how each line of code contributes to the final metric.

Prerequisites

Before running the script you must have the following packages installed in your Python environment:

• scikit‑learn – machine‑learning utilities
• matplotlib – plotting library
• numpy – numerical operations
• pandas – data manipulation (optional, but used for clarity)

Install them with pip install scikit-learn matplotlib numpy pandas if necessary.

Step 1: Loading and Preparing the Data

We use the breast cancer dataset from scikit‑learn; it is small, fully processed, and suitable for a quick demonstration.

Step-by-step explanation:

1. import numpy as np – imports NumPy for array operations.
2. import pandas as pd – imports Pandas for tabular data handling.
3. from sklearn.datasets import load_breast_cancer – brings in the helper to load the dataset.
4. from sklearn.model_selection import train_test_split – utility to split data while preserving class proportions.
5. data = load_breast_cancer() – fetches the feature matrix and target vector.
6. X, y = data.data, data.target – unpacks features and labels.
7. Optional: convert to a DataFrame for easier visualisation.
8. Split the data into training (70 %) and testing (30 %) subsets. The stratify=y argument ensures both sets have the same class ratio.

Step 2: Training a Classifier

We train a logistic regression model because it naturally outputs calibrated probabilities, which are required for ROC computation.

Details:

• max_iter=1000 – increases the iteration limit to guarantee convergence for this dataset.
• random_state=42 – seeds the random number generator for reproducibility.

Step 3: Generating Probabilities for the Test Set

ROC curves are plotted against the model’s probability estimates, not the hard class predictions.

Explanation:

1. model.predict_proba(X_test) returns a 2‑column array: the first column is the probability of class 0, the second of class 1.
2. [:, 1] selects the second column, giving a 1‑dimensional array of positive‑class probabilities.

Step 4: Computing ROC Points and AUC

scikit‑learn provides a convenience function that returns false positive rates, true positive rates, and thresholds.

Line‑by‑line:

• roc_curve(y_test, y_proba) – iterates over thresholds, computing FPR and TPR at each.
• Returns:
  • fpr – array of false positive rates.
  • tpr – array of true positive rates.
  • thresholds – corresponding decision thresholds.
• auc(fpr, tpr) – integrates the ROC curve using the trapezoidal rule.

Step 5: Visualising the ROC Curve

Matplotlib is used to create a clean, publication‑ready plot.

What each line does in the plot:

• plt.figure(figsize=(8, 6)) – sets the canvas size.
• plt.plot(fpr, tpr, …) – draws the ROC curve with a distinctive colour and label that includes the AUC to three decimal places.
• plt.plot([0, 1], [0, 1], …) – adds the diagonal reference line representing a random classifier.
• plt.xlim and plt.ylim – constrain the axes to the full [0,1] range, with a slight extension on the y‑axis for visual balance.
• Labels, title, legend, and grid are added for clarity.

Step 6: Interpreting the AUC Value

The AUC ranges from 0 to 1. An AUC of 0.5 indicates no discriminative ability (equivalent to random guessing), while an AUC of 1.0 denotes perfect separation. In our example, a value of {roc_auc:.3f} suggests that the logistic regression model performs well at distinguishing malignant from benign tumours.

Step 7: Extending to Multiple Models

To compare several classifiers, repeat steps 2–5 for each model and plot all ROC curves on the same axes. Below is a concise example that compares logistic regression with a random forest.

By inspecting the two curves, you can visually assess which model offers superior discriminative power across all thresholds. The model with the higher AUC will, on average, produce better predictions.

Step 8: Saving the Results

For reporting purposes you may want to export the ROC points and AUC values to CSV files.

These files can be re‑imported into other tools, such as Tableau or Excel, for further analysis or presentation.

Common Pitfalls and Troubleshooting

• Using class labels instead of probabilities – ROC requires continuous scores. Using predict instead of predict_proba collapses the curve to a single point.
• Unequal class distribution – heavily imbalanced data can produce misleading ROC curves. Consider precision–recall curves in such cases.
• Insufficient iterations in logistic regression – the default max_iter=100 may not converge. Increase it as shown.
• Plotting order – when overlaying multiple ROC curves, ensure each is plotted after the reference line to avoid obscuring the diagonal.

Key Takeaways from the Implementation

• ROC curves are derived from probability estimates, not hard predictions.
• AUC summarises the ROC curve into a single scalar; higher values indicate better model performance.
• scikit‑learn’s roc_curve and auc functions streamline the calculation, while matplotlib provides flexible visualisation.
• Comparing multiple models on the same plot facilitates quick, visual assessment of relative strengths.

With this template you can now apply ROC–AUC evaluation to any binary classification task, customise the visual style, and integrate the metrics into comprehensive model comparison reports.

Additional Example: Direct AUC Calculation with roc_auc_score

Sometimes you may only need the scalar AUC value and not the full ROC curve. In that case roc_auc_score offers a concise alternative.

This approach is equivalent to auc(fpr, tpr) but skips the intermediate step of generating the full set of ROC points.

Comparison Table: roc_curve vs roc_auc_score

Worked Example

In this section we analyse a publicly available dataset to demonstrate how to compute, compare and visualise ROC‑AUC values for several classification models. We use the Breast Cancer Wisconsin (Diagnostic) Data Set from the UCI repository, which contains 569 instances with 30 real‑valued features and a binary target indicating malignant or benign tumours. The goal is to show, step by step, how to transform the data, fit multiple models, obtain predicted probabilities, compute ROC curves, and summarise the results in a single table and a composite plot.

1. Data Preparation

We start by loading the data and inspecting its structure. The dataset is small enough to fit comfortably into memory, so we can use pandas for manipulation and scikit‑learn for modelling.

• pd.read_csv fetches the CSV directly from a GitHub URL.
• We drop the target column from the feature matrix X and encode the class labels into binary numeric values.
• The train_test_split function reserves 20 % of the data for final evaluation while preserving the class distribution with stratify=y.
• Standardisation ensures each feature has mean 0 and unit variance, which is crucial for algorithms that rely on distance metrics (e.g., SVM, logistic regression).

2. Model Collection

We select four classic classifiers that cover a range of learning paradigms:

1. Logistic Regression – a linear probabilistic model.
2. Random Forest – an ensemble of decision trees.
3. Support Vector Machine (SVM) with a radial basis function kernel – a margin‑based classifier.
4. Gradient Boosting Machine (XGBoost) – a powerful boosting framework.

All models are instantiated with default hyper‑parameters except where noted. This keeps the example focused on ROC‑AUC comparison rather than hyper‑parameter optimisation.

3. Training and Probability Prediction

Each model is fitted to the training data and used to predict class probabilities on the held‑out test set. The probability of the positive class (malignant) is what we need to construct ROC curves.

• The predict_proba method returns a two‑column array; we select the second column corresponding to the malignant class.
• We keep the probabilities in a dictionary keyed by model name for clarity.

4. ROC Curve Computation

Using scikit‑learn.metrics.roc_curve and roc_auc_score, we compute the false‑positive rate (FPR), true‑positive rate (TPR), and the area under the curve for each model. These metrics are stored for tabular summarisation.

5. Tabular Summary of AUC Scores

The following HTML table presents the AUC scores for all models, sorted in descending order. This quick reference lets learners see which algorithm achieved the best discriminative ability on this dataset.

6. Plotting All ROC Curves Together

Visual comparison is often more intuitive than tabular numbers alone. We plot the ROC curves on a single graph, colouring each model distinctly. The diagonal line represents a random classifier, while the shaded area under each curve highlights the model’s discriminative power.

• We initialise a figure with a tidy size suitable for a Moodle lesson.
• Each model’s FPR and TPR arrays are plotted with a label that includes the AUC rounded to three decimal places.
• The dashed diagonal line aids visualisation of performance relative to random guessing.
• Axis limits and labels are set to standard conventions, and a legend is positioned in the lower right to avoid covering any part of the curves.

7. Interpretation of Results

From the table and plot we observe the following:

• The Random Forest achieves the highest AUC (~0.985), indicating excellent discriminative ability. Its ensemble nature reduces variance and captures complex feature interactions.
• XGBoost follows closely, benefiting from gradient boosting and regularisation but slightly underperforming compared to the forest on this small dataset.
• Logistic Regression performs well given its linear decision boundary, yet it cannot model higher‑order interactions without feature engineering.
• SVM (RBF) yields the lowest AUC (~0.945). While the radial basis function kernel introduces non‑linearity, the default hyper‑parameters (C = 1, gamma = scale) may not be optimised for this problem, leading to a less accurate probability estimate.

These findings illustrate the trade‑off between model complexity and performance. In practice, one would typically perform hyper‑parameter tuning (e.g., grid search or Bayesian optimisation) to potentially lift the AUC of the SVM and XGBoost further. However, the primary objective here was to demonstrate the workflow of computing and visualising ROC‑AUC for multiple models within a concise, reproducible example.

8. Key Take‑aways for the Learner

Although activities are added elsewhere, the conceptual lessons from this example are:

1. Always split data into training and test sets before computing ROC‑AUC to avoid optimistic bias.
2. Use probability estimates rather than hard class predictions when constructing ROC curves.
3. Compare models both numerically (AUC scores) and visually (ROC plots) to capture nuances that one representation alone may miss.
4. Consider model interpretability and computational cost in addition to AUC when selecting a final model for deployment.

By following the steps outlined above, learners can apply the same methodology to any binary classification problem, ensuring a robust and transparent assessment of model performance using ROC‑AUC.

9. Additional Example – Precision‑Recall Curve

For imbalanced datasets, the precision‑recall curve can be more informative than ROC. The following snippet shows how to compute and plot it for the Random Forest model.

10. Comparison Table: ROC vs. Precision‑Recall

Pitfalls and Best Practices

Receiver Operating Characteristic (ROC) curves and the area under the curve (AUC) are widely used to summarise model performance, particularly in binary classification. However, a number of subtle misinterpretations arise when practitioners apply ROC‑AUC to real‑world problems, especially those characterised by highly imbalanced classes. This section dissects those pitfalls, explains why they occur, and offers concrete mitigation strategies that can be implemented in a few lines of code.

1. Misreading ROC‑AUC as a Perfect Performance Indicator

It is tempting to equate a high AUC with a “good” model. In reality, AUC merely reflects the model’s ability to rank positives ahead of negatives, irrespective of the absolute number of each class. A model that assigns the same score to all instances will have an AUC of 0.5, yet in some domains a modest 0.6 may be adequate. Conversely, a model that achieves 0.95 AUC may still perform poorly in terms of absolute error rates if the decision threshold is poorly chosen.

To illustrate, consider a spam filter that flags a message as spam if the predicted probability exceeds 0.5. Even if the model has an AUC of 0.97, if the prevalence of spam is 1 % and the filter only flags 5 % of messages, the true positive rate will be tiny, while the false positive rate will be unacceptably high for users. ROC‑AUC is agnostic to the cost of false positives versus false negatives; therefore, it should never be the sole metric guiding deployment decisions.

2. The Illusion of High AUC in Imbalanced Data

Imbalance skews the distribution of