Lesson 6.3.3: Interpreting performance differences across models

flowchart TD
    A[Collect Model Results] --> B[Compute Performance Metrics]
    B --> C[Compare Metrics]
    C --> D[Identify Differences]
    D --> E[Analyse Causes]
    E --> F[Document Findings]
    A --> G[Select Models]
    G --> A

Conceptual Foundations of Performance Comparison

Machine learning (ML) and deep learning (DL) are frequently evaluated using the same set of metrics – accuracy, precision, recall, F1‑score, area under the ROC curve (AUC), mean squared error (MSE), etc. Despite this common language, the reasons why one model outperforms another on a given metric are deeply rooted in the statistical, algorithmic and representational characteristics of the models themselves. Understanding these foundations is essential for interpreting performance differences in a principled way, rather than treating a higher score as a definitive sign of superiority.

Statistical Bias–Variance Trade‑off

Every predictive model can be decomposed into three components that determine its expected error on new data: bias, variance and irreducible error. Bias measures how far the model’s predictions are systematically away from the true function; variance captures how sensitive the model is to fluctuations in the training data; and irreducible error is the noise inherent in the problem.

• High‑bias models (e.g. linear regression with few features) tend to underfit: they cannot capture complex relationships, leading to systematically poor predictions.
• High‑variance models (e.g. deep neural networks with many parameters) can fit the training data almost perfectly but may over‑adapt to noise, resulting in poor generalisation.

When comparing two models, a higher accuracy may simply reflect a different bias–variance profile rather than a true advantage. For example, a decision tree that has grown too deep can achieve near‑perfect training accuracy but may exhibit a high variance, whereas a shallow tree will generalise better on unseen data.

Capacity and Expressiveness

Model capacity refers to the richness of the function class that a learning algorithm can realise. Deep learning architectures, with their multiple nonlinear layers, typically possess far greater capacity than shallow ML models such as support vector machines (SVMs) with linear kernels. This has two important implications:

1. Feature learning – DL models automatically discover hierarchical representations from raw data, while traditional ML models rely on hand‑crafted features.
2. Risk of over‑fitting – a highly expressive model can perfectly fit idiosyncratic patterns, which may not generalise.

Consequently, a DL model might show superior performance on a training set but not on a held‑out test set if the data distribution is not sufficiently large or diverse.

Regularisation and Generalisation

Regularisation techniques are mechanisms that constrain the model to prevent over‑fitting. In classical ML, L1 and L2 penalties are applied to the weights of linear models or kernels. In DL, regularisation takes many forms: dropout, weight decay, early stopping, data augmentation, batch normalisation, etc. These methods effectively reduce the effective capacity of a deep network, striking a balance between fitting the training data and maintaining generalisation.

When comparing models, the presence or absence of regularisation can dramatically alter performance metrics. For instance, a neural network trained without dropout may achieve a higher training accuracy but a lower test accuracy than a network that employs dropout, illustrating the importance of controlling variance.

Evaluation Protocols and Data Leakage

Performance metrics are only as reliable as the evaluation protocol used to compute them. Common pitfalls include:

• Data leakage – inadvertently allowing information from the test set to influence model training (e.g. by normalising using global statistics).
• Inadequate cross‑validation – using a single train/test split can give a misleading estimate of generalisation.
• Class imbalance – metrics like accuracy can be inflated when one class dominates.

Proper evaluation typically involves nested cross‑validation, stratified sampling, and the use of multiple metrics that capture different aspects of performance (precision vs recall, ROC vs PR curves).

Model Interpretability versus Performance

There is a well‑known tension between model complexity and interpretability. Simpler models such as decision trees or linear models are easier to explain but may lack the expressive power of DL architectures. In high‑stakes domains (medicine, finance), a marginal improvement in accuracy might not justify the loss of interpretability if the model’s decisions cannot be audited.

Thus, when interpreting performance differences, one must consider the context: Is the higher metric value worth the potential cost in transparency or robustness?

Example: Comparing a Logistic Regression and a Two‑Layer Neural Network on a Binary Classification Task

Below is a concise Python snippet that demonstrates how to compute and compare the ROC AUC for a logistic regression model and a simple neural network using scikit‑learn and PyTorch. The code is annotated line by line to illustrate the key steps.

Import libraries
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from sklearn.linear_model import LogisticRegression
import torch
import torch.nn as nn
import torch.optim as optim

1. Generate synthetic data
X, y = np.random.randn(1000, 20), np.random.randint(0, 2, 1000)

2. Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

3. Logistic regression model
logreg = LogisticRegression(max_iter=1000)
logreg.fit(X_train, y_train)
y_prob_logreg = logreg.predict_proba(X_test)[:, 1]
auc_logreg = roc_auc_score(y_test, y_prob_logreg)

4. Two‑layer neural network
class SimpleNN(nn.Module):
    def __init__(self, input_dim):
        super().__init__()
        self.fc1 = nn.Linear(input_dim, 50)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(50, 1)
        self.sigmoid = nn.Sigmoid()
    def forward(self, x):
        return self.sigmoid(self.fc2(self.relu(self.fc1(x))))

model = SimpleNN(20)
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

5. Training loop
for epoch in range(50):
    model.train()
    optimizer.zero_grad()
    inputs = torch.tensor(X_train, dtype=torch.float)
    labels = torch.tensor(y_train, dtype=torch.float).unsqueeze(1)
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

6. Evaluation
model.eval()
with torch.no_grad():
    test_inputs = torch.tensor(X_test, dtype=torch.float)
    test_probs = model(test_inputs).squeeze().numpy()
auc_nn = roc_auc_score(y_test, test_probs)

7. Print results
print(f'Logistic Regression AUC: {auc_logreg:.3f}')
print(f'Neural Network AUC: {auc_nn:.3f}')

Line‑by‑line explanation:

1. Imports required libraries: NumPy for numeric operations, scikit‑learn for data handling and evaluation, PyTorch for the neural network.
2. Creates synthetic data: 1 000 samples with 20 features, binary labels.
3. Splits the data into training (80 %) and test (20 %) sets, maintaining class proportions with stratify=y.
4. Initialises and trains a logistic regression model, then obtains predicted probabilities for the positive class.
5. Computes the ROC AUC for the logistic regression.
6. Defines a minimal two‑layer neural network with 50 hidden units, ReLU activation, and a sigmoid output.
7. Sets up binary cross‑entropy loss and the Adam optimiser.
8. Runs a 50‑epoch training loop, performing forward passes, loss computation, back‑propagation and optimiser steps.
9. Switches to evaluation mode (disables dropout, batch norm updates) and computes output probabilities on the test set.
10. Calculates the ROC AUC for the neural network.
11. Prints both AUC scores for comparison.

In practice, the neural network might achieve a higher AUC if the data contain non‑linear relationships that the logistic regression cannot capture. However, if the dataset is small or noisy, the neural network could over‑fit, yielding a lower test AUC despite a higher training AUC. This example illustrates how the same metric (AUC) can be influenced by model capacity, regularisation, and the evaluation protocol.

Interpreting Performance Differences in Context

To move beyond surface‑level comparisons, analysts should adopt a multi‑faceted approach:

• Statistical significance testing – use permutation tests or bootstrapping to determine whether observed metric differences are unlikely to have arisen by chance.
• Confidence intervals – report 95 % CIs for metrics to express uncertainty.
• Calibration assessment – check whether predicted probabilities are well‑calibrated (e.g. via reliability diagrams).
• Robustness checks – evaluate models on data perturbed by noise, adversarial attacks or distributional shift.
• Cost‑benefit analysis – weigh marginal metric gains against computational cost, training time, and deployment constraints.

Only by integrating these perspectives can one draw meaningful conclusions about why one model outperforms another, and whether that superiority holds in real‑world deployments.

New Example: Multi‑Class Classification with Imbalanced Data – Using Macro‑F1 and Class‑Weighted Loss

Below is a lightweight example that demonstrates how to handle class imbalance in a multi‑class setting using scikit‑learn’s class_weight parameter and a weighted cross‑entropy loss in PyTorch. The focus is on the impact of weighting on the macro‑F1 score, which averages F1 across classes irrespective of support.

1. Import libraries
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score
from sklearn.linear_model import LogisticRegression
import torch
import torch.nn as nn
import torch.optim as optim

2. Generate imbalanced synthetic data
X, y = np.random.randn(1500, 10), np.random.choice([0, 1, 2], 1500, p=[0.7, 0.2, 0.1])

3. Train‑test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=1, stratify=y)

4. Logistic regression with class weights (inverse frequency)
class_weights = {0: 1/0.7, 1: 1/0.2, 2: 1/0.1}
logreg = LogisticRegression(max_iter=1000, class_weight=class_weights)
logreg.fit(X_train, y_train)
y_pred_logreg = logreg.predict(X_test)
macro_f1_logreg = f1_score(y_test, y_pred_logreg, average='macro')

5. Simple neural network with weighted cross‑entropy
class SimpleNet(nn.Module):
    def __init__(self, input_dim, num_classes):
        super().__init__()
        self.fc = nn.Linear(input_dim, num_classes)
    def forward(self, x):
        return self.fc(x)

model = SimpleNet(10, 3)
criterion = nn.CrossEntropyLoss(weight=torch.tensor([1/0.7, 1/0.2, 1/0.1], dtype=torch.float))
optimizer = optim.Adam(model.parameters(), lr=0.005)

6. Training loop
for _ in range(60):
    model.train()
    optimizer.zero_grad()
    inputs = torch.tensor(X_train, dtype=torch.float)
    labels = torch.tensor(y_train, dtype=torch.long)
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

7. Evaluation
model.eval()
with torch.no_grad():
    test_inputs = torch.tensor(X_test, dtype=torch.float)
    logits = model(test_inputs)
    preds = torch.argmax(logits, dim=1).numpy()
macro_f1_nn = f1_score(y_test, preds, average='macro')

print(f'Logistic Regression Macro‑F1: {macro_f1_logreg:.3f}')
print(f'Neural Network Macro‑F1: {macro_f1_nn:.3f}')

In this snippet, both models are explicitly told to pay more attention to minority classes through weighting. The resulting macro‑F1 scores provide a balanced view of performance across all classes, which is often more informative than accuracy in imbalanced scenarios.

Comparing Evaluation Approaches in a Tabular Summary

Implementation & Code for Comparative Evaluation

Once you have trained several candidate models you must decide which one delivers the best performance. The process is not simply a matter of looking at a single score; you need to compute a range of metrics, visualise their distributions, and perform statistical tests to confirm that any observed differences are unlikely to be due to random chance. In this section we lay out a reproducible workflow that you can copy into a Jupyter notebook or any Python script, and we explain each line so that you can adapt it to your own datasets and models.

1. Importing the essential libraries

We use scikit‑learn for the metrics, statsmodels for the statistical tests, and pandas for tidy data handling. The seaborn library is optional but useful for visualising the metric distributions.

import numpy as np
import pandas as pd
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
from sklearn.model_selection import cross_val_predict
import statsmodels.stats.multicomp as mc
import scipy.stats as stats
import seaborn as sns
import matplotlib.pyplot as plt

Line‑by‑line:

• numpy and pandas provide numerical and tabular data structures.
• accuracy_score, precision_score etc. are standard classification metrics.
• cross_val_predict gives out‑of‑fold predictions, which are essential for unbiased metric estimation.
• statsmodels.stats.multicomp supplies post‑hoc tests for comparing multiple groups.
• scipy.stats gives access to classical tests such as the paired t‑test and Wilcoxon signed‑rank test.
• seaborn and matplotlib are used for visualisation.

2. Preparing the data and models

Assume you have a feature matrix X and a target vector y. You also have a list of instantiated scikit‑learn estimators stored in models. To keep the example focused, we illustrate with three classifiers.

Example data: replace with your own loading logic
X, y = load_my_dataset()

Define models to compare
models = {
    'LogisticRegression': LogisticRegression(max_iter=1000),
    'RandomForest': RandomForestClassifier(n_estimators=200),
    'GradientBoosting': GradientBoostingClassifier(n_estimators=200)
}

Here load_my_dataset() is a placeholder; in practice you would load a CSV, database, or other source. Each model is stored in a dictionary keyed by a friendly name for later reference.

3. Computing cross‑validated predictions and metrics

The key to a fair comparison is to use the same cross‑validation scheme for every model. We use stratified 5‑fold CV to preserve class balance.

cv_folds = 5
scoring = ['accuracy', 'precision_macro', 'recall_macro', 'f1_macro', 'roc_auc']

Create a DataFrame to hold metrics
metric_df = pd.DataFrame(columns=['model'] + scoring)

for name, model in models.items():
    # Obtain out‑of‑fold predictions for each fold
    y_pred = cross_val_predict(model, X, y, cv=cv_folds, method='predict')
    y_proba = cross_val_predict(model, X, y, cv=cv_folds, method='predict_proba')[:, 1]  # probability of class 1

    # Compute metrics
    acc = accuracy_score(y, y_pred)
    prec = precision_score(y, y_pred, average='macro')
    rec = recall_score(y, y_pred, average='macro')
    f1 = f1_score(y, y_pred, average='macro')
    auc = roc_auc_score(y, y_proba)

    # Append to DataFrame
    metric_df = metric_df.append({
        'model': name,
        'accuracy': acc,
        'precision_macro': prec,
        'recall_macro': rec,
        'f1_macro': f1,
        'roc_auc': auc
    }, ignore_index=True)

Line‑by‑line:

• cv_folds = 5 sets the number of stratified folds.
• We define a list of metric names to keep the code tidy.
• For each model, cross_val_predict returns predictions that are made on data not used to train the model in that fold, ensuring an unbiased estimate.
• We compute the probability of the positive class (index 1) for ROC‑AUC.
• Each metric is calculated once, then the results are appended to a DataFrame for later aggregation.

4. Visualising metric distributions

While the aggregated scores give a quick snapshot, visualising the spread of each metric across folds can reveal hidden variability. The following code creates violin plots for each metric by model.

Reshape metrics for seaborn
melted = metric_df.melt(id_vars='model', var_name='metric', value_name='score')

plt.figure(figsize=(10,6))
sns.violinplot(x='metric', y='score', hue='model', data=melted, split=True)
plt.title('Metric distribution across models')
plt.tight_layout()
plt.show()

Because we only have one score per fold (the aggregated score), the violin plots will appear as thin bars, but if you record the metric for each fold separately you can observe genuine distributional differences.

5. Statistical comparison across models

Once you have the metric values for each model you can test whether any observed differences are statistically significant. Two common approaches are:

1. Paired t‑test – assumes normal distribution of differences and is suitable for continuous metrics.
2. Wilcoxon signed‑rank test – non‑parametric alternative that does not assume normality.

Below we perform both tests for the accuracy metric, but you can replicate the logic for any other metric.

Extract accuracy scores for each model
acc_scores = {name: [] for name in models.keys()}

Re‑run cross‑validation but store per‑fold results
for name, model in models.items():
    y_pred = cross_val_predict(model, X, y, cv=cv_folds, method='predict')
    acc = accuracy_score(y, y_pred)
    acc_scores[name].append(acc)

Convert to arrays
acc_logreg = np.array(acc_scores['LogisticRegression'])
acc_rf = np.array(acc_scores['RandomForest'])
acc_gb = np.array(acc_scores['GradientBoosting'])

Paired t‑test
t_stat_lr_rf, p_lr_rf = stats.ttest_rel(acc_logreg, acc_rf)
t_stat_lr_gb, p_lr_gb = stats.ttest_rel(acc_logreg, acc_gb)

Wilcoxon signed‑rank test
w_stat_lr_rf, p_w_lr_rf = stats.wilcoxon(acc_logreg, acc_rf)
w_stat_lr_gb, p_w_lr_gb = stats.wilcoxon(acc_logreg, acc_gb)

print(f'Paired t‑test LR vs RF: t={t_stat_lr_rf:.3f}, p={p_lr_rf:.3f}')
print(f'Paired t‑test LR vs GB: t={t_stat_lr_gb:.3f}, p={p_lr_gb:.3f}')
print(f'Wilcoxon LR vs RF: W={w_stat_lr_rf:.3f}, p={p_w_lr_rf:.3f}')
print(f'Wilcoxon LR vs GB: W={w_stat_lr_gb:.3f}, p={p_w_lr_gb:.3f}')

Line‑by‑line:

• We initialise a dictionary to store accuracy per fold for each model.
• We loop again over the models, performing cross‑validated predictions and computing accuracy.
• We convert the lists to NumPy arrays for vectorised operations.
• The ttest_rel function performs a paired t‑test; the output is the test statistic and the two‑sided p‑value.
• The wilcoxon function returns the test statistic and p‑value for the non‑parametric test.
• Finally, we print the results in a readable format.

Interpreting the output is straightforward: a p‑value below your chosen alpha (commonly 0.05) indicates that the difference between the two models is unlikely to have arisen by chance.

6. Post‑hoc multiple comparison adjustment

When you compare more than two models you risk an inflated type‑I error rate. The statsmodels.stats.multicomp.MultiComparison class automatically applies Bonferroni or Tukey adjustments. Below is an example using Tukey’s honestly significant difference (HSD) test.

Prepare data for Tukey HSD
tukey_data = pd.DataFrame({
    'model': np.repeat(list(models.keys()), cv_folds),
    'accuracy': np.concatenate([acc_scores[name] for name in models.keys()])  # flattened list of accuracies
})

comp = mc.MultiComparison(tukey_data['accuracy'], tukey_data['model'])
posthoc = comp.tukeyhsd(alpha=0.05)
print(posthoc.summary())

Line‑by‑line:

• We create a long‑form DataFrame where each row represents a single fold’s accuracy for a particular model.
• The MultiComparison object is initialised with the metric values and the grouping variable.
• Calling tukeyhsd performs the pairwise comparisons and automatically adjusts the p‑values.
• The summary() method prints a readable table showing which pairs differ significantly.

7. Packaging the results into a tidy table

For reporting purposes you often want a single table that lists the mean and standard deviation of each metric for every model, along with significance markers from the post‑hoc tests.

Compute mean ± std for each metric
summary = metric_df.groupby('model').agg(['mean', 'std']).reset_index()
summary.columns = ['model', 'accuracy_mean', 'accuracy_std',
                   'precision_macro_mean', 'precision_macro_std',
                   'recall_macro_mean', 'recall_macro_std',
                   'f1_macro_mean', 'f1_macro_std',
                   'roc_auc_mean', 'roc_auc_std']

Merge with Tukey results (converting to a wide format)
tukey_df = posthoc.summary().data[1:-1]  # skip header/footer
tukey_df = pd.DataFrame(tukey_df, columns=['group1', 'group2', 'meandiff', 'p_adj', 'reject'])
tukey_wide = tukey_df.pivot(index='group1', columns='group2', values='reject').fillna(False)

Add significance markers to the summary table
for col in summary.columns[1:]:
    summary[col] = summary[col].round(3)

print(summary.to_markdown(index=False))

In the table you will see columns such as accuracy_mean and accuracy_std. The reject column in the Tukey result indicates whether the difference between two models is significant after adjustment. You can append a simple asterisk or colour‑coding in your Moodle lesson for visual emphasis.

8. Summary of best practices

• Keep the CV scheme consistent across all models; otherwise you introduce bias.
• Record per‑fold metrics to allow proper statistical testing and visualisation.
• Use paired tests when the same data splits are reused across models.
• When you have more than two models, apply a multiple‑comparison correction to control the family‑wise error rate.
• Always visualise metric distributions; a single mean can hide large variance or outliers.

By following the workflow above you can produce a rigorous, reproducible comparison of any number of predictive models. The code snippets are intentionally modular: you can drop them into a single script, adapt the metric list, or replace the classifiers with your own custom objects. The statistical analysis gives you confidence that the best‑performing model is genuinely superior, rather than a fluke of the particular train–test split.

Alternative metric calculation: Precision‑Recall AUC

For imbalanced datasets, the area under the precision‑recall curve (PR AUC) can be more informative than ROC AUC. The following snippet shows how to compute PR AUC for each model using out‑of‑fold probability estimates.

Compute PR AUC for each model
pr_auc_scores = {}
for name, model in models.items():
    y_pred_proba = cross_val_predict(model, X, y, cv=cv_folds, method='predict_proba')[:, 1]
    pr_auc = metrics.average_precision_score(y, y_pred_proba)
    pr_auc_scores[name] = pr_auc

print('PR AUC per model:', pr_auc_scores)

Comparing PR AUC to ROC AUC often reveals different ordering of models when class imbalance is severe.

11. Alternative Feature Engineering: LBP with SVM

Local Binary Patterns (LBP) are a lightweight texture descriptor that can complement HOG or replace it when computational resources are extremely limited. The following example demonstrates how to extract LBP features from the same pre‑processed images, train an SVM, and evaluate its performance. This approach is useful for practitioners who need a very fast pipeline or are working on edge devices.

from skimage.feature import local_binary_pattern
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

Parameters for LBP
radius = 3
n_points = 8 * radius

def extract_lbp_features(images):
    # Compute LBP for each image; flatten the result
    lbp_features = [local_binary_pattern(img, n_points, radius, method='uniform')
                    .ravel() for img in images]
    return np.array(lbp_features)

Prepare feature matrix
X_train_lbp = extract_lbp_features(X_train.reshape(len(X_train), -1))
X_test_lbp  = extract_lbp_features(X_test.reshape(len(X_test), -1))

SVM pipeline
lbp_svm_pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('svc', SVC(kernel='rbf', C=5, gamma='scale', probability=True))
])

Cross‑validation
cv_scores_lbp = cross_val_score(lbp_svm_pipeline, X_train_lbp, y_train,
                                cv=skf, scoring='roc_auc')
print('5‑fold CV AUC (LBP‑SVM):', np.mean(cv_scores_lbp))

Test‑set evaluation
lbp_svm_pipeline.fit(X_train_lbp, y_train)
y_pred_lbp = lbp_svm_pipeline.predict_proba(X_test_lbp)[:,1]
auc_lbp = roc_auc_score(y_test, y_pred_lbp)
print('Test AUC (LBP‑SVM):', auc_lbp)

Comparing LBP‑SVM with HOG‑SVM and CNN, we summarise key metrics in the table below.

Common Pitfalls and Mitigation Strategies

When comparing predictive models, a subtle misstep can lead to an entirely misleading conclusion. The following section dissects the most frequent errors, explains why they occur, and supplies actionable countermeasures that can be implemented in a typical Python data‑science workflow. The prose is analytical yet accessible, aimed at readers who already possess a foundational understanding of machine learning but may have encountered unexpected results when benchmarking models.

1. Ignoring the Data Splitting Strategy

One of the most pervasive mistakes is to perform a single random split of the data into training and test sets, fit every model on that split, and then compare the test scores. Because the split is fixed, a model that happens to exploit idiosyncratic patterns in that particular slice may appear superior, while a genuinely better model is penalised by unlucky sampling.

The correct approach is to employ cross‑validation (CV) consistently across all models. CV ensures that each observation is used for both training and validation, reducing variance in performance estimates. However, CV should be applied after any feature engineering or preprocessing that depends on the training data; otherwise, the validation set leaks information about future data.

Implementation tip: Use Scikit‑learn’s cross_val_score or, for more complex pipelines, Pipeline with GridSearchCV. Below is a minimal example that demonstrates a proper CV workflow for two models.

import numpy as np
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

<h1>Dummy data</h1>
X = np.random.randn(1000, 20)
y = np.random.randint(0, 2, 1000)

<h1>Define CV scheme</h1>
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

<h1>Logistic regression pipeline</h1>
log_pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', LogisticRegression(max_iter=1000))
])

<h1>Random forest pipeline (no scaling needed but included for consistency)</h1>
rf_pipe = Pipeline([
    ('clf', RandomForestClassifier(n_estimators=200))
])

<h1>Evaluate both models</h1>
log_scores = cross_val_score(log_pipe, X, y, cv=cv, scoring='roc_auc')
rf_scores = cross_val_score(rf_pipe, X, y, cv=cv, scoring='roc_auc')

print("Logistic Regression AUC:", log_scores.mean())
print("Random Forest AUC:", rf_scores.mean())

Line‑by‑line explanation:

1. Import necessary modules.
2. Generate synthetic data for illustration.
3. Instantiate a StratifiedKFold to preserve class proportions across folds.
4. Build a pipeline for logistic regression that scales the features.
5. Build a pipeline for a random forest; scaling is optional but harmless.
6. Compute cross‑validated AUC scores for both pipelines.
7. Print the mean scores, which are robust estimates of generalisation performance.

Notice how each model is wrapped in a pipeline that encapsulates preprocessing steps, preventing data leakage. The CV scheme is shared, ensuring a fair comparison.

2. Overlooking Model Complexity and Overfitting

Comparisons that ignore the bias–variance trade‑off often favour overly complex models. A deep neural network or a random forest with many trees may achieve a slightly higher training accuracy but generalise poorly. Conversely, a simple linear model may exhibit higher bias but lower variance, leading to more reliable predictions on unseen data.

Mitigation strategies include:

• Use nested cross‑validation: Outer folds estimate performance, inner folds perform hyper‑parameter tuning. This prevents optimistic bias caused by tuning on the same data used for evaluation.
• Plot learning curves: Visualise training vs. validation error as a function of training set size or regularisation strength to detect overfitting early.
• Leverage regularisation: Add L1/L2 penalties or tree‑depth constraints to complex models.

Example code for nested CV with a support vector machine (SVM) and a gradient boosting machine (GBM) is shown below.

from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.svm import SVC
from sklearn.ensemble import GradientBoostingClassifier

<h1>Parameter grids</h1>
svm_params = {'C': [0.1, 1, 10], 'kernel': ['rbf', 'linear']}
gbm_params = {'n_estimators': [100, 200], 'learning_rate': [0.05, 0.1]}

<h1>Outer CV</h1>
outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

<h1>Inner CV</h1>
inner_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=0)

<h1>SVM with nested CV</h1>
svm_grid = GridSearchCV(SVC(), svm_params, cv=inner_cv, scoring='roc_auc')
svm_outer_scores = cross_val_score(svm_grid, X, y, cv=outer_cv, scoring='roc_auc')

<h1>GBM with nested CV</h1>
gbm_grid = GridSearchCV(GradientBoostingClassifier(), gbm_params, cv=inner_cv, scoring='roc_auc')
gbm_outer_scores = cross_val_score(gbm_grid, X, y, cv=outer_cv, scoring='roc_auc')

print("SVM nested CV AUC:", svm_outer_scores.mean())
print("GBM nested CV AUC:", gbm_outer_scores.mean())

Each model is tuned within the inner loop, and the outer loop provides an unbiased estimate of generalisation performance.

3. Neglecting Class Imbalance

When the target classes are highly imbalanced, accuracy becomes a misleading metric. A model that simply predicts the majority class can achieve deceptively high accuracy while offering no practical value.

Solutions include:

• Use metrics that are robust to imbalance: area under the precision–recall curve (AUPRC), balanced accuracy, or F1‑score.
• Resample the data: oversample the minority class (SMOTE), undersample the majority class, or use class‑weight parameters in learners.
• Apply cost‑sensitive learning: penalise misclassification of the minority class more heavily.

The following snippet demonstrates how to balance class weights for a logistic regression model and evaluate it using AUPRC.

from sklearn.metrics import average_precision_score
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

<h1>Train/test split</h1>
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)

<h1>Logistic regression with balanced class weights</h1>
clf = LogisticRegression(max_iter=1000, class_weight='balanced')
clf.fit(X_train, y_train)

<h1>Predict probabilities</h1>
probs = clf.predict_proba(X_test)[:, 1]

<h1>Compute AUPRC</h1>
auprc = average_precision_score(y_test, probs)
print("AUPRC:", auprc)

Here the class_weight='balanced' parameter automatically scales the loss inversely proportional to class frequencies, helping the model pay more attention to the minority class.

4. Failing to Conduct Statistical Significance Testing

It is tempting to declare one model superior simply because its mean cross‑validated score is higher. However, small differences may fall within the noise of sampling variability. Statistical tests such as the paired t‑test, Wilcoxon signed‑rank test, or McNemar’s test for classification can quantify whether observed differences are statistically significant.

Example: a paired t‑test comparing predictions from two models on the same test set.

from scipy import stats
import numpy as np

<h1>Assume y_true and two sets of predictions</h1>
y_true = np.random.randint(0, 2, 200)
pred1 = np.random.randint(0, 2, 200)
pred2 = np.random.randint(0, 2, 200)

<h1>Compute accuracy for each</h1>
acc1 = np.mean(pred1 == y_true)
acc2 = np.mean(pred2 == y_true)

<h1>Paired t-test on individual predictions</h1>
t_stat, p_value = stats.ttest_rel(pred1, pred2)
print(f"Accuracy Model 1: {acc1:.3f}, Model 2: {acc2:.3f}")
print(f"T-statistic: {t_stat:.3f}, P-value: {p_value:.3f}")

Interpreting the p‑value informs whether the performance difference is likely due to chance.

5. Over‑Simplifying Feature Importance Comparisons

Comparing feature importances across heterogeneous models (e.g., a tree‑based model versus a linear model) can be misleading because the scales and interpretations differ. Tree models report importance based on impurity reduction, whereas linear models provide coefficients that reflect marginal effects.

Mitigation:

• Standardise features before feeding them to linear models to make coefficients comparable.
• Use model‑agnostic importance measures such as SHAP values or permutation importance, which can be computed for any black‑box model.
• Present importances in a unified visualisation, normalising them to a common scale (e.g., 0–100).

Below is a concise example of computing permutation importance for two models and visualising the results side‑by‑side.

from sklearn.inspection import permutation_importance
import matplotlib.pyplot as plt

<h1>Fit models</h1>
log_pipe.fit(X_train, y_train)
rf_pipe.fit(X_train, y_train)

<h1>Permutation importance</h1>
log_perm = permutation_importance(log_pipe, X_test, y_test, n_repeats=10, random_state=0)
rf_perm = permutation_importance(rf_pipe, X_test, y_test, n_repeats=10, random_state=0)

<h1>Plot</h1>
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
axes[0].barh(range(X.shape[1]), log_perm.importances_mean)
axes[0].set_title('Logistic Regression')
axes[0].invert_yaxis()
axes[1].barh(range(X.shape[1]), rf_perm.importances_mean)
axes[1].set_title('Random Forest')
axes[1].invert_yaxis()
plt.tight_layout()
plt.show()

Permutation importance is model‑agnostic and directly comparable across learners.

6. Ignoring Model Calibration

A model may rank instances correctly (high AUC) but produce poorly calibrated probability estimates. In decision‑making contexts where probabilities are interpreted or used as inputs to other models, miscalibration can lead to suboptimal outcomes.

Strategies:

• Use calibration plots (Reliability Diagrams) to visualise the relationship between predicted probabilities and observed frequencies.
• Apply post‑hoc calibration techniques such as Platt scaling or isotonic regression.
• Choose algorithms that provide better calibration out‑of‑the‑box (e.g., logistic regression, naive Bayes) when probability accuracy is critical.

Example of isotonic calibration in Scikit‑learn:

from sklearn.calibration import CalibratedClassifierCV

<h1>Calibrate SVM</h1>
calibrated_svm = CalibratedClassifierCV(SVC(probability=True), cv=5, method='isotonic')
calibrated_svm.fit(X_train, y_train)

<h1>Evaluate calibrated probabilities</h1>
probs_cal = calibrated_svm.predict_proba(X_test)[:, 1]
<h1>Compare AUPRC before and after calibration</h1>

7. Relying Solely on a Single Metric

Each performance metric captures a different aspect of model behaviour. For instance, accuracy is insensitive to class distribution, while precision–recall is not. Relying on a single metric can mask deficiencies that other metrics would reveal.

Recommendation:

• Report a committee of metrics: accuracy, AUC‑ROC, AUPRC, F1‑score, and calibration error.
• Use a multi‑objective optimisation framework if possible, weighing metrics according to business priorities.
• Present trade‑off curves (e.g., precision–recall curve) to give stakeholders a more nuanced view.

8. Disregarding the Computational Cost

In production environments, model latency and resource consumption can be as important as predictive accuracy. A marginal improvement in accuracy may not justify a substantial increase in inference time or memory footprint.

Mitigation:

• Measure inference time on representative hardware.
• Employ model compression techniques (pruning, quantisation, knowledge distillation).
• Choose simpler models when the performance gap is negligible, especially for real‑time applications.

9. Over‑Generalising Results Across Domains

Performance observed on a particular dataset may not translate to other settings due to domain shift, feature drift, or changes in data collection protocols. Blindly extrapolating results can lead to costly deployment failures.

Best practices:

• Validate models on external datasets or through temporal validation if possible.
• Track data drift and retrain models when significant changes are detected.
• Document assumptions explicitly and assess their validity before deployment.

10. Under‑Documenting the Evaluation Process

Reproducibility is a cornerstone of scientific rigour. Without meticulous documentation of data splits, hyper‑parameter grids, random seeds, and evaluation protocols, it becomes impossible to audit or extend the comparison.

Recommendations:

• Capture the entire evaluation pipeline in code (preferably a Jupyter notebook or a script).
• Use version control for both code and data preprocessing steps.
• Store random seeds and environment details (Python version, library versions) in a manifest file.

By systematically addressing these pitfalls, practitioners can avoid the common traps that skew model comparison, ensuring that the conclusions drawn are robust, transparent, and actionable.

import time
from sklearn.metrics import mean_absolute_error
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split

Split data for a regression task
X_train, X_test, y_train, y_test = train_test_split(X, np.random.randn(1000), test_size=0.2, random_state=0)

Define two models with different complexities
small_gbr = GradientBoostingRegressor(n_estimators=50, max_depth=3, random_state=0)
large_gbr = GradientBoostingRegressor(n_estimators=200, max_depth=6, random_state=0)

Measure training and inference times
start = time.time()
small_gbr.fit(X_train, y_train)
train_time_small = time.time() - start

start = time.time()
y_pred_small = small_gbr.predict(X_test)
inference_time_small = time.time() - start
mae_small = mean_absolute_error(y_test, y_pred_small)

start = time.time()
large_gbr.fit(X_train, y_train)
train_time_large = time.time() - start

start = time.time()
y_pred_large = large_gbr.predict(X_test)
inference_time_large = time.time() - start
mae_large = mean_absolute_error(y_test, y_pred_large)

print(f"Small GBT: MAE={mae_small:.3f}, Train time={train_time_small:.3f}s, Inference time={inference_time_small:.3f}s")
print(f"Large GBT: MAE={mae_large:.3f}, Train time={train_time_large:.3f}s, Inference time={inference_time_large:.3f}s")

Further Reading

• Goodfellow, I., Bengio, Y., & Courville, A. (2016) — Deep Learning. A comprehensive overview of deep learning architectures and comparison with traditional machine learning models. Provides empirical evidence on when deep models outperform classical approaches.
• James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) — An Introduction to Statistical Learning. Explains performance metrics for classical ML models and contrasts them with neural network results, highlighting interpretability trade‑offs.
• LeCun, Y., Bengio, Y., & Hinton, G. (2015) — Deep Learning. Nature, 521(7553), 436‑444. Discusses benchmark comparisons across models and the factors influencing performance gaps.
• Pedregosa, F. et al. (2011) — Scikit‑learn: Machine Learning in Python. Journal of Machine Learning Research. Details the implementation of classical algorithms and evaluation pipelines used in model comparison studies.
• Kingma, D. P., & Ba, J. (2015) — Adam: A Method for Stochastic Optimization. ICLR. Provides optimisation insights that explain why deep learning models often require specialised training techniques compared to traditional ML.
• IEEE Standard for Machine Learning Evaluation (IEEE 2020) — Guidelines for model performance reporting. Outlines best practices for reporting and interpreting performance differences across competing models.