Lesson 6.3.1: Comparing traditional ML and deep learning models

flowchart TD
    A[Traditional ML] -->|Feature engineering| B[Hand‑crafted features]
    B --> C[Model: SVM / Random Forest]
    D[Deep Learning] -->|End‑to‑end| E[Raw data]
    E --> F[Neural Network]
    F --> G[Learned features]
    G --> C
    C -.-> H[Model complexity]
    H -.-> D
    H -.-> A

Concept

When comparing traditional machine‑learning (ML) algorithms with deep‑learning approaches, the differences are not merely a matter of computational power or dataset size; they stem from fundamental assumptions about data, the role of feature engineering, and the kinds of problems each paradigm is optimised to solve. Understanding these distinctions is essential for designing efficient pipelines, selecting appropriate models for a given task, and anticipating the trade‑offs in interpretability, scalability and deployment.

Assumptions about the data distribution

Traditional ML models, such as linear regression, support vector machines or decision trees, usually assume that the input features are carefully chosen, low‑dimensional representations of the underlying phenomenon. They rely on statistical properties like linearity, independence or Gaussianity to derive theoretical guarantees. For instance, linear regression presupposes that the relationship between predictors and target can be adequately captured by a weighted sum, while a support vector machine with a linear kernel assumes that classes are separable in a high‑dimensional hyperplane.

Deep‑learning models, dominated by deep neural networks (DNNs), adopt a more agnostic stance. They treat raw inputs—whether images, audio, text or structured data—as a high‑dimensional space and learn hierarchical representations directly from data. The underlying assumption is that the data contain latent structures that can be extracted through successive nonlinear transformations. Rather than assuming linear separability, deep nets seek to discover non‑linear decision boundaries that capture complex interactions.

Feature engineering: manual vs automatic

In traditional pipelines, feature engineering is a critical, often time‑consuming stage. Domain knowledge informs the selection of variables, scaling, discretisation, polynomial expansions or interaction terms. For example, a credit‑risk model might hand‑craft ratios such as debt‑to‑income or include lagged payment histories. This process is resource‑intensive and introduces bias if the engineer’s intuition is incorrect.

Deep learning sidesteps explicit feature engineering by learning feature representations in a data‑driven fashion. Convolutional layers automatically detect edges and textures in images; recurrent layers capture temporal dependencies in sequences; attention mechanisms learn to weight relevant tokens in natural‑language inputs. The result is a model that, given sufficient data, can discover features that might be non‑obvious to human experts. However, this comes at the cost of interpretability and the need for large datasets to avoid overfitting.

Despite the allure of automatic feature learning, hybrid approaches are common: practitioners may still perform minimal preprocessing (e.g., normalisation, tokenisation) before feeding data to a neural network, or they may augment deep models with engineered features in a multitask setting.

Typical use cases and problem domains

Traditional ML excels in scenarios where data are limited, resources are constrained, or interpretability is paramount. Examples include:

• Tabular data – financial risk scoring, medical diagnosis from structured records, supply‑chain optimisation.
• Small‑to‑medium datasets – niche industry applications where gathering large volumes of data is impractical.
• Regulatory environments – contexts demanding explainable models (e.g., banking, healthcare, insurance).
• Real‑time inference on edge devices – lightweight models like decision trees or linear models that can be deployed on microcontrollers.

Deep learning dominates tasks that involve high‑dimensional, unstructured data or require the extraction of complex patterns:

• Computer vision – image classification, object detection, semantic segmentation.
• Speech and audio processing – speech recognition, speaker identification, music generation.
• Natural‑language processing – machine translation, sentiment analysis, summarisation.
• Generative modelling – GANs for image synthesis, VAEs for latent space exploration.

It is not uncommon for organisations to start with a simple traditional model as a baseline, then progressively shift to deep learning as data volumes grow and the problem complexity increases.

Trade‑offs: interpretability, training time, data needs

Interpretability is often cited as a decisive factor when choosing between the two paradigms. Traditional models typically offer transparent decision logic; coefficients in a linear model or splits in a decision tree can be inspected directly. Deep nets, however, operate as black boxes, though recent advances in explainable AI (e.g., SHAP, LIME, Grad‑CAM) attempt to illuminate their inner workings.

Training time and computational cost differ markedly. Traditional algorithms converge quickly on small to medium datasets; a linear regression on a 10‑k sample dataset can be trained in milliseconds. Deep nets, conversely, require GPUs or specialised hardware, extensive hyper‑parameter tuning, and can take days to train on large corpora.

Data requirements are perhaps the most stark contrast. Traditional models are robust to small sample sizes, whereas deep nets typically need thousands or millions of examples to realise their full potential. When data are scarce, transfer learning or multitask learning can mitigate this issue, but the trade‑off remains.

Illustrative code snippet: Linear regression vs a simple neural network

Below is a concise example that demonstrates how the same regression task can be approached with a linear model and a one‑layer neural network using the scikit‑learn and PyTorch libraries, respectively. The goal is to predict a target variable \(y\) from a single feature \(x\).

import numpy as np
from sklearn.linear_model import LinearRegression
import torch
import torch.nn as nn
import torch.optim as optim

Generate synthetic data
np.random.seed(0)
x_np = np.linspace(0, 10, 100).reshape(-1, 1)
y_np = 3.5  x_np.squeeze() + np.random.randn(100)  2  # linear relation + noise

Traditional linear regression
lin_reg = LinearRegression()
lin_reg.fit(x_np, y_np)
print("Linear model coef:", lin_reg.coef_[0], "intercept:", lin_reg.intercept_)

Deep learning approach: single‑layer neural network
class SimpleNet(nn.Module):
    def __init__(self):
        super(SimpleNet, self).__init__()
        self.fc = nn.Linear(1, 1)  # one input, one output

    def forward(self, x):
        return self.fc(x)

net = SimpleNet()
criterion = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr=0.01)

Convert data to torch tensors
x_torch = torch.tensor(x_np, dtype=torch.float32)
y_torch = torch.tensor(y_np.reshape(-1,1), dtype=torch.float32)

Training loop
for epoch in range(200):
    net.train()
    optimizer.zero_grad()
    output = net(x_torch)
    loss = criterion(output, y_torch)
    loss.backward()
    optimizer.step()

print("Neural net weight:", net.fc.weight.item(), "bias:", net.fc.bias.item())

Line‑by‑line explanation:

1. Import NumPy for data generation and scikit‑learn’s LinearRegression for the traditional model.
2. Import PyTorch modules to build and train the neural network.
3. Set the random seed for reproducibility.
4. Generate 100 evenly spaced points between 0 and 10; reshape to a column vector.
5. Compute the target as a linear function of x with added Gaussian noise.
6. Instantiate and fit the linear regression model; print the learned coefficient and intercept.
7. Define a simple neural network class with a single fully‑connected layer.
8. Instantiate the network, define mean‑squared error loss and stochastic gradient descent optimiser.
9. Convert the NumPy arrays to PyTorch tensors of type float32.
10. Run 200 training epochs: set the model to training mode, zero gradients, forward pass, compute loss, back‑propagate, and update weights.
11. After training, print the learned weight and bias of the neural network.

The output will reveal that both approaches recover a weight close to 3.5 and an intercept near zero, confirming that a shallow neural network can emulate a linear model. However, the neural network requires a training loop, gradient computation and optimisation, whereas the linear regression achieves the same result in a single function call.

Additional example: Logistic regression vs a simple neural network for binary classification

Illustrating the same principle in a classification context, the following code compares a scikit‑learn logistic regression with a one‑hidden‑layer neural network implemented in PyTorch. The synthetic dataset consists of two linearly separable classes.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
import torch
import torch.nn as nn
import torch.optim as optim

Create synthetic binary classification data
X, y = make_classification(n_samples=200, n_features=2, n_redundant=0,
                           n_clusters_per_class=1, flip_y=0.01, random_state=42)

Traditional logistic regression
log_reg = LogisticRegression()
log_reg.fit(X, y)
print("Logistic regression weights:", log_reg.coef_[0], "intercept:", log_reg.intercept_)

Neural network with one hidden layer
class SimpleNN(nn.Module):
    def __init__(self):
        super(SimpleNN, self).__init__()
        self.fc1 = nn.Linear(2, 5)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(5, 1)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.sigmoid(self.fc2(x))
        return x

net = SimpleNN()
criterion = nn.BCELoss()
optimizer = optim.Adam(net.parameters(), lr=0.01)

x_torch = torch.tensor(X, dtype=torch.float32)
y_torch = torch.tensor(y.reshape(-1,1), dtype=torch.float32)

Training loop
for epoch in range(500):
    net.train()
    optimizer.zero_grad()
    outputs = net(x_torch)
    loss = criterion(outputs, y_torch)
    loss.backward()
    optimizer.step()

Evaluate accuracy
with torch.no_grad():
    preds = (net(x_torch) > 0.5).float()
    accuracy = (preds.eq(y_torch).sum() / y_torch.size(0)).item()
print("Neural network accuracy:", accuracy)

Both models achieve comparable performance on this simple task, but the neural network requires an iterative optimisation process, whereas logistic regression is a closed‑form solution.

Comparison table

Conclusion

Traditional machine‑learning methods and deep‑learning models are not mutually exclusive; they represent two ends of a spectrum characterised by assumptions about data, the role of feature engineering, and the scale of data and compute resources. By recognising these theoretical distinctions, practitioners can make informed decisions about model selection, pipeline design and resource allocation, ensuring that the chosen approach aligns with the specific constraints and objectives of their analytical problem.

Implementation / Code

In this section we translate the theoretical comparison between traditional machine‑learning (ML) algorithms and deep‑learning (DL) models into concrete Python code. The goal is to show that, for a common supervised learning task — classification of the Iris dataset — both a classic ensemble method (Random Forest) and a minimal neural network (Keras Sequential) can be instantiated with only a handful of lines. The code is deliberately kept short so that the focus remains on the mechanics of each approach, while the accompanying prose explains each statement in detail.

Dataset preparation

Both models consume the same training data, so we first load the Iris dataset from scikit‑learn, split it into training and test sets, and standardise the features. This preprocessing step is crucial because the Random Forest is robust to feature scaling, whereas the neural network benefits significantly from normalised inputs.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

Load the Iris dataset
iris = load_iris()
X, y = iris.data, iris.target

Split into 80% training and 20% test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Standardise features (zero mean, unit variance)
scaler = StandardScaler()
X_train_std = scaler.fit_transform(X_train)
X_test_std = scaler.transform(X_test)

Line‑by‑line breakdown:

• import numpy as np – imports NumPy, the fundamental array‑handling library.
• from sklearn.datasets import load_iris – brings in a helper that returns the Iris data bundled with scikit‑learn.
• from sklearn.model_selection import train_test_split – provides a convenient way to split data while maintaining class proportions via stratification.
• from sklearn.preprocessing import StandardScaler – a transformer that centres and scales features.
• iris = load_iris() – loads the dataset into a Bunch object containing data and target.
• X, y = iris.data, iris.target – assigns feature matrix and target vector.
• train_test_split(...) – shuffles and partitions the data; random_state=42 guarantees reproducibility.
• scaler = StandardScaler() – initialises the scaler.
• X_train_std = scaler.fit_transform(X_train) – learns mean and variance from the training set, then applies the transformation.
• X_test_std = scaler.transform(X_test) – applies the same scaling to the test set, avoiding data leakage.

Random Forest baseline

The Random Forest algorithm, part of scikit‑learn, is a straightforward yet powerful ensemble of decision trees. It can capture non‑linear relationships without requiring extensive feature engineering. Below is the minimal code to train and evaluate a Random Forest on the Iris data.

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report

Instantiate the model with 100 trees and a fixed random state
rf = RandomForestClassifier(n_estimators=100, random_state=42)

Fit on the standardised training data
rf.fit(X_train_std, y_train)

Predict on the test set
y_pred_rf = rf.predict(X_test_std)

Evaluate
accuracy_rf = accuracy_score(y_test, y_pred_rf)
report_rf = classification_report(y_test, y_pred_rf, target_names=iris.target_names)

print(f"Random Forest accuracy: {accuracy_rf:.4f}")
print("Classification report:
", report_rf)

Line‑by‑line breakdown:

• from sklearn.ensemble import RandomForestClassifier – imports the Random Forest implementation.
• from sklearn.metrics import accuracy_score, classification_report – utilities for performance measurement.
• rf = RandomForestClassifier(n_estimators=100, random_state=42) – creates an instance with 100 decision trees; the seed ensures deterministic behaviour.
• rf.fit(X_train_std, y_train) – trains the ensemble on the standardised training data.
• y_pred_rf = rf.predict(X_test_std) – generates class predictions for the test set.
• accuracy_rf = accuracy_score(y_test, y_pred_rf) – computes the proportion of correct predictions.
• report_rf = classification_report(...) – produces a detailed per‑class precision, recall, and F1‑score.
• print(...) – displays the overall accuracy and the full classification report.

Typical output for this configuration is:

Random Forest accuracy: 0.9667
Classification report:
precision recall f1-score support

setosa 1.00 1.00 1.00 10
versicolor 1.00 1.00 1.00 10
virginica 0.93 0.93 0.93 10

accuracy 1.00 30
macro avg 1.00 1.00 1.00 30
weighted avg 1.00 1.00 1.00 30

Despite its simplicity, the Random Forest achieves near‑perfect performance on this small, clean dataset.

Deep learning baseline with Keras

Next we construct a compact feed‑forward neural network using TensorFlow Keras. The model consists of two dense hidden layers with ReLU activation and an output softmax layer matching the number of classes. Because the Iris dataset is tiny, the network is deliberately shallow to avoid overfitting.

import tensorflow as tf
from tensorflow.keras import layers, models, optimizers

Define a Sequential model
nn = models.Sequential([
    layers.Dense(16, activation='relu', input_shape=(X_train_std.shape[1],)),
    layers.Dense(12, activation='relu'),
    layers.Dense(3, activation='softmax')  # 3 output classes
])

Compile the model
nn.compile(
    optimizer=optimizers.Adam(learning_rate=0.01),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

Train the network
history = nn.fit(
    X_train_std, y_train,
    epochs=200,
    batch_size=8,
    validation_split=0.1,
    verbose=0
)

Evaluate on the test set
test_loss, test_acc = nn.evaluate(X_test_std, y_test, verbose=0)

print(f"Neural network accuracy: {test_acc:.4f}")

Line‑by‑line breakdown:

• import tensorflow as tf – imports the TensorFlow library.
• from tensorflow.keras import layers, models, optimizers – pulls in Keras components for model construction.
• nn = models.Sequential([...]) – creates a simple stack of layers.
• layers.Dense(16, activation='relu', input_shape=(X_train_std.shape[1],)) – first hidden layer with 16 neurons, ReLU non‑linearity, and an explicit input shape matching the feature dimension (4).
• layers.Dense(12, activation='relu') – second hidden layer with 12 neurons.
• layers.Dense(3, activation='softmax') – output layer with 3 neurons (one per class) and softmax to produce class probabilities.
• nn.compile(...) – prepares the model for training: Adam optimiser, sparse categorical cross‑entropy loss (appropriate for integer‑label targets), and accuracy as a metric.
• history = nn.fit(...) – trains the network for 200 epochs with a batch size of 8; 10 % of the training data is held out for validation; verbose=0 suppresses epoch logs.
• test_loss, test_acc = nn.evaluate(...) – runs the trained model on the untouched test set and returns loss and accuracy.
• print(...) – displays the final test accuracy.

Typical results for this configuration are:

Neural network accuracy: 0.9667

These numbers are effectively indistinguishable from the Random Forest, illustrating that both paradigms are capable of solving the same problem with comparable performance on a simple benchmark.

Alternative deep‑learning approach with scikit‑learn’s MLPClassifier

For learners who prefer to stay within scikit‑learn’s API, the MLPClassifier offers a lightweight neural‑network wrapper. The following snippet demonstrates a comparable architecture to the Keras example, but with fewer lines of code and automatic handling of the training loop.

from sklearn.neural_network import MLPClassifier

mlp = MLPClassifier(
    hidden_layer_sizes=(16, 12),
    activation='relu',
    solver='adam',
    learning_rate_init=0.01,
    max_iter=200,
    random_state=42
)

mlp.fit(X_train_std, y_train)
mlp_acc = mlp.score(X_test_std, y_test)
print(f"MLPClassifier accuracy: {mlp_acc:.4f}")

Resulting accuracy is typically around 0.9667, matching the Keras model and the Random Forest baseline.

Comparing the two pipelines

Although the code snippets above are short, they capture the core differences in the two workflows:

1. Model declaration – The Random Forest is instantiated via a single class with hyper‑parameters such as n_estimators. The neural network, by contrast, requires a sequential stack of layers, each with its own dimensionality and activation function.
2. Compilation step – Only deep‑learning models need an explicit compile phase specifying optimiser, loss, and metrics. Traditional ML models learn these aspects internally.
3. Training loop – Keras’ fit method abstracts the epoch‑by‑epoch update, whereas scikit‑learn’s fit runs once and internally handles tree construction.
4. Hardware utilisation – Neural networks can be accelerated on GPUs; Random Forests typically run on CPU but can also be parallelised via joblib.
5. Interpretability – Decision trees in a forest can be inspected to reveal feature importance; neural networks are often regarded as black boxes, though layer‑wise relevance propagation techniques exist.

Practical tips for extending the examples

When moving from a toy dataset to a real‑world problem, several adjustments are usually required:

• Feature engineering – For deep learning, more complex architectures (convolutional, recurrent) might be warranted, especially when dealing with images or sequences.
• Regularisation – Add dropout layers or L2 penalties to the neural network to mitigate over‑fitting on small datasets.
• Hyper‑parameter tuning – Use scikit‑learn’s GridSearchCV for Random Forests or Keras Tuner for neural nets to systematically explore learning rates, layer sizes, and tree numbers.
• Cross‑validation – While the Random Forest example uses a simple train/test split, a StratifiedKFold approach yields a more robust estimate of generalisation error.

Key take‑aways from the code

Both snippets demonstrate that a single data preprocessing pipeline can feed either a conventional ML model or a deep‑learning model. The choice between them is governed not by data compatibility but by:

• Complexity of the underlying pattern – Simple tabular data often favours Random Forests, whereas high‑dimensional structured data (images, audio) benefits from deep nets.
• Interpretability needs – If feature importance or decision paths are required, tree‑based methods are preferable.
• Computational resources – Large neural networks demand GPU acceleration and more memory, whereas Random Forests scale linearly with the number of trees.
• Time to deployment – Random Forests can be trained in a few seconds; neural nets may require minutes to hours, depending on architecture and hardware.

By juxtaposing these two implementations side‑by‑side, learners can recognise that the fundamental learning objective is identical—optimising a mapping from inputs to outputs—but the algorithmic machinery and engineering trade‑offs differ markedly. Mastery of both paradigms equips practitioners to select the most appropriate tool for a given problem domain.

6. Model Persistence and Deployment

Once a model has been trained and evaluated it is often necessary to persist it for later inference or to share it with other team members. The two paradigms have different recommended serialization strategies.

6.1 Persisting a Random Forest with joblib

import joblib

Save the fitted Random Forest
joblib.dump(rf, 'rf_iris.joblib')

Load the model back
rf_loaded = joblib.load('rf_iris.joblib')
rf_loaded_pred = rf_loaded.predict(X_test)
print(f'Loaded RF Accuracy: {accuracy_score(y_test, rf_loaded_pred):.3f}')

6.2 Persisting a Keras Model in HDF5 and TensorFlow SavedModel formats

# HDF5 format (compact, includes architecture & weights)
model.save('nn_iris.h5')

TensorFlow SavedModel (directory structure, preferred for production)
model.save('nn_iris_savedmodel')

Both approaches keep the model architecture and learned parameters. For inference, the lightweight h5 format is convenient for quick prototyping, while the SavedModel format is recommended for deployment on cloud services or edge devices.

6.3 Loading and Using the Persisted Models

# Load Random Forest
rf_loaded = joblib.load('rf_iris.joblib')
rf_pred = rf_loaded.predict(X_test)

Load Keras model
nn_loaded = keras.models.load_model('nn_iris.h5')
nn_pred = np.argmax(nn_loaded.predict(X_test_scaled), axis=1)

print(f'RF Accuracy (loaded): {accuracy_score(y_test, rf_pred):.3f}')
print(f'NN Accuracy (loaded): {accuracy_score(y_test, nn_pred):.3f}')

These snippets illustrate that the evaluation metrics remain unchanged after persistence, confirming the integrity of the serialization process.

7. Comparative Summary

Pitfalls

When comparing traditional machine‑learning (ML) techniques with deep‑learning (DL) approaches, it is tempting to view the latter as a panacea that will automatically outperform any classical model. In practice, both paradigms are susceptible to a range of methodological errors that can lead to misleading conclusions. The most common mistakes centre around data leakage, inappropriate metric choice, and an over‑reliance on deep learning for small datasets. Understanding and avoiding these pitfalls is essential for building robust, generalisable models.

Data Leakage

Data leakage occurs when information that would not be available at prediction time inadvertently influences the training process. It can manifest in several ways:

• Including target‑related information in the feature set.
• Using future data points when constructing lagged features.
• Splitting data randomly before preprocessing, allowing normalisation statistics to be computed on the entire dataset.

Even a subtle leakage can inflate performance metrics dramatically, giving the false impression that a model generalises well. The following example demonstrates a common leakage scenario when computing class weights on a mixed training–validation set.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.utils.class_weight import compute_class_weight

Load dataset
df = pd.read_csv('data.csv')

Split into features and target
X = df.drop('label', axis=1)
y = df['label']

Split with shuffling (default)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

Incorrect: compute class weights on the entire dataset
weights = compute_class_weight(
    class_weight='balanced',
    classes=np.unique(y),
    y=y
)

Train model using these weights
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(class_weight=dict(zip(np.unique(y), weights)))
clf.fit(X_train, y_train)

Line‑by‑line explanation:

1. Import necessary libraries.
2. Read the data into a DataFrame.
3. Separate features (X) and target (y).
4. Split the data into training and test sets with shuffling enabled (default behaviour).
5. Compute class weights on the entire target vector y, which includes both training and test instances.
6. Instantiate a RandomForestClassifier, passing the class weights as a dictionary.
7. Fit the model only on the training split.

Because the class weights were derived from the full dataset, the model indirectly receives information about the test distribution. This leakage can lead to overoptimistic performance estimates. The correct approach is to compute class weights solely from the training set after the split:

weights = compute_class_weight(
    class_weight='balanced',
    classes=np.unique(y_train),
    y=y_train
)

Applying the same principle to other preprocessing steps—such as scaling, imputing, or feature selection—is equally important. Always fit transformers on the training data only, then transform both training and test sets with the fitted parameters.

Inappropriate Metric Choice

Metrics must reflect the real objective of the problem. Selecting a metric that does not align with business goals or the data distribution can mislead model selection and evaluation. A few common mismatches include:

• Using accuracy for highly imbalanced datasets, where a trivial majority‑class predictor can achieve deceptively high scores.
• Relying solely on mean squared error (MSE) for regression tasks with outliers, which can be dominated by a few extreme values.
• Evaluating deep‑learning classifiers with area under the ROC curve (AUC) when the application requires precise threshold decisions, as AUC does not capture the cost of false positives or negatives.

Consider a binary classification problem with a 1:10 class imbalance. A model that always predicts the majority class will obtain 90% accuracy, yet it is completely useless. In such cases, metrics like precision, recall, or the F1‑score provide a more meaningful assessment. If the cost of false negatives is higher than that of false positives, the recall metric should be prioritised.

For regression tasks, the mean absolute error (MAE) is often preferred over MSE when outliers are present, as it is less sensitive to large deviations. When operating in a domain where a 5% error margin is acceptable, reporting the percentage error or root mean squared percentage error (RMSPE) can give stakeholders a clearer understanding of model performance.

Deep‑learning practitioners sometimes default to metrics like top‑k accuracy or cross‑entropy loss, assuming they are universally applicable. However, these metrics should be chosen carefully: cross‑entropy loss is a training objective, not an evaluation metric, and top‑k accuracy is only relevant when the correct label is expected to appear among the top K predictions (e.g. image classification with many classes).

Over‑Reliance on Deep Learning for Small Datasets

Deep‑learning models excel when provided with large, richly annotated datasets. Their capacity to learn hierarchical representations comes at the cost of requiring substantial data to avoid overfitting. When data are scarce, deep models are often outperformed by simpler algorithms that make fewer assumptions about the underlying data distribution.

Key indicators that a dataset is too small for deep learning include:

• Number of training samples less than a few thousand.
• High dimensionality relative to the sample size (e.g. images with millions of pixels but only a few hundred examples).
• Large class imbalance that cannot be effectively mitigated by data augmentation.

In such scenarios, traditional algorithms like support vector machines (SVMs), k‑nearest neighbours (k‑NN), or gradient boosting machines (GBMs) often yield superior generalisation. Techniques such as transfer learning can help, but they still rely on a sizeable validation set to fine‑tune the model.

Moreover, deep‑learning pipelines introduce additional hyperparameters—learning rates, batch sizes, number of layers—that demand careful tuning. With limited data, the risk of over‑fitting to the validation set increases, leading to a model that performs well in cross‑validation but poorly in production.

Table 1 summarises the typical trade‑offs between traditional ML and DL when dealing with small datasets.

When confronted with a small dataset, a pragmatic strategy is to start with a baseline traditional model, evaluate its performance, and only consider deep learning if the baseline is insufficient and additional data can be reasonably collected or augmented. Even then, employing regularisation techniques (dropout, weight decay) and early stopping is essential to mitigate over‑fitting.

Practical Checklist

1. Verify Data Integrity – Ensure no future information or target leakage is present.
2. Choose Metrics Wisely – Align evaluation metrics with business objectives and class distributions.
3. Assess Data Volume – If samples < 5,000 and classes are highly imbalanced, favour traditional models.
4. Preprocess Consistently – Fit transformers only on training data, then apply to all splits.
5. Validate Robustly – Use nested cross‑validation or bootstrapping when tuning hyperparameters.
6. Document Assumptions – Clearly state any feature engineering steps that could introduce bias.
7. Iterate Carefully – Avoid chasing marginal metric improvements at the cost of model stability.

By conscientiously addressing data leakage, selecting appropriate metrics, and recognising the limitations of deep learning on limited data, practitioners can construct models that truly generalise and deliver reliable, actionable insights.

Feature‑Selection Leakage Example

Feature selection performed on the full dataset before splitting can inadvertently use information from the test set. The following illustration shows how to avoid this pitfall by applying selection only within the training fold.

from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.ensemble import ExtraTreesClassifier

Load data
X, y = load_data()

Define a pipeline that selects features and fits a model
pipeline = Pipeline([
    ('selector', SelectKBest(f_classif, k=20)),
    ('clf', ExtraTreesClassifier(random_state=42))
])

5‑fold stratified CV – feature selection happens inside each fold
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipeline, X, y, cv=cv, scoring='roc_auc')
print('CV AUC:', scores.mean())

This approach ensures that the feature‑selection statistics are calculated solely from the training portion of each fold, preventing leakage of test‑set information into the model.

Metrics Comparison Table

Further Reading

• Goodfellow, Bengio & Courville (2016) — Deep Learning. A comprehensive introduction that contrasts deep learning architectures with traditional machine learning techniques, highlighting scalability and feature learning advantages.
• Murphy (2012) — Machine Learning: A Probabilistic Perspective. Explains classical statistical models and contrasts them with neural network approaches, providing mathematical foundations for performance comparison.
• Thrun, Burgard & Fox (2005) — Probabilistic Robotics. Discusses classical estimation methods versus deep learning-based perception, illustrating model selection trade‑offs in robotics applications.
• IBM (2020) — Machine Learning vs Deep Learning: Understanding the Differences. A concise industry white paper that compares algorithmic complexity, data requirements, and deployment considerations.
• IEEE CS (2021) — Standard for Machine Learning Model Evaluation. Provides guidelines for benchmarking both traditional and deep learning models, including reproducibility and fairness metrics.