Lesson 5.3.2: Hyperparameter tuning basics

flowchart TD
    A[Start] --> B[Define Hyperparameters]
    B --> C[Set Search Space]
    C --> D[Choose Search Strategy]
    D --> E[Run Experiments]
    E --> F[Evaluate Metrics]
    F --> G[Select Best Hyperparameters]
    G --> H[End]

Illustrative Code Example (continued)

Below is an additional Python snippet that demonstrates hyperparameter tuning for a Support Vector Machine using scikit‑learn’s GridSearchCV. The example focuses on the kernel type and regularisation parameter C, common hyperparameters for SVMs.

import numpy as np
from sklearn import datasets
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.metrics import accuracy_score

Load a simple dataset
X, y = datasets.load_iris(return_X_y=True)

Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Define the hyperparameter grid
param_grid = {
    'C': [0.1, 1, 10, 100],           # Regularisation strength
    'kernel': ['linear', 'rbf', 'poly'],  # Kernel type
    'gamma': ['scale', 'auto']        # Kernel coefficient for rbf/poly
}

Initialise the classifier
svc = SVC()

Set up GridSearchCV
grid_search = GridSearchCV(estimator=svc, param_grid=param_grid,
                           scoring='accuracy', cv=5, n_jobs=-1)

Fit the model
grid_search.fit(X_train, y_train)

Best hyperparameters
print("Best parameters:", grid_search.best_params_)

Evaluate on the held‑out test set
best_model = grid_search.best_estimator_
y_pred = best_model.predict(X_test)
print("Test accuracy:", accuracy_score(y_test, y_pred))

In this example, the search space covers four values of C, three kernel choices, and two options for gamma, resulting in 24 combinations evaluated via 5‑fold cross‑validation.

Comparison of Hyperparameter Search Strategies

Implementation/Code: Grid and Random Search in Scikit‑Learn

When we talk about hyperparameter tuning in machine learning, the distance between a well‑fitted model and a mediocre one is often governed by how skilfully we search the parameter space. Two of the most commonly used search strategies in scikit‑learn are GridSearchCV and RandomizedSearchCV. Both are wrappers that automatically cross‑validate a model over a predefined grid or random distribution of parameters, returning the best‑performing combination according to a specified scoring metric. In this section we will walk through an explicit implementation of these two methods for a Random Forest classifier, analysing each line of code to illuminate the mechanics behind the scenes.

Preparing the data and baseline model

Before we dive into hyperparameter search we need a dataset to work with. For the sake of demonstration we’ll use the classic Iris dataset, which is small enough to keep the runtime negligible yet rich enough to expose the value of tuning. We also define a baseline Random Forest model with default hyperparameters so that we can gauge the impact of optimisation.

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report

Load the Iris data
iris = load_iris()
X, y = iris.data, iris.target

Split into train and test to evaluate final performance
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Baseline model
baseline_clf = RandomForestClassifier(random_state=42)
baseline_clf.fit(X_train, y_train)
baseline_pred = baseline_clf.predict(X_test)
print(f"Baseline accuracy: {accuracy_score(y_test, baseline_pred):.4f}")
print(classification_report(y_test, baseline_pred))

Line by line:

1. import numpy as np and import pandas as pd are standard imports for numerical and tabular data handling, though pandas isn’t directly used here.
2. We import load_iris to pull the dataset, and the three cross‑validation utilities from sklearn.model_selection.
3. RandomForestClassifier is the algorithm to tune, and accuracy_score and classification_report are metrics for quick evaluation.
4. We load the data into X (features) and y (labels).
5. The train_test_split call reserves 20 % of the data for a final hold‑out test; random_state=42 ensures reproducibility and stratify=y keeps class proportions balanced.
6. We instantiate a Random Forest with default settings but seed the random number generator for deterministic behaviour.
7. The model is trained on the training split, predictions are made on the test split, and we print the baseline accuracy and detailed classification report.

Running this code will typically yield an accuracy of around 0.9667, reflecting the ease of the Iris classification task. Nevertheless, this baseline serves as a reference point against which we will compare the tuned models.

Defining the hyperparameter search space

Both GridSearchCV and RandomizedSearchCV rely on a dictionary that maps hyperparameter names (as they appear in the estimator’s constructor) to a list or distribution of values to explore. The key is to strike a balance between breadth (covering enough possibilities) and depth (keeping the search tractable).

param_grid = {
    'n_estimators': [100, 200, 400],
    'max_depth': [None, 5, 10, 20],
    'min_samples_split': [2, 5, 10],
    'min_samples_leaf': [1, 2, 4],
    'bootstrap': [True, False]
}

Explanation of the entries:

• n_estimators – the number of trees in the forest. A higher count generally improves stability but increases training time.
• max_depth – restricts how deep each tree can grow. None means grow until all leaves are pure or until min_samples_split stops further splitting.
• min_samples_split – the minimum number of samples required to attempt a split at an internal node.
• min_samples_leaf – the minimum number of samples that must be present in a leaf node.
• bootstrap – whether bootstrap samples are used when building trees (True is the default).

For a Randomised search we might want to sample from continuous distributions, e.g. scipy.stats.randint or scipy.stats.loguniform. In this example we keep the grid simple for clarity.

GridSearchCV implementation

GridSearchCV exhaustively tries every combination in param_grid, which can be computationally expensive. We will use 5‑fold cross‑validation and optimise for accuracy.

grid_search = GridSearchCV(
    estimator=RandomForestClassifier(random_state=42),
    param_grid=param_grid,
    scoring='accuracy',
    cv=5,
    n_jobs=-1,
    verbose=1
)

grid_search.fit(X_train, y_train)

print("Best parameters found by GridSearchCV:")
print(grid_search.best_params_)
print(f"Best CV accuracy: {grid_search.best_score_:.4f}")

Evaluate on hold‑out test set
grid_best = grid_search.best_estimator_
grid_pred = grid_best.predict(X_test)
print(f"Test accuracy (GridSearchCV): {accuracy_score(y_test, grid_pred):.4f}")
print(classification_report(y_test, grid_pred))

Line‑by‑line walk‑through:

1. Instantiate GridSearchCV with the same base estimator. scoring='accuracy' tells it what metric to optimise.
2. cv=5 sets five‑fold cross‑validation; the dataset is split into five parts, each time training on four parts and validating on the remaining part.
3. n_jobs=-1 leverages all CPU cores to parallelise the grid; this is optional but often speeds up execution.
4. verbose=1 prints progress for each fold, useful for monitoring.
5. The fit call triggers the search. Internally, GridSearchCV iterates over every key‑value combination, builds a new Random Forest, cross‑validates it, and records the mean accuracy.
6. best_params_ reveals the hyperparameter set that achieved the highest mean accuracy across folds.
7. best_score_ is that mean accuracy.
8. We retrieve the estimator that achieved this performance via best_estimator_ and evaluate it on the untouched test set, providing an unbiased estimate of generalisation performance.

Typical output for the Iris dataset might be something like:

Best parameters found by GridSearchCV:
{'bootstrap': True, 'max_depth': 10, 'min_samples_leaf': 1, 'min_samples_split': 2, 'n_estimators': 400}
Best CV accuracy: 0.9778
Test accuracy (GridSearchCV): 0.9833

Notice how the tuned model slightly outperforms the baseline, demonstrating the benefit of systematic exploration.

RandomizedSearchCV implementation

When the parameter space is large or continuous, exhaustive grid search becomes impractical. RandomisedSearchCV samples a fixed number of parameter settings from the defined distributions, offering a more efficient search with comparable performance. We will perform 20 iterations of random sampling.

from scipy.stats import randint, loguniform

param_distributions = {
    'n_estimators': randint(100, 501),          # integers between 100 and 500
    'max_depth': [None] + list(range(5, 21)),   # None or 5–20
    'min_samples_split': randint(2, 11),        # 2–10
    'min_samples_leaf': randint(1, 5),          # 1–4
    'bootstrap': [True, False]
}

random_search = RandomizedSearchCV(
    estimator=RandomForestClassifier(random_state=42),
    param_distributions=param_distributions,
    n_iter=20,
    scoring='accuracy',
    cv=5,
    random_state=42,
    n_jobs=-1,
    verbose=1
)

random_search.fit(X_train, y_train)

print("Best parameters found by RandomizedSearchCV:")
print(random_search.best_params_)
print(f"Best CV accuracy: {random_search.best_score_:.4f}")

Evaluate on hold‑out test set
rand_best = random_search.best_estimator_
rand_pred = rand_best.predict(X_test)
print(f"Test accuracy (RandomizedSearchCV): {accuracy_score(y_test, rand_pred):.4f}")
print(classification_report(y_test, rand_pred))

Key details:

• We use scipy.stats.randint to generate discrete uniform distributions for integer parameters.
• For max_depth we combine None with a list of integers 5 to 20 to mimic the grid’s depth control.
• The n_iter=20 argument tells RandomisedSearchCV to evaluate twenty random combinations. Increasing this number improves the chance of finding a near‑optimal set but also raises runtime.
• Setting random_state=42 ensures that the random sampling is reproducible.

Typical output may read:

Best parameters found by RandomizedSearchCV:
{'bootstrap': True, 'max_depth': 10, 'min_samples_leaf': 1, 'min_samples_split': 2, 'n_estimators': 400}
Best CV accuracy: 0.9789
Test accuracy (RandomizedSearchCV): 0.9833

In this case, RandomisedSearchCV converged on the same hyperparameter set as GridSearchCV, illustrating that a modest number of random samples can be enough for a small dataset. On larger, more complex problems, RandomisedSearchCV often finds competitive solutions with far less computational effort.

Comparing the two search strategies

Both strategies aim to maximise the chosen metric by exploring the hyperparameter space. Their differences can be summarised in the following table:

In practice, a common workflow is to perform a coarse RandomisedSearchCV to identify promising regions, then refine with a focused GridSearchCV. Alternatively, one can use Bayesian optimisation libraries such as scikit-optimise or Optuna for more sophisticated search strategies.

Practical tips for effective hyperparameter tuning

1. Start simple: Initialise with a modest grid or distribution that captures the rough scale of each parameter.
2. Use cross‑validation wisely: For small datasets, use nested cross‑validation to guard against overfitting during tuning.
3. Leverage parallelism: Set n_jobs=-1 unless you are on a shared cluster with limited resources.
4. Monitor runtime: Keep an eye on the combinatorial explosion; a grid of 5 values each for 5 parameters yields 3,125 models.
5. Inspect intermediate results: With verbose=2 you can see the mean score for each parameter set, helping you spot anomalies.
6. Document everything: Store the best_params_ and the corresponding cross‑validation score for reproducibility.

By incorporating these practices, you ensure that your hyperparameter search is both rigorous and efficient, ultimately leading to models that generalise better to unseen data.

Example: Using a Pipeline with Hyperparameter Search

When preprocessing steps are involved, it is often convenient to embed them inside a Pipeline so that the entire workflow is tuned end‑to‑end. The following example shows how to combine a scaler, a dimensionality‑reduction step, and a Random Forest, then search for the best combination of preprocessing and model parameters.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('pca', PCA()),
    ('rf', RandomForestClassifier(random_state=42))
])

param_grid_pipe = {
    'pca__n_components': [2, 3, 4, 5],
    'rf__n_estimators': [100, 200],
    'rf__max_depth': [None, 10],
    'rf__min_samples_split': [2, 5]
}

grid_pipe = GridSearchCV(
    estimator=pipeline,
    param_grid=param_grid_pipe,
    scoring='accuracy',
    cv=5,
    n_jobs=-1,
    verbose=1
)

grid_pipe.fit(X_train, y_train)
print("Best pipeline parameters:")
print(grid_pipe.best_params_)
print(f"Best CV accuracy: {grid_pipe.best_score_:.4f}")

Evaluate on hold‑out test set
pipe_best = grid_pipe.best_estimator_
pipe_pred = pipe_best.predict(X_test)
print(f"Test accuracy (Pipeline + GridSearchCV): {accuracy_score(y_test, pipe_pred):.4f}")

Worked Example: Optimising an SVM

In this section we will walk through a concrete example of tuning the two most influential hyper‑parameters of a Support Vector Machine (SVM) with a radial basis function (RBF) kernel: the regularisation parameter C and the kernel width gamma. We will use a toy dataset, Wine, which is part of the scikit‑learn library, and perform a grid search with 5‑fold cross‑validation. By analysing the results we will see how the choice of C and gamma affects model bias, variance and overall performance.

Dataset preparation

The Wine dataset contains 178 samples described by 13 physicochemical features and labelled with one of three wine cultivars. For illustration we will treat it as a binary classification task by selecting only two classes (class 0 and class 1). This keeps the problem simple while still exposing the typical trade‑offs involved in hyper‑parameter tuning.

import numpy as np
from sklearn import datasets
from sklearn.model_selection import train_test_split

Load the full dataset
wine = datasets.load_wine()
X, y = wine.data, wine.target

Keep only classes 0 and 1
mask = y < 2
X, y = X[mask], y[mask]

Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Line‑by‑line explanation:

1. import numpy as np – imports NumPy for array handling (not strictly required for the following steps but a common import).
2. from sklearn import datasets – brings in the built‑in dataset loader.
3. from sklearn.model_selection import train_test_split – function to split data into training and test subsets.
4. wine = datasets.load_wine() – loads the dataset into a Bunch object containing data, target, feature names, etc.
5. X, y = wine.data, wine.target – extracts the feature matrix and target vector.
6. mask = y < 2 – creates a Boolean mask that is true for classes 0 and 1.
7. X, y = X[mask], y[mask] – filters the dataset to keep only the two chosen classes.
8. X_train, X_test, y_train, y_test = train_test_split(...) – splits the data, reserving 20 % for final evaluation. random_state=42 ensures reproducibility, and stratify=y keeps the class proportions balanced in both splits.

Baseline SVM model

Before tuning, we train a baseline SVM with the default hyper‑parameters (C=1.0, gamma='scale'). This gives us a reference point.

from sklearn.svm import SVC
from sklearn.metrics import accuracy_score

baseline_clf = SVC()
baseline_clf.fit(X_train, y_train)
baseline_pred = baseline_clf.predict(X_test)
baseline_acc = accuracy_score(y_test, baseline_pred)
print(f"Baseline accuracy: {baseline_acc:.3f}")

Running this code typically yields an accuracy of ~0.9 on the test set, indicating that the default settings already capture most of the structure in the data. However, there is room for improvement, especially if we aim to maximise performance for a production system.

Defining the hyper‑parameter grid

We construct a grid over which to search. For C we use a logarithmic scale from 0.01 to 100, and for gamma we use a similar logarithmic spread from 0.0001 to 1. This captures both very regularised (high bias) and very flexible (high variance) models.

param_grid = {
    'C': np.logspace(-2, 2, 5),          # [0.01, 0.1, 1, 10, 100]
    'gamma': np.logspace(-4, 0, 5)       # [0.0001, 0.001, 0.01, 0.1, 1]
}

Using np.logspace ensures that we cover several orders of magnitude, which is important for SVMs because the effect of C and gamma is highly non‑linear.

Cross‑validation procedure

We employ 5‑fold cross‑validation to estimate the generalisation error for each hyper‑parameter combination. The GridSearchCV class automates this process and returns the best parameters based on mean cross‑validated accuracy.

from sklearn.model_selection import GridSearchCV

grid_clf = GridSearchCV(
    estimator=SVC(kernel='rbf'),
    param_grid=param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1
)
grid_clf.fit(X_train, y_train)
print("Best hyper‑parameters:", grid_clf.best_params_)
print("Best CV accuracy: {:.3f}".format(grid_clf.best_score_))

Key points in the code:

• estimator=SVC(kernel='rbf') – sets the model type and kernel.
• param_grid=param_grid – supplies the grid defined earlier.
• cv=5 – specifies 5‑fold cross‑validation.
• scoring='accuracy' – metric used to evaluate each fold.
• n_jobs=-1 – utilises all available CPU cores for speed.
• After fitting, best_params_ holds the optimal C and gamma, and best_score_ is the mean accuracy across the folds.

Interpreting the results

Suppose the grid search returns C=10 and gamma=0.01 with a best CV accuracy of 0.975. We can visualise the performance surface to understand the sensitivity of the model to these parameters.

import matplotlib.pyplot as plt
import itertools

scores = grid_clf.cv_results_['mean_test_score'].reshape(
    len(param_grid['C']), len(param_grid['gamma'])
)

C_vals, gamma_vals = np.meshgrid(param_grid['C'], param_grid['gamma'], indexing='ij')
plt.figure(figsize=(8,6))
contour = plt.contourf(gamma_vals, C_vals, scores, levels=20, cmap='viridis')
plt.colorbar(contour, label='CV Accuracy')
plt.xlabel('gamma')
plt.ylabel('C')
plt.title('Hyper‑parameter search landscape')
plt.show()

The contour plot typically shows a band of high accuracy values. Values of C that are too small (e.g. 0.01) lead to under‑fitting: the decision boundary is overly smooth and misclassifies many points. Conversely, very large C values (e.g. 100) may over‑fit the training data, causing a reduction in cross‑validated accuracy. Similarly, a very small gamma (0.0001) makes the kernel extremely wide, yielding a linear‑looking decision surface, while a very large gamma (1) can make the boundary too wiggly, again risking over‑fitting.

Final model evaluation

Using the best parameters found, we retrain on the entire training set and evaluate on the held‑out test set to obtain an unbiased estimate of performance.

best_clf = SVC(kernel='rbf', C=grid_clf.best_params_['C'],
               gamma=grid_clf.best_params_['gamma'])
best_clf.fit(X_train, y_train)
test_pred = best_clf.predict(X_test)
test_acc = accuracy_score(y_test, test_pred)
print(f"Test accuracy with tuned SVM: {test_acc:.3f}")

In most runs this yields a test accuracy of 0.933 or higher, an improvement over the baseline. The result demonstrates the practical benefit of systematic hyper‑parameter optimisation.

Practical tips for hyper‑parameter tuning

• Start with a coarse grid. A wide range of values at low resolution lets you identify promising regions quickly.
• Refine with a finer grid. Once a region of high performance is located, use a denser grid around those values to fine‑tune.
• Use cross‑validation wisely. For small datasets, leave‑one‑out CV may be appropriate; for larger datasets, k‑fold with k=5–10 is standard.
• Consider computational cost. Grid search scales exponentially with the number of parameters; for many parameters, random search or Bayesian optimisation can be more efficient.
• Beware of data leakage. Always perform hyper‑parameter tuning within the training set. The test set should remain untouched until final evaluation.

Concluding observations

Through this worked example we have seen how C and gamma govern the bias–variance trade‑off in an RBF‑kernel SVM. By employing systematic cross‑validated grid search, we can discover combinations that yield a significant performance boost over default settings. The same methodology applies to other models and datasets, making hyper‑parameter optimisation a cornerstone of practical machine‑learning pipelines.

# Example: Using RandomisedSearchCV for a broader search space
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import expon, loguniform

random_param_grid = {
    'C': loguniform(1e-3, 1e3),          # wide range for C
    'gamma': loguniform(1e-4, 1)         # wide range for gamma
}

random_search = RandomizedSearchCV(
    estimator=SVC(kernel='rbf'),
    param_distributions=random_param_grid,
    n_iter=50,                           # number of random combinations to try
    cv=5,
    scoring='accuracy',
    random_state=42,
    n_jobs=-1
)

random_search.fit(X_train, y_train)
print("Best hyper‑parameters (randomised):", random_search.best_params_)
print("Best CV accuracy (randomised): {:.3f}".format(random_search.best_score_))

Pitfalls: Common Mistakes in Hyperparameter Tuning

When you begin to explore the vast landscape of hyperparameter optimisation, it is tempting to treat the process as a mechanical sweep of search space. In practice, however, the journey is fraught with subtle traps that can erode model performance, inflate variance, and ultimately erode trust in the results. This section dissects three of the most pernicious pitfalls: data leakage, over‑fitting to the validation set, and the curse of dimensionality.

Data Leakage: The Silent Assassin

Data leakage occurs when information that would not be available at prediction time leaks into the training or validation process. Even a single leaked data point can give a model an unfair advantage, leading to deceptively high validation scores that fail to generalise.

• Feature engineering after splitting: Creating new features from the entire dataset before partitioning allows future data to influence feature creation.
• Target leakage: Including a feature that is directly derived from the target variable (e.g. a lagged target in time‑series) gives the model access to information it will not have in real deployments.
• Cross‑validation mis‑use: Performing preprocessing that depends on the entire dataset (e.g. scaling or encoding) before cross‑validation will cause leakage across folds.

To guard against leakage, always perform preprocessing inside each training fold. The following Python snippet demonstrates a correct pipeline that avoids leakage by fitting the scaler only on the training data within each fold.

from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
import numpy as np

Synthetic data
X = np.random.randn(100, 5)
y = np.random.randint(0, 2, 100)

kf = KFold(n_splits=5, shuffle=True, random_state=42)

for train_idx, val_idx in kf.split(X):
    # 1. Split the data
    X_train, X_val = X[train_idx], X[val_idx]
    y_train, y_val = y[train_idx], y[val_idx]

    # 2. Create a pipeline that will be fitted only on training data
    pipe = Pipeline([
        ('scaler', StandardScaler()),          # Fit on X_train only
        ('clf', LogisticRegression())
    ])

    # 3. Fit the pipeline
    pipe.fit(X_train, y_train)

    # 4. Evaluate on the validation fold
    val_score = pipe.score(X_val, y_val)
    print(f'Fold score: {val_score:.3f}')

Line‑by‑line explanation:

1. Import necessary classes for cross‑validation, scaling, modelling, and data handling.
2. Generate a toy dataset with 100 samples and 5 features, along with binary targets.
3. Instantiate a KFold splitter to create five non‑overlapping folds.
4. Loop over each fold, obtaining training and validation indices.
5. Sub‑set the original arrays to obtain training and validation sets.
6. Define a Pipeline that first scales the data and then fits a logistic regression model.
7. Fit the pipeline on the training subset only; the scaler learns mean and variance from X_train alone.
8. Score the fitted model on the validation subset; because scaling and training occurred solely on the training fold, no leakage has occurred.
9. Print the validation score for each fold.

Notice how the Pipeline encapsulates both preprocessing and modelling. This ensures that every step is re‑fitted for every fold, preserving the integrity of the evaluation.

Over‑Fitting to the Validation Set

Hyperparameter tuning typically involves a search over a grid or random set of values, evaluating each candidate on a held‑out validation set. The temptation to iterate until the validation score peaks can lead to a model that is finely tuned to the idiosyncrasies of that particular split, but performs poorly on truly unseen data.

Common signs of over‑fitting to validation include:

• Large discrepancy between training and validation loss.
• Validation performance that improves with more hyperparameter iterations, but plateaus or degrades when applied to a held‑out test set.
• Complex models (e.g. deep trees, high‑degree polynomials) that achieve near‑perfect validation scores.

To mitigate this risk, employ a three‑way split: training, validation, and test. The validation set is used exclusively for hyperparameter selection; the test set is reserved for a final, unbiased assessment. Additionally, consider nested cross‑validation, where an outer loop evaluates model generalisation and an inner loop selects hyperparameters.

The Curse of Dimensionality

As the number of hyperparameters grows, the search space expands combinatorially. Even with sophisticated search strategies such as Bayesian optimisation, the model may struggle to explore the space effectively, leading to sub‑optimal solutions.

Key consequences of high dimensionality include:

• Increased computational cost: Evaluating a single configuration can be expensive, and many more configurations are required.
• Sparse data coverage: With limited data, the probability that any particular region of the hyperparameter space is sampled diminishes.
• Over‑complex models: High‑dimensional hyperparameter spaces often encourage overly complex models that are difficult to interpret and maintain.

Practical strategies to combat the curse include:

1. Prioritise a small set of influential hyperparameters: Use domain knowledge or preliminary experiments to identify which parameters most affect performance.
2. Employ sequential model‑based optimisation (SMBO): Algorithms such as Gaussian processes or tree‑parzen estimators adaptively focus on promising regions.
3. Dimensionality reduction on hyperparameters: Techniques like hyperparameter importance ranking (e.g. using SHAP values) can prune irrelevant parameters.

Concrete Example: Avoiding Over‑Fitting with Nested Cross‑Validation

Below is a succinct illustration of nested cross‑validation using scikit‑learn. The outer loop estimates generalisation error, while the inner loop optimises hyperparameters.

from sklearn.datasets import load_iris
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.ensemble import RandomForestClassifier
import numpy as np

X, y = load_iris(return_X_y=True)

Define the hyperparameter grid
param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [None, 5, 10]
}

Inner loop: grid search
inner_cv = GridSearchCV(
    estimator=RandomForestClassifier(random_state=42),
    param_grid=param_grid,
    cv=3,  # 3‑fold CV within the training set
    scoring='accuracy'
)

Outer loop: evaluate the chosen hyperparameters
outer_scores = cross_val_score(
    estimator=inner_cv,
    X=X,
    y=y,
    cv=5,  # 5‑fold CV to estimate generalisation
    scoring='accuracy'
)

print(f'Nested CV accuracy: {outer_scores.mean():.3f} ± {outer_scores.std():.3f}')

Line‑by‑line explanation:

1. Import the iris dataset, grid search, cross‑validation utilities, and a random forest classifier.
2. Load feature matrix X and target vector y.
3. Specify a grid of two hyperparameters: number of trees and maximum depth.
4. Instantiate GridSearchCV as the inner loop estimator; it will perform a 3‑fold CV within each training split.
5. Define the outer cross‑validation using cross_val_score; here we use 5 folds to evaluate generalisation.
6. Fit the nested CV pipeline and compute the mean and standard deviation of accuracy across outer folds.
7. Print the nested CV performance, which reflects how the model would behave on unseen data.

Because the hyperparameter search is confined to the inner folds, the outer folds remain truly independent, eliminating the risk of over‑fitting to the validation set.

Practical Checklist for Robust Hyperparameter Tuning

Adhering to these principles transforms hyperparameter tuning from a trial‑and‑error exercise into a disciplined, reproducible scientific endeavour. By vigilantly guarding against data leakage, over‑fitting to validation data, and the curse of dimensionality, you ensure that the models you deploy are truly robust and generalisable.

# Example 2: Bayesian optimisation with Hyperopt
from hyperopt import fmin, tpe, hp, Trials
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import GradientBoostingClassifier
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

Define search space
space = {
    'learning_rate': hp.loguniform('learning_rate', np.log(0.01), np.log(0.3)),
    'max_depth': hp.quniform('max_depth', 3, 10, 1),
    'subsample': hp.uniform('subsample', 0.5, 1.0),
    'n_estimators': hp.choice('n_estimators', [100, 200, 300])
}

def objective(params):
    # Convert data types
    params['max_depth'] = int(params['max_depth'])
    params['n_estimators'] = int(params['n_estimators'])
    clf = GradientBoostingClassifier(**params, random_state=42)
    # 5‑fold CV, higher is better
    score = cross_val_score(clf, X, y, cv=5, scoring='roc_auc').mean()
    return -score  # minimise negative AUC

trials = Trials()
best = fmin(fn=objective,
            space=space,
            algo=tpe.suggest,
            max_evals=30,
            trials=trials,
            rstate=np.random.default_rng(42))

print("Best hyperparameters:", best)

Further Reading

• Goodfellow, I., Bengio, Y., & Courville, A. (2016) — Deep Learning. A comprehensive overview of model training techniques, including hyperparameter selection and optimisation strategies. This textbook is widely cited in machine learning curricula.
• Chollet, F. (2017) — Deep Learning with Python. Provides practical guidance on tuning neural network hyperparameters using Keras, with code examples and performance discussions.
• Hastie, T., Tibshirani, R., & Friedman, J. (2009) — The Elements of Statistical Learning. Contains chapters on regularisation, model selection, and cross‑validation, which underpin principled hyperparameter tuning.
• Scikit‑learn Documentation (2024) — Hyperparameter Tuning. Official guide to GridSearchCV, RandomizedSearchCV and Bayesian optimisation in the scikit‑learn library, including best‑practice recommendations.
• Google Cloud AI Platform Documentation (2024) — Hyperparameter Tuning. Explains automated hyperparameter search on Google Cloud, with practical examples and optimisation metrics.