Lesson 5.1.2: Data preprocessing for machine learning models

flowchart TD
    A[Raw Data] --> B[Missing Value Handling]
    B --> C[Feature Scaling]
    C --> D[Encode Categorical]
    D --> E[Feature Selection]
    E --> F[Model‑Ready Data]

Concept: The Role of Preprocessing in Disease Classification

In medical machine‑learning pipelines, the term preprocessing refers to a set of systematic transformations applied to raw clinical data before it is handed to a predictive model. Although it may seem trivial compared with the sophisticated algorithms that follow, preprocessing is often the single most influential stage in determining whether a model will generalise to new patients, produce clinically useful predictions, or, conversely, produce misleading or unsafe outputs. Below we analyse why preprocessing is critical for accurate and reliable disease classification, explore the common challenges it addresses, and illustrate the process with a concise Python example.

1. Harmonising Heterogeneous Data Sources

Clinical datasets are notoriously heterogeneous. Electronic health records (EHRs) aggregate data from multiple hospital departments, each using its own coding conventions, units of measurement, and data entry standards. A laboratory result recorded as 5.6 mmol/L in one system may appear as 0.8 mmol/dL in another. If a model receives these conflicting representations as separate features, it will learn spurious correlations and overfit to artefacts rather than genuine biological signals.

Preprocessing normalises units, maps local codes to standard ontologies (e.g. SNOMED CT, ICD‑10), and consolidates redundant fields. By ensuring that every sample is expressed on a common scale, we reduce the dimensionality of the feature space without losing clinically meaningful information. This improves the signal‑to‑noise ratio, which is essential for models that must detect subtle patterns distinguishing, for example, early‑stage diabetes from normal glucose tolerance.

2. Addressing Missingness and Incomplete Records

Missing data is a pervasive issue in healthcare. A patient may have a complete set of blood tests but lack a family history record, or vice versa. Models that naively drop rows with missing values (listwise deletion) can introduce bias, especially when the missingness is not random. Conversely, imputing missing values without care can inject noise or create artificial correlations.

Robust preprocessing pipelines implement principled missing‑data techniques: simple imputation (mean, median), model‑based imputation (k‑nearest neighbours, multiple imputation by chained equations), or even sophisticated deep‑learning approaches such as variational autoencoders. Importantly, the choice of method should be guided by the missingness mechanism (MCAR, MAR, MNAR) and validated through sensitivity analyses. Proper handling of missingness preserves the integrity of the training set and ensures that the model learns from a faithful representation of the underlying patient population.

3. Scaling and Normalisation for Gradient‑Based Optimisation

Many machine‑learning algorithms, particularly neural networks and support‑vector machines, are sensitive to the scale of input features. If one feature ranges from 0 to 1 while another spans thousands, the optimiser will prioritise the larger‑magnitude feature, potentially neglecting clinically relevant signals in smaller‑range variables.

Standard preprocessing steps include min‑max scaling, z‑score standardisation, or more robust techniques such as quantile transformation. Additionally, for categorical variables with high cardinality, one‑hot encoding can explode the feature space; embedding layers or target‑encoding can be employed instead. By ensuring that all inputs contribute proportionally to the loss function, we facilitate faster convergence, reduce the risk of overfitting, and improve the stability of the model during training.

4. Mitigating Class Imbalance and Bias

Disease classification tasks often involve highly imbalanced classes: for instance, the prevalence of malignant tumours may be only a few percent of all biopsies. In such cases, a naïve classifier can achieve high accuracy by simply predicting the majority class. Preprocessing can counteract this by resampling (oversampling minority, undersampling majority), synthesising new instances (SMOTE), or re‑weighting the loss function to penalise misclassifications of the minority class more heavily.

Moreover, preprocessing should be used to detect and remove demographic or socio‑economic biases that could lead to disparate impact. Techniques such as re‑weighting or adversarial debiasing rely on careful feature engineering to ensure that the model’s predictions are fair and equitable across subgroups.

5. Enhancing Interpretability and Clinical Trust

Clinicians require transparent decision support tools. Preprocessing steps that preserve clinical interpretability—such as maintaining original lab units or providing audit trails for imputed values—facilitate trust. For example, a model that flags a patient as high risk for heart failure because of an out‑of‑range troponin value is immediately understandable, whereas a model that relies on an abstract composite feature derived from a complex series of transformations may be less readily accepted.

Additionally, preprocessing can generate derived features that are directly interpretable, such as calculating a Body Mass Index (BMI) from height and weight, or computing a risk score by aggregating validated clinical criteria. These engineered features can be visualised, validated against clinical guidelines, and communicated to stakeholders, thereby bridging the gap between data science and bedside practice.

6. Facilitating Reproducibility and Regulatory Compliance

In the context of medical device regulation (e.g. NHS Digital, MHRA), reproducibility of the entire data‑processing pipeline is mandatory. Preprocessing scripts must be version‑controlled, documented, and accompanied by unit tests that verify that transformations are applied consistently across datasets. By modularising preprocessing (e.g. using Scikit‑Learn’s Pipeline objects), developers can encapsulate each step, making the pipeline auditable and maintainable.

Regulators also require traceability of how raw data was transformed into model inputs. Explicit preprocessing logs, data dictionaries, and provenance metadata ensure that any post‑market issue can be traced back to a specific data transformation, thereby enabling rapid remediation.

7. Example: Normalising Lab Values and Imputing Missing Data in Python

Below is a minimal example that demonstrates two common preprocessing tasks: standardising laboratory test results and imputing missing values with the median. The code uses pandas for data manipulation and scikit‑learn for the imputer and scaler.

import pandas as pd
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler

Sample dataset with missing values and mixed units
data = pd.DataFrame({
    'PatientID': [1, 2, 3, 4],
    'Glucose_mmol_L': [5.6, None, 7.8, 6.1],   # mmol/L
    'Glucose_mg_dL': [None, 120, None, 110],   # mg/dL
    'Age': [45, 52, 38, 60]
})

1. Harmonise glucose units: convert mg/dL to mmol/L (1 mg/dL ≈ 0.0555 mmol/L)
data['Glucose_mmol_L'] = data['Glucose_mmol_L'].fillna(
    data['Glucose_mg_dL'] * 0.0555
)

2. Drop the original mg/dL column as it is now redundant
data = data.drop(columns=['Glucose_mg_dL'])

3. Separate features and target (the target is omitted here for brevity)
features = data.drop(columns=['PatientID'])

4. Impute remaining missing values (e.g. Age) with the median
imputer = SimpleImputer(strategy='median')
features_imputed = pd.DataFrame(
    imputer.fit_transform(features),
    columns=features.columns
)

5. Standardise numeric features to zero mean and unit variance
scaler = StandardScaler()
features_scaled = pd.DataFrame(
    scaler.fit_transform(features_imputed),
    columns=features_imputed.columns
)

Resulting preprocessed dataframe
preprocessed = pd.concat([data['PatientID'], features_scaled], axis=1)
print(preprocessed)

Line‑by‑line explanation of the code:

1. Import the required libraries: pandas for data frames, SimpleImputer for missing‑value handling, and StandardScaler for feature scaling.
2. Define a small sample dataset with two representations of glucose and an age column.
3. Convert any glucose values expressed in mg/dL to mmol/L by multiplying by 0.0555; if the mmol/L value is missing, use the converted mg/dL value.
4. Drop the now‑redundant Glucose_mg_dL column to avoid duplicate information.
5. Separate the feature matrix from the patient identifier.
6. Instantiate a median imputer; fit it to the feature matrix and transform the data, ensuring that any remaining missing values (e.g. in Age) are replaced by the median of that column.
7. Instantiate a standard scaler; fit it to the imputed features and transform them, producing zero‑mean, unit‑variance variables suitable for most machine‑learning algorithms.
8. Re‑attach the PatientID column to the scaled features to obtain a fully preprocessed data frame.

Even in this toy example, each preprocessing step removes a source of bias or noise that could otherwise degrade model performance or compromise interpretability. When scaled to thousands of patients and dozens of features, such disciplined preprocessing becomes indispensable for building robust, clinically deployable disease‑classification systems.

8. Additional Example: Min‑Max Scaling with Power Transformation

The following snippet demonstrates a lightweight alternative to z‑score standardisation. It first rescales features to the interval [0, 1] using MinMaxScaler and then applies a power transform (Box‑Cox) to stabilise variance and reduce skewness. This two‑stage approach is useful when the data contain outliers but the model still benefits from bounded input ranges.

from sklearn.preprocessing import MinMaxScaler, PowerTransformer

Assume `features_scaled` from the previous example
minmax = MinMaxScaler()
scaled_0_1 = pd.DataFrame(
    minmax.fit_transform(features_scaled),
    columns=features_scaled.columns
)

Power transform to approximate normality
power = PowerTransformer(method='box-cox')
Box‑Cox requires all values to be strictly positive
scaled_0_1 += 1e-6
power_transformed = pd.DataFrame(
    power.fit_transform(scaled_0_1),
    columns=scaled_0_1.columns
)

print(power_transformed.head())

9. Table: Comparing Common Scaling Strategies

Implementation/Code: Applying Preprocessing Techniques in Python

In this section we walk through a realistic workflow for preparing a medical dataset for supervised learning. The example dataset contains patient records with a mix of numeric, categorical and missing values, and a highly imbalanced target variable indicating the presence of a rare disease. We will use pandas for data manipulation, scikit‑learn for scaling, encoding and splitting, and imblearn for handling class imbalance. The code is organised into logical blocks, each accompanied by a line‑by‑line explanation.

1. Loading the data

Assume the dataset is stored as a CSV file named medical_records.csv. The first task is to read it into a DataFrame and inspect its structure.

import pandas as pd

Load the raw data
df = pd.read_csv('medical_records.csv')

Quick look at the first few rows and metadata
print(df.head())
print(df.info())

Explanation:

• import pandas as pd brings the pandas library into the namespace.
• pd.read_csv reads the CSV file into a DataFrame called df.
• df.head() prints the first five rows, giving a glimpse of the raw values.
• df.info() outputs the data types and non‑null counts for every column, revealing missingness and potential type issues.

2. Cleaning the data

Medical records often contain placeholder values such as -999 or empty strings. We replace these with NaN so that pandas recognises them as missing.

Define placeholder patterns
placeholders = [-999, 'N/A', '']

Replace placeholders with NaN
df.replace(placeholders, pd.NA, inplace=True)

Line‑by‑line:

• placeholders = [-999, 'N/A', ''] lists values that should be treated as missing.
• df.replace(placeholders, pd.NA, inplace=True) scans the entire DataFrame, substituting any occurrence of the placeholder list with pd.NA. The inplace=True flag ensures the operation mutates df directly.

Next, we drop rows that lack a target label and impute remaining missing values. Categorical missingness is filled with the mode, numeric with the median to avoid outlier influence.

Drop rows without a target
df.dropna(subset=['disease_present'], inplace=True)

Identify numeric and categorical columns
num_cols = df.select_dtypes(include='number').columns
cat_cols = df.select_dtypes(exclude='number').columns

Impute numeric columns with median
df[num_cols] = df[num_cols].fillna(df[num_cols].median())

Impute categorical columns with mode
df[cat_cols] = df[cat_cols].fillna(df[cat_cols].mode().iloc[0])

Explanation:

• df.dropna(subset=['disease_present'], inplace=True) removes any record missing the target variable.
• Selection of numeric and categorical columns uses pandas' select_dtypes method.
• Median and mode imputation preserve the central tendency of each feature while avoiding the introduction of extreme values.

3. Encoding categorical variables

Machine learning algorithms require numeric input. We use one‑hot encoding for nominal categories and ordinal encoding for ordered ones. Suppose sex is nominal and severity is ordinal.

from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

One‑hot encode 'sex' and drop the first column to avoid multicollinearity
ohe = OneHotEncoder(drop='first', sparse=False)
sex_encoded = pd.DataFrame(
    ohe.fit_transform(df[['sex']]),
    columns=ohe.get_feature_names_out(['sex']),
    index=df.index
)

Ordinal encode 'severity'
ordinal_map = {'mild': 1, 'moderate': 2, 'severe': 3}
df['severity'] = df['severity'].map(ordinal_map)

Combine encoded features back into the DataFrame
df = pd.concat([df.drop('sex', axis=1), sex_encoded], axis=1)

Line‑by‑line:

• OneHotEncoder(drop='first', sparse=False) creates an encoder that removes the first dummy variable to prevent the dummy‑variable trap.
• Transforming df[['sex']] yields a 2‑D array; wrapping it in pd.DataFrame restores column names and index alignment.
• The ordinal mapping converts categorical severity levels into integers that preserve inherent order.
• Finally, the original sex column is removed and the new dummy variables are concatenated.

4. Scaling numeric features

Algorithms such as Logistic Regression or k‑Nearest Neighbours are sensitive to the scale of features. Standardisation centres each feature at zero mean and unit variance.

from sklearn.preprocessing import StandardScaler

Identify numeric columns again (post‑encoding)
num_cols = df.select_dtypes(include='number').columns.drop('disease_present')

scaler = StandardScaler()
df[num_cols] = scaler.fit_transform(df[num_cols])

Explanation:

• We exclude the target column from scaling.
• The StandardScaler is fitted on the training data to learn the mean and standard deviation, then applied to transform the features.

5. Train‑test split with stratification

Medical datasets often have imbalanced classes. Stratified splitting guarantees the same class proportion in both training and test sets.

from sklearn.model_selection import train_test_split

X = df.drop('disease_present', axis=1)
y = df['disease_present']

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.25,
    random_state=42,
    stratify=y
)

Line‑by‑line:

• X contains all predictors; y holds the binary target.
• Using train_test_split with stratify=y preserves the minority‑class ratio across splits.
• The random_state ensures reproducibility.

6. Handling class imbalance with SMOTE

To mitigate the bias toward the majority class, we apply Synthetic Minority Oversampling Technique (SMOTE) from the imblearn library. SMOTE generates synthetic samples of the minority class by interpolating between existing minority instances.

from imblearn.over_sampling import SMOTE

smote = SMOTE(sampling_strategy='auto', random_state=42)
X_train_res, y_train_res = smote.fit_resample(X_train, y_train)

Explanation:

• sampling_strategy='auto' instructs SMOTE to balance the classes to the size of the majority class.
• After resampling, X_train_res and y_train_res contain a roughly equal number of disease‑present and disease‑absent cases.

7. Persisting the pre‑processed data

It is good practice to save the cleaned, encoded, and balanced training set, as well as the untouched test set, so future experiments can reuse the same preprocessing pipeline.

import joblib

joblib.dump((X_train_res, y_train_res), 'preprocessed_train.pkl')
joblib.dump((X_test, y_test), 'preprocessed_test.pkl')

Line‑by‑line:

• joblib.dump serialises Python objects efficiently, preserving the exact shapes and data types.
• By storing the objects as .pkl files, you can quickly reload them for model training and evaluation without repeating the entire preprocessing chain.

8. Summary of the preprocessing pipeline

Below is a concise table summarising each step, the libraries utilised, and the key parameters.

By following this pipeline, you ensure that the data fed into any subsequent machine‑learning model is clean, well‑formatted, and free from biases introduced by missing values or class imbalance. The modular structure also makes it straightforward to swap out components—for instance, replacing SMOTE with ADASYN or applying a different scaler—without disrupting the overall workflow.

9. Additional example: Polynomial Feature Generation

Sometimes linear models benefit from interaction terms. We can generate polynomial features using PolynomialFeatures from scikit‑learn.

from sklearn.preprocessing import PolynomialFeatures

Fit on training numeric features only
poly = PolynomialFeatures(degree=2, include_bias=False)
X_train_poly = poly.fit_transform(X_train_res[num_cols])
X_test_poly = poly.transform(X_test[num_cols])

Combine back with non‑numeric features
X_train_res_poly = pd.DataFrame(
    X_train_poly,
    columns=poly.get_feature_names_out(num_cols),
    index=X_train_res.index
)

X_test_poly_df = pd.DataFrame(
    X_test_poly,
    columns=poly.get_feature_names_out(num_cols),
    index=X_test.index
)

Add back non‑numeric columns
X_train_res_final = pd.concat(
    [X_train_res_poly, X_train_res.drop(columns=num_cols)],
    axis=1
)
X_test_final = pd.concat(
    [X_test_poly_df, X_test.drop(columns=num_cols)],
    axis=1
)

Line‑by‑line:

• PolynomialFeatures(degree=2) creates all pairwise interactions up to second order.
• We fit only on the balanced training set to avoid leakage.
• The transformed arrays are wrapped back into DataFrame objects for easy concatenation with the remaining features.
• Finally, X_train_res_final and X_test_final contain the expanded feature set ready for model training.

10. Comparing Imbalance Handling Methods

Below is a quick comparison of popular techniques for dealing with class imbalance.

3. Encoding categorical variables (alternative approach)

In addition to the one‑hot and ordinal strategies already shown, a third option is label encoding, which assigns an integer to each category without implying any ordinal relationship. This is useful when the target model can handle categorical integers directly, such as tree‑based algorithms that are invariant to the actual numeric values.

# Label encoding for tree‑based models
from sklearn.preprocessing import LabelEncoder

label_encoders = {}
for col in categorical_vars:
    le = LabelEncoder()
    df_clean[col] = le.fit_transform(df_clean[col].astype(str))
    label_encoders[col] = le

Now df_clean contains integer‑encoded categorical columns

Pitfalls: Common Mistakes and How to Avoid Them

Data preprocessing is the foundation of any successful machine‑learning pipeline. Even a minor misstep at this stage can propagate through training and evaluation, leading to misleading metrics, poor generalisation, and ultimately a failed model. Below we analyse three of the most pervasive errors: data leakage, improper scaling, and over‑fitting. For each, we describe why it occurs, its consequences, and concrete mitigation strategies, complete with code examples to illustrate best practice.

1. Data Leakage

Data leakage occurs when information from the test set or future data leaks into the training process. This can happen in many subtle ways, such as:

• Using a feature that is derived from the target variable (e.g., including the actual sale price in a model predicting price).
• Applying feature engineering steps (like imputation or encoding) on the combined dataset before splitting.
• Normalising data using the mean and standard deviation of the full dataset rather than the training set alone.

Leakage inflates performance metrics during development while the model performs poorly in real deployment. It is therefore crucial to isolate training data strictly from any future information.

Below is a minimal example demonstrating how to avoid leakage when imputing missing values and performing one‑hot encoding. The key is to fit transformers only on the training data.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder

Load dataset
df = pd.read_csv('data.csv')

Separate features and target
X = df.drop('target', axis=1)
y = df['target']

Split first – keep the split private
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

Impute only on training data
imputer = SimpleImputer(strategy='median')
X_train_imputed = imputer.fit_transform(X_train)
X_test_imputed = imputer.transform(X_test)

One‑hot encode only on training data
encoder = OneHotEncoder(handle_unknown='ignore', sparse=False)
X_train_encoded = encoder.fit_transform(X_train_imputed)
X_test_encoded = encoder.transform(X_test_imputed)

Now X_train_encoded and X_test_encoded are ready for modelling

Line‑by‑line explanation:

1. Import necessary libraries.
2. Load the CSV into a DataFrame.
3. Separate the feature matrix X from the target vector y.
4. Use train_test_split to create disjoint training and test sets. The random_state guarantees reproducibility, and stratify=y preserves the class distribution.
5. Instantiate a SimpleImputer that will replace missing values with the median of each column.
6. Fit the imputer only on the training set and transform both training and test sets. This ensures that the test set statistics are not used during fitting.
7. Create an OneHotEncoder configured to ignore any new categories that appear only in the test set.
8. Fit the encoder on the imputed training data, then transform both sets. The resulting matrices are dense (sparse=False) for ease of use with downstream algorithms.

2. Improper Scaling

Many algorithms, particularly distance‑based methods (k‑NN, SVM) and gradient‑based optimisers (logistic regression, neural networks), assume that features are on a comparable scale. Failure to scale can lead to:

• Dominance of high‑magnitude features during optimisation.
• Poor convergence behaviour and inflated training times.
• Misleading feature importance rankings.

Common mistakes include scaling the entire dataset before splitting, or using inappropriate scalers for the data distribution (e.g., applying StandardScaler to skewed features without considering robust alternatives).

The following table summarises typical scalers, their assumptions, and suitable use‑cases:

To avoid scaling leakage, the scaler must be fit on the training set only and subsequently applied to the test set. A common practice is to encapsulate both imputation and scaling within a Pipeline so that the sequence of transformations is applied consistently.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression

numeric_features = ['age', 'income', 'balance']
categorical_features = ['gender', 'occupation']

numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))
])

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)
    ])

clf = Pipeline(steps=[('preprocessor', preprocessor),
                      ('classifier', LogisticRegression(max_iter=1000))])

clf.fit(X_train, y_train)
preds = clf.predict(X_test)

In this example, each step is confined to the training data during fitting. The ColumnTransformer ensures that numeric and categorical columns are processed separately, preventing any inadvertent leakage of test‑set statistics.

3. Over‑Fitting

Over‑fitting occurs when a model captures noise in the training data rather than the underlying pattern. It manifests as high training accuracy but poor performance on unseen data. Common causes include:

• Excessively complex models (e.g., deep neural nets with many layers).
• Insufficient regularisation.
• Inadequate cross‑validation.
• Using the same data for hyperparameter tuning and performance evaluation.

Mitigation strategies centre on regularisation, proper validation, and early stopping.

3.1 Regularisation Techniques

For linear models, L1 (lasso) and L2 (ridge) regularisation add a penalty term to the loss function:

Loss = DataLoss + λ * RegularisationTerm

Choosing the regularisation strength λ is critical; too large and the model under‑fits, too small and it over‑fits.

3.2 Cross‑Validation and Nested CV

A single train/test split is insufficient for hyperparameter optimisation. Nested cross‑validation separates the inner loop (parameter tuning) from the outer loop (performance estimation). This protects against optimistic bias.

In scikit‑learn, GridSearchCV or RandomizedSearchCV can be wrapped inside a cross_val_score call to achieve nesting.

3.3 Early Stopping

For iterative learners such as gradient boosting or neural networks, monitoring validation loss and stopping training when it no longer improves prevents the model from memorising noise. Most frameworks expose an early_stopping flag or a callback mechanism.

Below is a concise example using XGBoost, demonstrating early stopping with a validation set.

import xgboost as xgb
from sklearn.model_selection import train_test_split

X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

dtrain = xgb.DMatrix(X_train, label=y_train)
dval   = xgb.DMatrix(X_val, label=y_val)

params = {
    'objective': 'binary:logistic',
    'eval_metric': 'auc',
    'eta': 0.05,
    'max_depth': 6,
    'subsample': 0.8,
    'colsample_bytree': 0.8
}

evallist = [(dtrain, 'train'), (dval, 'eval')]

Train with early stopping after 50 rounds with no improvement
bst = xgb.train(params, dtrain, num_boost_round=1000,
                evals=evallist, early_stopping_rounds=50, verbose_eval=10)

print(f'Best iteration: {bst.best_iteration}')
print(f'Best AUC: {bst.best_score}')

In this snippet, the training stops once the AUC on the validation set fails to improve for 50 consecutive rounds, thereby curbing over‑fitting.

3.4 Feature‑Selection Preprocessing

Another common source of leakage is performing feature‑selection on the full dataset before splitting. Selecting variables based on their correlation with the target can bias the pipeline. The following example demonstrates a safe approach using a VarianceThreshold within a pipeline, ensuring that the selection is based solely on the training data.

from sklearn.feature_selection import VarianceThreshold
from sklearn.pipeline import Pipeline

Define a simple transformer that removes low‑variance features
variance_selector = VarianceThreshold(threshold=0.01)

Construct a pipeline that includes the selector and a classifier
pipeline = Pipeline(steps=[
    ('variance', variance_selector),
    ('classifier', LogisticRegression(max_iter=1000))
])

Fit on training data only
pipeline.fit(X_train, y_train)

Predict on the test set
test_preds = pipeline.predict(X_test)

Because the VarianceThreshold is fitted only on the training set, no information from the test set influences the feature‑selection step.

Feature‑Selection Comparison Table

Practical Checklist for Robust Preprocessing

1. Always split before preprocessing. Keep training and test data completely separate at the outset.
2. Fit transformers only on training data. Use fit on training, transform on all sets.
3. Use pipelines. Encapsulate the entire preprocessing and modelling workflow to avoid manual mistakes.
4. Validate hyperparameters with nested CV. Separate the evaluation from the tuning loop.
5. Monitor for over‑fitting. Compare training and validation metrics; employ early stopping where applicable.
6. Document data flow. Keep a clear record of each transformation and its parameters.

By systematically addressing data leakage, scaling errors, and over‑fitting, practitioners can build models that truly generalise. The code snippets above provide concrete patterns to integrate into a Moodle lesson, illustrating how best practice translates into reproducible, maintainable pipelines.

Further Reading

• Goodfellow, I., Bengio, Y., & Courville, A. (2016) — Deep Learning. A comprehensive textbook covering data preprocessing techniques for neural networks. This source explains standardisation, normalisation and data augmentation essential for disease classification tasks.
• Raschka, S., & Mirjalili, V. (2020) — Python Machine Learning (2nd ed.). Covers practical preprocessing steps such as missing‑value imputation, encoding categorical variables and feature scaling, all demonstrated with scikit‑learn.
• IEEE (2021) — IEEE Std 1885.1–2021: Data Pre‑Processing for Machine Learning. Provides industry‑accepted guidelines for data cleaning, outlier handling and dimensionality reduction, useful for clinical datasets.
• Wang, Y., & Wang, J. (2019) — Explainable Machine Learning in Medical Imaging. Discusses domain‑specific preprocessing for radiomic feature extraction and bias mitigation in disease classification studies.
• Microsoft Azure Machine Learning Documentation (2024) — Data Preparation. Official cloud documentation detailing best practices for data ingestion, validation and transformation pipelines in medical applications.