Lesson 5.1.3: Building a baseline crop disease classifier

flowchart TD
    A[Collect image data] --> B[Annotate with disease labels]
    B --> C[Pre‑process images (resize, augment)]
    C --> D[Split into training / validation / test]
    D --> E[Train baseline classifier (CNN)]
    E --> F[Validate & evaluate performance]
    F --> G[Deploy model for inference]

Illustrative Code Snippet: A Simple Logistic Regression Baseline

Below is a minimal Python example that demonstrates how to construct a baseline classifier using histogram of oriented gradients (HOG) features and a logistic regression model. The code is deliberately kept simple so that the logic behind each step is transparent.

import cv2
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

1. Load image and convert to grayscale
image = cv2.imread('leaf.jpg')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

2. Compute HOG features
winSize = (64, 64)
blockSize = (16, 16)
blockStride = (8, 8)
cellSize = (8, 8)
nbins = 9
hog = cv2.HOGDescriptor(winSize, blockSize, blockStride, cellSize, nbins)
features = hog.compute(gray).flatten()

3. Prepare dataset (X, y) – here we mock a small array
X = np.array([features])  # feature matrix
y = np.array([0])         # 0 = healthy, 1 = diseased

4. Train-test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

5. Train logistic regression
clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)

6. Predict and evaluate
y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred))

Line‑by‑line explanation:

• import cv2 – Imports OpenCV, a library for computer‑vision tasks.
• import numpy as np – Provides array operations.
• from sklearn.linear_model import LogisticRegression – Brings in the logistic regression classifier.
• from sklearn.model_selection import train_test_split – Utility to split the data.
• from sklearn.metrics import classification_report – Helper to summarise performance.
• image = cv2.imread('leaf.jpg') – Reads an image file into memory.
• gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) – Converts the image to greyscale, simplifying the feature space.
• HOG parameters are defined to extract edge‑orientation statistics from the greyscale image.
• features = hog.compute(gray).flatten() – Computes the HOG descriptor and flattens it into a 1‑D array.
• Mock dataset creation: X holds the feature vector, y holds the label.
• Data is split into training and testing sets using a 80/20 ratio.
• The logistic regression model is instantiated with a higher iteration limit to ensure convergence.
• The model is trained on the training subset.
• Predictions are made on the test set, and a classification report is printed, showing precision, recall, F1‑score, and support.

While this example uses a single image and a dummy label, the pattern scales to thousands of images: loop over the dataset, compute HOG features for each, stack them into a matrix, and fit the logistic regression. The resulting baseline will give you an initial accuracy that you can improve upon with more advanced techniques.

Illustrative Code Snippet: A Baseline Using Raw Pixels and a Support‑Vector Machine

For comparison, the following code shows how to build a baseline that uses raw pixel values as features and a linear support‑vector machine (SVM). This approach is often faster to prototype and can be surprisingly effective on small, well‑labelled datasets.

import cv2
import numpy as np
from sklearn.svm import LinearSVC
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

1. Load and resize to a fixed size
image = cv2.imread('leaf.jpg')
resized = cv2.resize(image, (128, 128))   # 128x128 pixels

2. Flatten to a 1‑D feature vector (raw RGB pixels)
features = resized.reshape(-1).astype(np.float32) / 255.0

3. Mock dataset (repeat for illustration)
X = np.array([features])                 # feature matrix
y = np.array([0])                        # 0 = healthy, 1 = diseased

4. Train‑test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

5. Train linear SVM
clf = LinearSVC(max_iter=10000)
clf.fit(X_train, y_train)

6. Predict and evaluate
y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred))

Implementation/Code: Building a Simple CNN

In this section we walk through the construction of a minimal convolutional neural network (CNN) that can be trained to recognise crop diseases from leaf images. The network will be built using TensorFlow 2.x and Keras, the high‑level API that makes model definition straightforward while still offering fine‑grained control. By the end of this section you will understand how each layer contributes to feature extraction, how the model is compiled, and how to visualise training progress.

1. Importing the Essentials

Before any modelling can commence we must import the libraries that provide numerical support, data handling utilities, and the deep‑learning framework itself. Note that TensorFlow 2.x includes Keras as a sub‑package, so there is no need to install Keras separately.

import tensorflow as tf
from tensorflow.keras import layers, models
import matplotlib.pyplot as plt
import numpy as np
import os

Line 1 imports the core TensorFlow library, giving access to tensors, automatic differentiation, and GPU support if available. Line 2 pulls in the layers and models modules – these contain the building blocks for our CNN. Line 3 imports Matplotlib for visualising training curves. Line 4 brings in NumPy, which will be useful for any post‑processing. Line 5 imports the OS module to help with file path handling later on.

2. Preparing the Data

We assume that the dataset has already been split into training and validation directories, each containing sub‑folders named after the disease classes. Keras provides the convenient image_dataset_from_directory function, which handles loading, resizing, and batching automatically.

IMG_SIZE = (128, 128)           # Standardise all images to 128×128
BATCH_SIZE = 32                 # Batch size for efficient training

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    "data/train",
    seed=123,
    image_size=IMG_SIZE,
    batch_size=BATCH_SIZE
)

val_ds = tf.keras.preprocessing.image_dataset_from_directory(
    "data/val",
    seed=123,
    image_size=IMG_SIZE,
    batch_size=BATCH_SIZE
)

Line 1–2 declare constants for image dimensions and batch size. Line 4–10 create the training dataset. The seed parameter ensures reproducible shuffling. Line 12–18 do the same for validation data. Each dataset yields batches of tuples (images, labels) where images is a tensor of shape (BATCH_SIZE, 128, 128, 3) and labels is a one‑hot encoded vector corresponding to the disease class.

3. Normalising the Pixel Values

Deep learning models train faster and more stably when inputs are normalised. Keras can automatically rescale pixel values to the [0,1] range via a Rescaling layer. We insert this layer at the very start of the network so that every subsequent layer receives normalised data.

normalisation_layer = layers.Rescaling(1./255)

Line 1 creates a rescaling layer that divides each pixel value by 255. This transformation is applied lazily – it will only be executed when the model processes a batch during training or inference.

4. Defining the CNN Architecture

Our baseline CNN will consist of two convolutional blocks followed by a fully connected classifier. Each block contains a convolutional layer, a rectified linear unit (ReLU) activation, and a max‑pooling layer. The final dense layer uses softmax to produce class probabilities.

def build_simple_cnn(num_classes):
    model = models.Sequential([
        normalisation_layer,                             # Normalise inputs

        # Block 1
        layers.Conv2D(32, (3, 3), padding='same', activation='relu',
                      input_shape=IMG_SIZE + (3,)),
        layers.MaxPooling2D((2, 2)),

        # Block 2
        layers.Conv2D(64, (3, 3), padding='same', activation='relu'),
        layers.MaxPooling2D((2, 2)),

        layers.Flatten(),
        layers.Dense(128, activation='relu'),
        layers.Dense(num_classes, activation='softmax')
    ])
    return model

Line 1 declares a function that returns a compiled model once invoked. Line 2 starts a Keras Sequential model, which is appropriate for a linear stack of layers. Line 3 adds the normalisation layer defined earlier.

Block 1 comprises a Conv2D layer with 32 filters of size 3×3. The padding='same' argument ensures that the output spatial dimensions match the input. The activation='relu' applies the rectified linear unit non‑linearity. The input_shape parameter is only required on the first layer and is set to the image size plus three colour channels. Line 7 adds a 2×2 max‑pooling layer, halving the spatial dimensions.

Block 2 mirrors Block 1 but with 64 filters, allowing the network to learn more complex features. After the second pooling step the feature maps are flattened into a one‑dimensional vector. Line 13 introduces a fully connected layer with 128 units and ReLU activation, which mixes the extracted features. Line 14 adds the output layer; the number of units equals num_classes and the softmax activation converts logits into a probability distribution over classes.

5. Adding Dropout for Regularisation

To mitigate over‑fitting we can insert a Dropout layer after the flattening step. The following example shows a common placement that preserves the model’s simplicity while providing a small regularisation effect. The dropout probability is set to 0.5, meaning half of the activations are randomly set to zero during each training step.

def build_cnn_with_dropout(num_classes, dropout_rate=0.5):
    model = models.Sequential([
        normalisation_layer,

        layers.Conv2D(32, (3, 3), padding='same', activation='relu',
                      input_shape=IMG_SIZE + (3,)),
        layers.MaxPooling2D((2, 2)),

        layers.Conv2D(64, (3, 3), padding='same', activation='relu'),
        layers.MaxPooling2D((2, 2)),

        layers.Flatten(),
        layers.Dropout(dropout_rate),          # New line
        layers.Dense(128, activation='relu'),
        layers.Dense(num_classes, activation='softmax')
    ])
    return model

When comparing the baseline and dropout‑enhanced models, you may observe a slight increase in validation accuracy and a reduction in the training‑validation accuracy gap.

6. Compiling the Model

Compilation specifies the optimisation strategy, the loss function and the metrics to monitor. For a multi‑class classification problem the categorical cross‑entropy loss is appropriate. We use the Adam optimiser, which adapts the learning rate during training.

model = build_simple_cnn(num_classes=len(train_ds.class_names))

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

Line 1 creates an instance of the model, passing the number of disease classes inferred from the training dataset. Line 3 compiles the model: Adam is instantiated with a learning rate of 0.001, a sensible default for many problems. Line 5 sets the loss to sparse_categorical_crossentropy, which is used when labels are integer‑encoded rather than one‑hot. Line 6 requests that the accuracy metric be computed during training.

7. Training the Network

Training is performed by calling model.fit. We provide the training and validation datasets, specify the number of epochs, and optionally add a callback to stop training early if the validation loss plateaus.

EPOCHS = 20

early_stop = tf.keras.callbacks.EarlyStopping(
    monitor='val_loss',
    patience=3,
    restore_best_weights=True
)

history = model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=EPOCHS,
    callbacks=[early_stop]
)

Line 1 sets the maximum number of epochs. Line 3–7 configure an early‑stopping callback that watches the validation loss; if it does not improve for three consecutive epochs, training halts and the weights from the best epoch are restored. Line 9–15 launch training. The history object returned by fit contains per‑epoch records of loss and accuracy for both training and validation sets.

8. Visualising Training Progress

Plotting the loss and accuracy curves is essential for diagnosing over‑fitting or under‑fitting. The following snippet uses Matplotlib to produce two sub‑plots side by side.

acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs_range = range(len(acc))

plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
plt.plot(epochs_range, acc, label='Training Accuracy')
plt.plot(epochs_range, val_acc, label='Validation Accuracy')
plt.legend(loc='lower right')
plt.title('Accuracy over Epochs')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')

plt.subplot(1, 2, 2)
plt.plot(epochs_range, loss, label='Training Loss')
plt.plot(epochs_range, val_loss, label='Validation Loss')
plt.legend(loc='upper right')
plt.title('Loss over Epochs')
plt.xlabel('Epoch')
plt.ylabel('Loss')

plt.tight_layout()
plt.show()

Lines 1–4 extract the relevant metrics from the history dictionary. Line 6 creates a range object that will be used as the x‑axis. Line 8 initialises a figure of width 12 inches and height 5 inches. Lines 10–15 plot training and validation accuracy on the left subplot, adding a legend and axis labels. Lines 17–22 produce the loss curves on the right subplot. Line 24 ensures that sub‑plots do not overlap, and Line 25 renders the figure. By inspecting these plots you can decide whether to increase model capacity, adjust the learning rate, or collect more data.

9. Saving and Loading the Model

After satisfactory performance, the model can be persisted to disk for later deployment. TensorFlow offers two convenient formats: the legacy H5 format and the TensorFlow SavedModel format, the latter being recommended for production.

# SavedModel format
model.save('models/simple_cnn', save_format='tf')

H5 format (optional)
model.save('models/simple_cnn.h5')

Line 2 saves the model in the default SavedModel format. Line 5 provides an alternative H5 file; this format is useful for compatibility with other frameworks but is less efficient for TensorFlow serving.

To reload the model later, simply call tf.keras.models.load_model and pass the path.

loaded_model = tf.keras.models.load_model('models/simple_cnn')

At this point the loaded_model instance is fully functional and can be used to predict on new leaf images, or further fine‑tuned if new disease categories are added.

10. Interpreting the Results

When evaluating the trained network, pay close attention to the following:

• Training Accuracy vs. Validation Accuracy: A large gap indicates over‑fitting. Consider adding dropout layers or collecting more data.
• Loss Curves: If the training loss continues to decrease while validation loss plateaus or increases, the model is memorising the training set.
• Confusion Matrix: Use scikit‑learn’s confusion_matrix to visualise which disease classes are most frequently mis‑classified, providing insights into data imbalance or ambiguous imagery.

By systematically analysing these diagnostics you can iteratively refine the architecture, hyper‑parameters, and data pipeline, ultimately improving the baseline classifier’s robustness and generalisation.

Worked Example: Applying the Baseline to a Wheat Disease Dataset

In this section we conduct a step‑by‑step walk‑through of a typical machine‑learning pipeline using the baseline classifier described earlier. The dataset we use comprises images of wheat leaves that are either healthy or infected with various fungal diseases such as rust, powdery mildew or leaf spot. The goal is to build a classifier that can recognise disease presence in a new leaf image, and to evaluate its performance against a simple baseline.

1. Loading the Dataset

We organise the data into two directories – healthy and diseased – each containing JPEG images. Using Python’s pathlib and skimage.io libraries we can read the images into memory and tag them with the appropriate class label. The following snippet demonstrates the essential steps.

import pathlib
from skimage import io
import numpy as np

Define the base directory where the two sub‑folders reside
BASE_DIR = pathlib.Path("data/wheat")

Initialise lists to hold image data and labels
images, labels = [], []

Iterate over the two classes
for label in ["healthy", "diseased"]:
    class_dir = BASE_DIR / label
    for img_path in class_dir.glob("*.jpg"):
        # Read the image as a NumPy array
        image = io.imread(img_path)
        images.append(image)
        # Encode the label as 0 (healthy) or 1 (diseased)
        labels.append(0 if label == "healthy" else 1)

Convert to NumPy arrays for downstream processing
X = np.array(images)
y = np.array(labels)

In the code above, pathlib.Path provides a convenient, platform‑independent way to construct file paths. skimage.io.imread loads the image into a 3‑dimensional NumPy array (height × width × channels). By appending the label after the image we create a parallel array that will be used for training and evaluation.

2. Pre‑processing the Images

Raw images vary in size and illumination. To feed them into a simple convolutional neural network (CNN) we normalise the pixel values and resize each image to a fixed resolution. This also reduces memory consumption.

from skimage.transform import resize
from sklearn.preprocessing import StandardScaler

Target resolution for all images
TARGET_SIZE = (128, 128)

Resize images
X_resized = np.array([resize(img, TARGET_SIZE, anti_aliasing=True) for img in X])

Flatten images for the baseline model (non‑CNN)
X_flat = X_resized.reshape((X_resized.shape[0], -1))

Normalise pixel values to zero‑mean, unit‑variance
scaler = StandardScaler()
X_normalised = scaler.fit_transform(X_flat)

Explanation of the code:

• resize scales each image to TARGET_SIZE while preserving aspect ratio and applying anti‑aliasing to avoid aliasing artefacts.
• We reshape the 3‑D array into a 2‑D array where each row is a flattened image, which is required for the simple baseline classifier (e.g. logistic regression).
• The StandardScaler centres and scales each feature (pixel) across the training set, ensuring that all inputs contribute equally to the learning process.

3. Splitting the Data

It is crucial to evaluate the model on data it has never seen. We therefore split the dataset into training and test sets, maintaining the class balance using stratified sampling.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X_normalised, y, test_size=0.2, stratify=y, random_state=42
)

The random_state parameter guarantees reproducibility, while stratify=y ensures that the proportion of healthy and diseased samples is preserved in both splits.

4. Training the Baseline Classifier

For a baseline we employ a simple logistic regression model, which is fast to train and provides a clear benchmark. In practice, more sophisticated models (e.g. ResNet) would be used, but the baseline serves to highlight the added value of deeper architectures.

from sklearn.linear_model import LogisticRegression

Instantiate the model with a fixed random state
baseline = LogisticRegression(max_iter=1000, random_state=42)

Fit the model to the training data
baseline.fit(X_train, y_train)

Key points:

• Setting max_iter to 1000 ensures convergence for small datasets.
• The logistic regression likelihood is maximised via an interior‑point optimiser, producing a probabilistic output.

5. Evaluating Performance

After training we assess the model on the held‑out test set, computing several metrics that are standard in medical image analysis.

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, classification_report

Predict class labels on the test set
y_pred = baseline.predict(X_test)

Compute metrics
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred)

print(f"Accuracy: {accuracy:.3f}")
print(f"Precision: {precision:.3f}")
print(f"Recall: {recall:.3f}")
print(f"F1‑Score: {f1:.3f}")

Display a confusion matrix
cm = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:")
print(cm)

Detailed classification report
print("
Classification Report:")
print(classification_report(y_test, y_pred, target_names=["Healthy", "Diseased"]))

Interpretation of metrics:

• Accuracy indicates the proportion of correctly classified samples.
• Precision measures how many of the predicted diseased samples were truly diseased, thereby reflecting the rate of false positives.
• Recall (sensitivity) tells us how many of the actual diseased samples were identified, which is critical in a medical context where missing a disease is costly.
• F1‑Score is the harmonic mean of precision and recall, providing a single value that balances the two.

Suppose the output is:

Accuracy: 0.842
Precision: 0.815
Recall: 0.876
F1‑Score: 0.844

Confusion Matrix:
[[ 45 7]
[ 12 36]]

Classification Report:
precision recall f1-score support

Healthy 0.82 0.86 0.84 52
Diseased 0.81 0.88 0.84 48

accuracy 0.84 100
macro avg 0.82 0.87 0.84 100
weighted avg 0.82 0.84 0.84 100

This indicates that the baseline performs reasonably well, correctly classifying over 84 % of the samples. The confusion matrix reveals 12 false negatives (healthy samples mislabelled as diseased) and 7 false positives (diseased samples mislabelled as healthy). Depending on the operational requirements, a higher recall might be preferred to minimise missed disease cases.

6. Visualising Misclassifications

Understanding where the model fails is as important as knowing its accuracy. We can plot a few misclassified images to identify common failure modes such as subtle disease symptoms or lighting variations.

import matplotlib.pyplot as plt

Identify indices of misclassifications
mis_idx = np.where(y_test != y_pred)[0]

Plot the first five misclassified images
fig, axes = plt.subplots(1, 5, figsize=(15, 3))
for ax, idx in zip(axes, mis_idx[:5]):
    # Retrieve the original image before flattening
    img = X_resized[(X_train.shape[0] + idx)]
    ax.imshow(img)
    ax.set_title(f"True: {y_test[idx]}, Pred: {y_pred[idx]}")
    ax.axis('off')
plt.tight_layout()
plt.show()

The visual inspection often reveals that the baseline struggles with images exhibiting low contrast or with diseases that present only as small, diffuse spots. These insights guide the next iteration: augmenting the dataset with more difficult examples, adding colour normalisation, or switching to a deeper CNN architecture.

7. Baseline vs. Advanced Models

While a logistic regression baseline offers a quick benchmark, it is limited by its linear decision boundary. In contrast, a convolutional neural network can capture spatial hierarchies and learn disease‑specific texture patterns. A simple transfer‑learning experiment using a pre‑trained ResNet‑18 could improve accuracy to 92 % or higher, at the cost of increased training time and GPU requirements.

When deciding which model to deploy, stakeholders must weigh the trade‑offs between performance, computational resources, and interpretability. The baseline serves as a transparent reference point, while more complex models provide higher accuracy at the expense of explainability.

8. Reproducibility Checklist

For rigorous scientific practice, the following steps should be documented:

1. Set a global random seed (np.random.seed and random_state in scikit‑learn functions).
2. Version‑control the dataset and pre‑processing scripts.
3. Record the exact versions of libraries used (scikit‑learn==1.3.0, skimage==0.19.3, etc.).
4. Save the trained model and scaler objects using joblib.dump for future inference.
5. Log hyperparameters and evaluation metrics in a reproducible format (e.g. CSV or JSON).

Adhering to these practices ensures that the results can be replicated by others and that the model can be audited or updated as new data arrive.

9. Summary

In this worked example we:

• Loaded a wheat leaf image dataset and encoded labels.
• Pre‑processed images through resizing, flattening and normalisation.
• Split the data into stratified training and test sets.
• Trained a logistic regression baseline classifier.
• Computed key performance metrics and inspected misclassifications.
• Compared the baseline to a more advanced CNN approach.
• Outlined best practices for reproducibility.

These steps constitute a complete, end‑to‑end pipeline that can be adapted to other plant disease datasets or imaging modalities. By starting with a simple baseline, we establish a clear benchmark against which more sophisticated models can be objectively evaluated.

10. Data Augmentation Example

To improve robustness we can perform on‑the‑fly augmentation using TensorFlow’s ImageDataGenerator. This example applies random rotations, flips and brightness shifts while training a small CNN.

import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras import layers, models

Define augmentation strategy
datagen = ImageDataGenerator(
    rotation_range=15,
    width_shift_range=0.1,
    height_shift_range=0.1,
    horizontal_flip=True,
    brightness_range=[0.8, 1.2],
    fill_mode='nearest'
)

Prepare a simple CNN
model = models.Sequential([
    layers.InputLayer(input_shape=(128, 128, 3)),
    layers.Conv2D(32, (3, 3), activation='relu'),
    layers.MaxPooling2D(),
    layers.Conv2D(64, (3, 3), activation='relu'),
    layers.MaxPooling2D(),
    layers.Flatten(),
    layers.Dense(64, activation='relu'),
    layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

Fit with augmented data
model.fit(
    datagen.flow(X_train.reshape(-1, 128, 128, 3), y_train, batch_size=32),
    epochs=10,
    validation_data=(X_test.reshape(-1, 128, 128, 3), y_test)
)

Pitfalls: Common Errors and How to Avoid Them

When building a baseline crop disease classifier, the technical journey is often punctuated by subtle mistakes that can erode model performance. A solid understanding of these pitfalls and the strategies to sidestep them is essential for any practitioner aiming to deliver reliable, reproducible results. Below we dissect three of the most pervasive errors: overfitting, data leakage, and misinterpreting evaluation metrics. Each subsection provides concrete examples, explains why the mistake is harmful, and offers pragmatic solutions.

1. Overfitting: Learning the Noise Instead of the Signal

Overfitting occurs when a model captures idiosyncrasies of the training data that do not generalise to new images. This is particularly acute in image classification where high‑capacity architectures like ResNet or DenseNet can memorize even modest datasets.

1. Symptoms: A training accuracy that climbs to 99 % while validation accuracy stagnates around 70 %. The model performs well on the training set but poorly on unseen data.
2. Causes: 1) Insufficient data, 2) Lack of augmentation, 3) Excessive depth or width of the network, 4) Inadequate regularisation.
3. Prevention:
   • Use cross‑validation to gauge stability.
   • Apply data augmentation: random flips, rotations, colour jitter, and random crops.
   • Introduce dropout layers or weight decay (L2 regularisation).
   • Employ early stopping based on validation loss.
4. Practical tip: Monitor the ratio of training to validation loss. A widening gap is a red flag.

2. Data Leakage: Unintended Information Flow

Data leakage is the inadvertent introduction of test information into the training process. In crop disease classification, this can arise from several sources.

• Pre‑processing leakage: Standardising pixel values across the entire dataset before splitting. This shares mean and variance values from test images with the training set.
• Augmentation leakage: Applying augmentation that mixes training and test images (e.g., stitching patches).
• Metadata leakage: Using labels or attributes that are correlated with the target but not part of the image content (e.g., GPS coordinates that inadvertently correlate with disease prevalence).

Leakage inflates performance metrics during development, only to collapse when the model is deployed.

Detecting Leakage

One robust approach is to perform a sanity check: create a dummy model that predicts the mean of the training labels and compare its performance to the real model. Any significant advantage suggests leakage. Also, visualise the distribution of pixel intensities in training versus validation sets; large discrepancies may indicate leakage.

Mitigation Strategy

1. Apply all transformations (normalisation, resizing) inside a pipeline that is fitted exclusively on the training data.
2. Ensure that data augmentation is applied per‑sample and not across splits.
3. Keep metadata separate and evaluate its influence independently.

3. Misinterpreting Evaluation Metrics: Accuracy Is Not Enough

In imbalanced datasets—common in disease detection where healthy examples vastly outnumber diseased ones—accuracy can be misleading. A model that always predicts “healthy” may achieve 90 % accuracy but zero recall for disease.

Key metrics to consider:

For crop disease, recall is often prioritised: missing a disease can lead to crop loss. However, an overly high recall may inflate false positives, burdening farmers with unnecessary treatments.

Choosing the Right Threshold

Many classifiers output probabilities. Default threshold of 0.5 is arbitrary. Plotting the ROC curve or Precision‑Recall curve allows selection of a threshold that balances recall and precision to meet domain constraints.

4. Practical Implementation: Regularisation with Dropout

Below is a minimal example of adding dropout to a convolutional base to combat overfitting. The code uses PyTorch, a popular deep learning framework.

import torch
import torch.nn as nn
import torchvision.models as models

1. Load a pre‑trained ResNet‑18 model
model = models.resnet18(pretrained=True)

2. Replace the final fully‑connected layer to match our classes
num_features = model.fc.in_features
model.fc = nn.Sequential(
    nn.Dropout(p=0.5),          # 3. Dropout with 50% keep probability
    nn.Linear(num_features, 2)  # 4. Binary classification: healthy vs diseased
)

5. Define loss and optimiser
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)

Explanation of the code:

1. The models.resnet18(pretrained=True) call downloads a ResNet‑18 model pre‑trained on ImageNet. This provides a robust feature extractor for our crop images.
2. We replace the original fully‑connected head with a new nn.Sequential block. The original head is unsuitable because its output dimension (1000 classes) does not match our binary problem.
3. Dropout is inserted before the final linear layer. With a probability of 0.5, each neuron’s output is set to zero during training, forcing the network to learn redundant representations and reducing overfitting.
4. The linear layer maps the extracted features to two output scores, one per class.
5. We use cross‑entropy loss, appropriate for multi‑class classification, and the Adam optimiser, which adapts learning rates on a per‑parameter basis.

When training, monitor the training and validation losses. If the gap widens, consider increasing dropout or adding weight decay.

4.1 Alternative Example: Data Augmentation Pipeline in Keras

Below demonstrates a simple yet powerful augmentation strategy using TensorFlow/Keras. This approach is often overlooked but can significantly reduce overfitting.

import tensorflow as tf

datagen = tf.keras.preprocessing.image.ImageDataGenerator(
    rescale=1./255,
    rotation_range=30,
    width_shift_range=0.1,
    height_shift_range=0.1,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    fill_mode='nearest'
)

train_generator = datagen.flow_from_directory(
    'data/train',
    target_size=(224, 224),
    batch_size=32,
    class_mode='binary',
    subset='training'
)

This pipeline automatically resizes, normalises, and augments each image on the fly, ensuring that the model sees a diverse set of examples without the need for manual pre‑processing.

5. Checklist for Robust Model Development

Implement the following steps before publishing a model:

• Data hygiene: Verify that no test images appear in the training set.
• Balanced splits: Use stratified k‑fold cross‑validation to preserve class distribution.
• Augmentation strategy: Apply random transformations that reflect real‑world variability (e.g., lighting changes, occlusions).
• Metric selection: Report at least accuracy, precision, recall, F1‑score, and a confusion matrix.
• Threshold optimisation: Tune the decision threshold on a validation set using a grid search over recall‑precision trade‑offs.
• Model explainability: Visualise class activation maps to confirm that the model focuses on disease symptoms rather than background artefacts.

Adhering to this checklist mitigates the common pitfalls discussed and lays a solid foundation for a trustworthy crop disease classifier.

Further Reading

• Goodfellow, I., Bengio, Y., & Courville, A. (2016) — Deep Learning. A foundational text covering convolutional neural networks for image classification, essential for understanding crop disease detection pipelines.
• He, K., Zhang, X., Ren, S., & Sun, J. (2016) — Deep residual learning for image recognition. Presents ResNet architecture widely used in agricultural disease classifiers for improved accuracy.
• International Plant Protection Convention (IPPC) (2021) — Guidelines for Plant Health Risk Assessment. Provides standardised criteria for disease surveillance that inform the data curation stage of AI workflows.
• Chiu, T., Huang, Y., & Huang, J. (2020) — A survey of machine learning methods for plant disease detection. Reviews state‑of‑the‑art classifiers and dataset challenges relevant to baseline model building.
• Li, Y., Li, Z., & Wang, J. (2022) — Transfer learning for plant disease classification using limited data. Discusses practical transfer‑learning strategies that are often applied before training a baseline CNN.