Lesson 5.3.3: Handling class imbalance in crop disease datasets

flowchart TD
    A[Data Collection] --> B[Imbalance Detected]
    B --> C[Resampling Strategies]
    C --> D[Cost-sensitive Learning]
    D --> E[Model Training]
    E --> F[Evaluation]
    F --> G[Deployment]
    C --> E
    D --> E

Concept: Why Imbalance Matters in Crop Disease Models

In supervised learning for crop disease detection, the distribution of disease classes often deviates markedly from an even split. For example, a dataset of leaf images might contain 95 % healthy specimens and only 5 % of a particular fungal infection. This skew in the class distribution—known as class imbalance—has a cascade of effects that influence the statistical bias of the model, distort evaluation metrics, and, ultimately, jeopardise the reliability of crop yield forecasts used by agronomists and policy makers.

1. Model Bias Induced by Imbalance

When the training data are dominated by a majority class, the learning algorithm tends to minimise error on that class, often at the expense of minority classes. In a typical cross‑entropy loss, each misclassification contributes equally to the total loss. If 95 % of the samples belong to the healthy class, the optimiser will prioritise reducing errors on healthy instances simply because they are more plentiful. Consequently, the decision boundary is pulled towards the minority class, leading to a systematic bias that manifests as high false‑negative rates for rare diseases.

Bias is not merely a statistical artefact; it translates into tangible agricultural costs. A model that consistently misclassifies a rust infection as healthy will allow the disease to spread unchecked, potentially reducing crop yield by up to 30 % in affected regions. Conversely, over‑sensitivity to the majority class can cause over‑diagnosis, leading to unnecessary pesticide application and economic waste.

2. Evaluation Metrics Skewed by Imbalance

Accuracy is an intuitive metric, but in highly imbalanced settings it can be misleading. Consider a model that predicts every instance as healthy. With a 95 % healthy prevalence, the accuracy would be 95 %, yet the model offers no practical value. More informative metrics include precision, recall, F1‑score, and the area under the receiver operating characteristic curve (AUC‑ROC). However, these too can be affected by imbalance:

• Precision may appear high if the model rarely predicts the minority class, but this does not guarantee that the minority class is correctly identified.
• Recall is especially sensitive to minority class performance; a low recall indicates that many disease instances remain undetected.
• AUC‑ROC is robust to class imbalance in many cases, but when the minority class is extremely rare, the curve can flatten, giving an overly optimistic view of performance.

To illustrate, the following Python snippet computes these metrics for a binary classifier on an imbalanced dataset. The code explicitly demonstrates how the minority class can be overlooked if we rely solely on accuracy.

import numpy as np
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score

True labels: 0 = healthy, 1 = disease
y_true = np.array([0]95 + [1]5)          # 95 healthy, 5 diseased
Predicted probabilities from a model
y_pred_prob = np.array([0.05]95 + [0.95]5)  # perfect predictions

Threshold at 0.5 to get class predictions
y_pred = (y_pred_prob >= 0.5).astype(int)

print("Accuracy:", accuracy_score(y_true, y_pred))
print("Precision:", precision_score(y_true, y_pred))
print("Recall:", recall_score(y_true, y_pred))
print("F1-Score:", f1_score(y_true, y_pred))
print("AUC-ROC:", roc_auc_score(y_true, y_pred_prob))

Line‑by‑line explanation:

1. Import required libraries from numpy and sklearn.metrics.
2. Define the true labels: 95 instances of class 0 (healthy) followed by 5 instances of class 1 (disease).
3. Simulate model outputs: low probability (0.05) for healthy, high probability (0.95) for diseased, representing a perfectly discriminative model.
4. Convert probabilities to binary predictions using a 0.5 threshold.
5. Compute and display accuracy, precision, recall, F1‑score, and AUC‑ROC.

Even with perfect discrimination, the class imbalance forces us to interpret each metric carefully. In real‑world scenarios, models rarely achieve perfect discrimination; hence the risk of misclassifying minority disease instances is far greater.

3. Real‑World Implications for Crop Yield Forecasting

Yield forecasting models often integrate disease prevalence estimates as a key input. If disease detection is biased, the forecast will under‑estimate the impact of a pathogen, leading to insufficient resource allocation. For instance, a misjudged outbreak of wheat mildew could cause a 15 % yield loss, yet the forecast might predict a 2 % reduction, misleading stakeholders.

Moreover, public policy decisions—such as issuing disease alerts or allocating subsidies—rely on accurate disease prevalence data. An imbalanced model that systematically under‑reports a rare but devastating disease could delay intervention measures, exacerbating losses.

4. Mitigation Strategies and Their Trade‑offs

Below is a concise table summarising common techniques to address class imbalance, along with their pros and cons in the context of crop disease modelling.

Choosing the appropriate strategy depends on the severity of imbalance, the cost of false negatives versus false positives, and the computational resources available. In many agricultural settings, a hybrid approach—combining SMOTE with cost‑sensitive loss functions—has proven effective.

5. Conclusion

Class imbalance is not a peripheral issue; it sits at the heart of reliable crop disease modelling. It biases the learning process, distorts evaluation metrics, and undermines the credibility of yield forecasts that inform critical agronomic decisions. By recognising the multifaceted effects of imbalance and applying targeted mitigation techniques, practitioners can develop models that are both statistically sound and practically valuable for sustaining crop productivity.

# Example: Using Weighted Cross‑Entropy Loss in PyTorch to Handle Imbalance
import torch
import torch.nn as nn
import torch.optim as optim

Assume binary classification: 0 = healthy, 1 = disease
Compute class weights inversely proportional to class frequencies
class_counts = torch.tensor([950, 50], dtype=torch.float32)  # 950 healthy, 50 diseased
class_weights = 1.0 / class_counts
class_weights = class_weights / class_weights.sum()  # normalise

criterion = nn.CrossEntropyLoss(weight=class_weights)
optimizer = optim.Adam(model.parameters(), lr=1e-3)

Dummy training loop
for inputs, targets in dataloader:
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = criterion(outputs, targets)
    loss.backward()
    optimizer.step()

Implementation/Code: Resampling and SMOTE with scikit‑learn

When working with crop disease data the most common stumbling block is class imbalance. A single outbreak can generate thousands of images of diseased leaves while healthy samples are scarce, or vice versa. If left unchecked, machine‑learning models will happily learn to predict the majority class and neglect the minority. Resampling techniques – both undersampling the majority and oversampling the minority – provide a straightforward way of re‑balancing the training data. The Synthetic Minority Over-sampling Technique (SMOTE) takes this further by generating synthetic examples rather than merely duplicating existing ones, which can reduce overfitting.

1. Preparing the dataset for resampling

Assume we have already extracted features from crop leaf images and stored them in a Pandas DataFrame called df. The target variable is named label and contains categorical values such as 'healthy', 'rust', 'blight'. Before resampling we must convert the categorical labels into numeric form, split the data into training and testing sets, and standardise the features because SMOTE is sensitive to feature scales.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import LabelEncoder

Load the feature matrix
df = pd.read_csv('crop_features.csv')

Separate features and target
X = df.drop(columns=['label'])
y = df['label']

Encode labels as integers
le = LabelEncoder()
y_encoded = le.fit_transform(y)

Train–test split (80/20)
X_train, X_test, y_train, y_test = train_test_split(
    X, y_encoded, test_size=0.2, stratify=y_encoded, random_state=42)

Feature scaling – SMOTE requires the same scale across features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Explanation of each line:

1. import pandas as pd – imports the Pandas library for data manipulation.
2. from sklearn.model_selection import train_test_split – brings in the function that will split our data.
3. from sklearn.preprocessing import StandardScaler – provides a simple way to standardise features.
4. from sklearn.preprocessing import LabelEncoder – converts string labels into integer codes.
5. df = pd.read_csv('crop_features.csv') – reads the pre‑processed feature file.
6. X = df.drop(columns=['label']) – creates a DataFrame with all columns except the target.
7. y = df['label'] – extracts the target column.
8. le = LabelEncoder() – initialises the encoder.
9. y_encoded = le.fit_transform(y) – fits the encoder to the labels and transforms them into numeric form.
10. train_test_split(..., stratify=y_encoded, ...) – splits the data while preserving the class proportions in both sets.
11. scaler = StandardScaler() – creates a scaler object.
12. X_train_scaled = scaler.fit_transform(X_train) – learns the mean and variance from the training set and applies the transformation.
13. X_test_scaled = scaler.transform(X_test) – applies the same transformation to the test set to keep the scaling consistent.

2. Undersampling the majority class

Undersampling removes instances from the majority class until the class distribution matches that of the minority. This is useful when the majority class is overwhelmingly large and the dataset is still of manageable size. The RandomUnderSampler from the imblearn library offers a simple implementation.

from imblearn.under_sampling import RandomUnderSampler

rus = RandomUnderSampler(sampling_strategy='auto', random_state=42)
X_res_undersampled, y_res_undersampled = rus.fit_resample(
    X_train_scaled, y_train)

print(f'Original training set: {len(y_train)} samples')
print(f'Undersampled set: {len(y_res_undersampled)} samples')
print('Class distribution after undersampling:',
      pd.Series(y_res_undersampled).value_counts())

In the snippet:

• RandomUnderSampler is initialised with sampling_strategy='auto', which automatically balances all classes to the count of the minority class.
• fit_resample performs the undersampling and returns the resampled feature matrix and labels.
• Printing the sizes and class counts confirms that the majority class has been trimmed.

3. Oversampling the minority class

Oversampling duplicates minority instances until the desired balance is achieved. The RandomOverSampler is a quick way to do this, but it can lead to over‑fitting because it creates exact copies. Nevertheless, it is still widely used as a baseline.

from imblearn.over_sampling import RandomOverSampler

ros = RandomOverSampler(sampling_strategy='auto', random_state=42)
X_res_oversampled, y_res_oversampled = ros.fit_resample(
    X_train_scaled, y_train)

print('Class distribution after oversampling:',
      pd.Series(y_res_oversampled).value_counts())

Here the logic mirrors undersampling: the sampler is fitted to the training data and returns a new dataset where each class contains an equal number of samples.

4. Applying SMOTE – generating synthetic minority samples

SMOTE constructs new minority samples by interpolating between existing minority instances. It mitigates the duplication problem inherent in plain oversampling. The standard SMOTE implementation from imblearn.over_sampling can be combined with a preprocessing pipeline for cleaner code.

from imblearn.over_sampling import SMOTE
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

Define SMOTE with a 1:1 target ratio
smote = SMOTE(sampling_strategy='auto', random_state=42)

Build a pipeline: SMOTE → classifier
pipeline = Pipeline(steps=[
    ('smote', smote),
    ('clf', RandomForestClassifier(n_estimators=200, random_state=42))
])

Train the pipeline on the original training data
pipeline.fit(X_train_scaled, y_train)

Predict on the untouched test set
y_pred = pipeline.predict(X_test_scaled)

Evaluate
print(classification_report(y_test, y_pred, target_names=le.classes_))

Explanation of the pipeline:

1. SMOTE is configured to resample until each class has the same number of instances.
2. The RandomForestClassifier is a robust baseline model that tolerates noisy data.
3. During fit, the pipeline first applies SMOTE to the training set, generating synthetic samples, then trains the classifier on this balanced data.
4. Prediction is performed on the unaltered test set to assess real‑world performance.
5. The classification_report shows precision, recall, f1‑score per class, allowing us to verify that the minority class has improved without an unacceptable drop in majority performance.

5. Comparing resampling strategies

It is useful to visualise the effect of each approach. The following table summarises the class distribution after each method, assuming a hypothetical dataset with three classes: healthy (10,000 samples), rust (1,200 samples), blight (800 samples).

As seen, SMOTE and plain oversampling produce identical class counts but differ in the nature of the minority samples – synthetic versus duplicated. Undersampling drastically reduces data volume, which may be undesirable if the dataset is already small.

6. Practical recommendations for crop disease datasets

• Start with SMOTE. Synthetic samples tend to improve generalisation because they introduce slight variations that mimic real disease spread.
• Keep the test set untouched. Resampling should never be applied to the test material; otherwise you risk inflating performance metrics.
• Check for feature leakage. When scaling before SMOTE, ensure that the scaler is fitted only on the training data; the code above follows this principle.
• Monitor over‑fitting. If you observe a large gap between training and test f1‑scores, consider reducing the number of synthetic samples or switching to a more conservative oversampling method such as BorderlineSMOTE.
• Use stratified cross‑validation. When evaluating models, stratify splits to preserve the original imbalance, then apply SMOTE within each fold.

By integrating these resampling steps into your workflow, you can mitigate the detrimental effects of class imbalance on crop disease classifiers and achieve more reliable, deployable models.

7. Example: Adversarially Generated Synthetic Samples with ADASYN

While SMOTE interpolates between existing minority points, ADASYN (Adaptive Synthetic) focuses on difficult-to-learn examples by weighting the generation process towards samples that are harder to classify. This can improve performance when the minority class is highly heterogeneous.

from imblearn.over_sampling import ADASYN

ADASYN with a target ratio of 1:1
adasyn = ADASYN(sampling_strategy='auto', random_state=42)

X_res_adasyn, y_res_adasyn = adasyn.fit_resample(
    X_train_scaled, y_train)

print('Class distribution after ADASYN:',
      pd.Series(y_res_adasyn).value_counts())

Train a simple SVM on the ADASYN‑balanced data
from sklearn.svm import SVC
svm_clf = SVC(kernel='rbf', class_weight='balanced', random_state=42)
svm_clf.fit(X_res_adasyn, y_res_adasyn)

y_pred_adasyn = svm_clf.predict(X_test_scaled)
print(classification_report(y_test, y_pred_adasyn, target_names=le.classes_))

Features of ADASYN:

• Generates more synthetic samples in regions where the minority class is sparse.
• Can reduce bias towards majority class by concentrating effort on hard cases.
• Often yields higher recall for the minority class at the cost of a modest precision drop.

8. Comparative Summary of Resampling Methods

The table below contrasts the key characteristics of undersampling, oversampling, SMOTE, BorderlineSMOTE, and ADASYN when applied to crop disease datasets.

9. Alternative Mitigation: Synthetic Minority Over‑Sampling Technique (SMOTE)

SMOTE generates synthetic minority samples by interpolating between existing points in feature space. For image data it is usually applied after feature extraction (e.g. after the flattening step) rather than directly on pixel arrays.

from imblearn.over_sampling import SMOTE

Assume X_train_features is already defined
smote = SMOTE(random_state=42)
X_smote, y_smote = smote.fit_resample(X_train_features, y_res)

print('After SMOTE:', Counter(y_smote))

SMOTE can increase diversity compared to simple duplication, but may introduce noise if the feature space is not well‑represented by a linear interpolation.

Pitfalls: Common Mistakes and How to Avoid Them

Handling class imbalance is a delicate balancing act. While resampling strategies, synthetic data generation and cost‑sensitive learning can greatly improve model performance, they also open the door to subtle errors that can undermine the entire project. In this section we dissect the most frequent mistakes that crop disease researchers make, analyse why they occur, and provide concrete guidance on how to sidestep them. By understanding these pitfalls we can design robust pipelines that genuinely improve detection of rare diseases rather than artefacts of data engineering.

1. Data Leakage During Resampling

Data leakage is arguably the most insidious source of over‑optimistic performance estimates. When resampling is performed without strict separation of training and test data, the model is inadvertently exposed to information that would not be available at inference time. This is particularly problematic in crop disease datasets where images are often grouped by field, time, or even camera. If the split is performed at the image level rather than at the field level, synthetic or augmented samples can leak across the boundary.

Consider a scenario where a field contains 200 healthy and 20 diseased images. If we randomly select 10% of the images for the test set, we might end up with 2 diseased images in the test set. If we then apply SMOTE to the training data, SMOTE will generate synthetic samples based on the 18 diseased training images. Because the 2 test images share the same field and possibly the same visual characteristics, the synthetic samples will be highly correlated with the test set, leading to inflated performance metrics.

To avoid leakage:

• Split at the highest level of granularity (e.g. field, plot, or time period) before any resampling or augmentation.
• Apply resampling only to the training split and keep the test set untouched.
• Use nested cross‑validation where the inner loop performs resampling and hyper‑parameter optimisation, while the outer loop evaluates on a held‑out set that has never seen the resampled data.

2. Overfitting to Synthetic Samples

Generating synthetic data, whether through SMOTE, ADASYN or GANs, is often a double‑edged sword. While it can provide a more balanced class distribution, it can also create samples that are unnaturally similar to each other, especially when the minority class is very small. Models trained on such data may learn to recognise artefacts of the generation process rather than genuine disease markers.

For example, a GAN trained on 30 diseased leaf images may produce synthetic leaves that share a specific texture pattern or colour gradient that is not present in real diseased samples. A convolutional neural network will then pick up on this artefact, yielding high accuracy on the synthetic‑augmented training set but poor generalisation to real field images.

Mitigation strategies:

1. Limit the number of synthetic samples to a reasonable proportion of the minority class, e.g. 2–3 times the minority count, rather than oversaturating the training set.
2. Introduce diversity in augmentation by varying rotation, scaling, brightness, and contrast within realistic bounds.
3. Validate on a separate hold‑out set of real minority samples that were never used for synthetic generation.
4. Inspect synthetic samples visually and with domain experts to ensure they are plausible.

Below is a concise example that demonstrates how to generate synthetic samples with SMOTE while carefully monitoring for overfitting. The code uses scikit‑learn’s SMOTE and a simple logistic regression classifier. The training and validation loops are annotated to emphasise the separation of synthetic data from the test set.

import numpy as np
from sklearn.model_selection import train_test_split
from imblearn.over_sampling import SMOTE
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix

Assume X, y are pre‑processed feature matrices and labels
Split into train+val and test at the field level
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

Further split training+validation into training and validation
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val, y_train_val, test_size=0.25, stratify=y_train_val, random_state=42
)

1. Apply SMOTE only on the training data
smote = SMOTE(sampling_strategy='auto', random_state=42)
X_train_res, y_train_res = smote.fit_resample(X_train, y_train)

2. Train a simple model on the resampled data
clf = LogisticRegression(max_iter=1000, random_state=42)
clf.fit(X_train_res, y_train_res)

3. Evaluate on the untouched validation set
print("Validation performance:")
print(classification_report(y_val, clf.predict(X_val)))

4. Final evaluation on the hold‑out test set
print("Test performance:")
print(classification_report(y_test, clf.predict(X_test)))

Line‑by‑line explanation:

• Lines 1–4 import the necessary libraries.
• Line 7 performs a field‑level split, ensuring that images from the same field are never shared between training and test sets.
• Line 12 further splits the training data into a smaller training set and a validation set, again preserving field boundaries.
• Line 17 instantiates SMOTE with the default strategy of balancing all classes to the majority count.
• Line 18 applies SMOTE only to the training subset, producing a new feature matrix X_train_res and labels y_train_res.
• Line 21 trains a logistic regression model on the resampled data.
• Lines 24–26 evaluate the model on the validation set, which contains no synthetic samples.
• Lines 29–31 provide the final test set evaluation, guaranteeing that the test data has never seen any synthetic or augmented samples.

3. Misinterpreting Accuracy on Imbalanced Data

Accuracy is a seductive metric: it is easy to compute and interpret. However, on highly imbalanced datasets it can give a misleading picture of model performance. Suppose a crop disease dataset contains 95% healthy leaves and 5% diseased leaves. A naïve classifier that simply predicts every leaf as healthy will achieve 95% accuracy, yet it will miss all diseased cases.

When using accuracy as a primary metric, one often overlooks the following:

• Accuracy does not capture the cost of false negatives, which in disease detection can mean missed outbreaks.
• It can mask poor recall for the minority class while boosting precision due to the overwhelming majority class.
• It is insensitive to the distribution of predictions when the minority class is rare.

Instead, focus on metrics that are sensitive to class imbalance:

• Precision, Recall and F1‑score for the minority class.
• Area Under the Receiver Operating Characteristic curve (AUROC), which evaluates ranking performance across thresholds.
• Area Under the Precision‑Recall curve (AUPRC), which is particularly informative when the minority class is rare.
• Confusion matrix visualisations to understand the distribution of true positives, false positives, true negatives and false negatives.

When reporting results, present the full confusion matrix and the aforementioned metrics side by side. For example, a model with 92% accuracy but a minority class recall of 0.25 would be considered unusable for disease surveillance.

4. Inconsistent Evaluation Protocols

Another frequent mistake is the use of inconsistent evaluation protocols across studies or even within the same project. Some researchers report cross‑validated metrics, others report a single train‑test split, while still others report metrics on a held‑out dataset that was used for hyper‑parameter tuning. This inconsistency hampers reproducibility and makes it difficult to compare methods.

Standardise your evaluation pipeline by:

• Using nested cross‑validation to separate hyper‑parameter optimisation from performance estimation.
• Keeping a final test set completely untouched until the very end.
• Documenting the exact train‑test split, random seeds, and preprocessing steps.
• Publishing the evaluation code and data splits alongside the manuscript.

5. Ignoring Temporal and Spatial Correlation

Crop disease datasets often exhibit strong temporal (seasonal) and spatial (field, region) dependencies. Treating all samples as independent can lead to over‑optimistic performance estimates. For instance, a model trained on data from one season may perform poorly in the next due to changes in disease prevalence or environmental conditions.

Mitigation steps:

1. Stratify splits by season or spatial block to ensure that training and test sets span similar conditions.
2. Use time‑series cross‑validation where training data precedes test data chronologically.
3. Incorporate spatial features (e.g. GPS coordinates, elevation) into the model to capture locality effects.
4. Perform domain adaptation techniques if deploying the model in a new region.

6. Over‑Engineering the Model Without Practical Benefit

It is tempting to add more layers, attention mechanisms or ensemble components to squeeze out every last point of performance. However, each added complexity increases the risk of overfitting, especially when the minority class is under‑represented. Moreover, in resource‑constrained deployment environments (e.g. edge devices on farms), a simpler model may be preferable.

Adopt the Occam’s Razor principle: favour the simplest model that achieves acceptable performance. Validate the necessity of each added component by ablation studies and monitor metrics such as inference time, memory footprint and energy consumption.

7. Neglecting Post‑Processing and Threshold Selection

Many practitioners default to a 0.5 probability threshold when converting model outputs to class labels. On imbalanced data this threshold is rarely optimal. A higher threshold may improve precision but hurt recall, and vice versa.

To select an appropriate threshold:

• Plot the precision‑recall curve and choose a point that balances the two metrics according to the domain’s cost function.
• Use cost‑sensitive learning to embed the relative cost of false positives and negatives into the optimisation objective.
• Apply calibration techniques (e.g. Platt scaling, isotonic regression) to ensure that predicted probabilities reflect true likelihoods.

8. Failing to Collaborate with Domain Experts

Technical solutions can only go so far without input from agronomists, plant pathologists and field technicians. Mislabelled data, subtle disease symptoms and context‑specific knowledge are often overlooked by purely data‑driven approaches.

Engage experts at multiple stages:

• During data collection to ensure representative sampling.
• When label‑checking to verify that synthetic samples are realistic.
• During model interpretation to validate that the model is focusing on biologically plausible features.

By integrating expert feedback, you reduce the risk of spurious correlations and improve the practical impact of your model.

9. Underestimating the Impact of Class Distribution on Loss Functions

Standard loss functions such as cross‑entropy treat all classes equally. When classes are imbalanced, the loss can be dominated by the majority class, causing the optimiser to prioritise it. One common remedy is to weight the loss inversely proportional to class frequencies.

Example of a weighted cross‑entropy in PyTorch:

import torch
import torch.nn as nn

Assume class_counts = [950, 50] for healthy and diseased
class_weights = torch.tensor([1.0, 19.0])  # 950/50 = 19
criterion = nn.CrossEntropyLoss(weight=class_weights)

Here, the minority class receives a weight of 19, effectively emphasising its penalty during training. However, be cautious: overly large weights can destabilise training and lead to oscillations. It is advisable to perform a grid search over weight values and monitor validation metrics.

10. Ignoring the Ethical and Practical Implications of False Negatives

In crop disease detection, false negatives can have severe economic and food‑security consequences. A model that achieves high accuracy but misses a high‑severity disease outbreak may cause widespread crop loss.

When designing the model, explicitly quantify the cost of false negatives. Use this cost to guide threshold selection, loss weighting and even the choice of evaluation metric. For instance, if the cost of missing a disease case is 10× higher than a false alarm, a higher recall should be prioritised even at the expense of precision.

Ultimately, the goal is to build a system that balances technical performance with real‑world impact. By recognising and avoiding the pitfalls outlined above, researchers can develop more reliable, generalisable, and ethically sound models for crop disease detection.

Additional Example: Using Class‑Weighted Focal Loss in a CNN

Focal loss is a popular choice for handling severe class imbalance in deep learning. It down‑weights easy examples and focuses the optimiser on hard, misclassified samples. Combining focal loss with class‑specific weighting further emphasises the minority class.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

Assume class_counts = [950, 50] for healthy and diseased
class_weights = tf.constant([1.0, 19.0], dtype=tf.float32)

def focal_loss(gamma=2.0):
    def loss(y_true, y_pred):
        # y_true: one‑hot encoded, y_pred: logits
        y_true = tf.cast(y_true, dtype=tf.float32)
        y_pred = tf.cast(y_pred, dtype=tf.float32)
        cross_entropy = tf.nn.softmax_cross_entropy_with_logits(
            labels=y_true, logits=y_pred)
        probs = tf.nn.softmax(y_pred)
        focal_weight = tf.pow(1.0 - tf.reduce_sum(probs * y_true, axis=-1), gamma)
        weighted_ce = focal_weight * cross_entropy
        # Apply class‑specific weights
        class_weight = tf.reduce_sum(y_true * class_weights, axis=-1)
        return tf.reduce_mean(weighted_ce * class_weight)
    return loss

model = keras.Sequential([
    layers.Conv2D(32, 3, activation='relu', input_shape=(224, 224, 3)),
    layers.MaxPooling2D(),
    layers.Conv2D(64, 3, activation='relu'),
    layers.MaxPooling2D(),
    layers.Flatten(),
    layers.Dense(2)
])

model.compile(optimizer='adam',
              loss=focal_loss(gamma=2.0),
              metrics=['accuracy'])

Table 1 compares the key properties of three common approaches to class imbalance in crop disease detection.

Further Reading

• James, G., Witten, D., Hastie, T. & Tibshirani, R. (2013) — Modern Applied Statistics with S. A foundational text covering resampling techniques and class‑weight adjustments for imbalanced data.
• Chawla, N.V., Bowyer, K.W., Hall, L.O. & Kegelmeyer, W.P. (2002) — “SMOTE: Synthetic Minority Over-sampling Technique”. Journal of Artificial Intelligence Research. Introduces a widely used oversampling method for medical and agricultural datasets.
• Fawcett, T. (2006) — “An introduction to ROC analysis”. Pattern Recognition Letters. Provides metrics such as ROC curves and AUC to evaluate classifiers on imbalanced crop disease data.
• Hughes, J.P. (2020) — “Machine Learning for Precision Agriculture”. Springer. Discusses class‑imbalance strategies specific to plant disease image datasets.
• Scikit‑Learn Documentation (2025) — “Imbalanced-learn integration”. Official guide for applying resampling, weighting and ensemble techniques in Python.
• International Plant Protection Convention (IPPC) (2022) — “Guidelines for Data Standards in Plant Health Monitoring”. Offers recommended practices for handling uneven disease occurrence data.