Lesson 4.1.2: Labeling healthy and diseased crop samples

flowchart TD
    A[Collect Crop Sample] --> B[Observe Physical Symptoms]
    B --> C[Initial Health Assessment]
    C --> D{Is Sample Healthy?}
    D -- Yes --> E[Label as "Healthy"]
    D -- No --> F[Label as "Diseased"]
    E --> G[Record in Database]
    F --> G

Concept: Understanding Sample States

In a hyperspectral imaging workflow, the ability to distinguish between healthy and diseased crop samples is fundamental. The term “sample state” refers to the physiological condition of a plant or plant part at the time of data acquisition. A healthy sample exhibits normal spectral signatures across its spectral range, while a diseased sample shows deviations that correspond to pathological changes such as chlorosis, necrosis, fungal infection or pest damage. Understanding these states is essential for model training, validation and for any downstream decision‑making process in precision agriculture.

1. Physiological Basis of Spectral Signatures

Healthy vegetation normally displays a characteristic “red edge” – a sharp rise in reflectance between roughly 680 nm and 750 nm – and high near‑infrared (NIR) reflectance due to the cellular structure of leaves. This pattern is a consequence of chlorophyll absorption in the visible region and internal leaf scattering in the NIR. When a plant becomes diseased, one or more of these spectral characteristics changes:

• Chlorosis: Loss of chlorophyll leads to increased reflectance in the green and yellow bands, and a reduced depth of the red edge.
• Necrosis: Tissue death causes higher reflectance in the visible region and a dampened NIR peak.
• Fungal infection: Often introduces subtle changes in the mid‑infrared due to altered leaf moisture content.
• Pest damage: Results in irregular spatial patterns of reflectance, often detectable through high‑resolution imaging.

The spectral deviations are not merely aesthetic; they encode biochemical and structural information that can be extracted using statistical or machine‑learning techniques.

2. Acquisition Considerations

To reliably capture the differences between healthy and diseased samples, several acquisition parameters must be controlled:

1. Illumination: Uniform, broadband light is required to avoid artificial spectral features. LED arrays or halogen lamps with calibrated spectra are common choices.
2. Sensor calibration: Radiometric calibration against a white reference panel (e.g., Spectralon) ensures that measured reflectance values are comparable across sessions.
3. Spatial resolution: Higher pixel densities enable the detection of small lesions. For field‑scale surveys, a resolution of 1–5 cm/pixel is typical.
4. Temporal frequency: Repeated scans over time allow monitoring of disease progression and early detection.

In practice, a trade‑off exists between spatial resolution, spectral range, and acquisition speed. A well‑designed protocol balances these factors to maximise diagnostic capability.

3. Spectral Indicators of Disease

Below are some of the most widely used spectral indicators for disease detection:

These indices are simple to compute and provide interpretable metrics that can be fed into classification algorithms.

4. Modelling Healthy vs. Diseased States

Once the spectral data are acquired, the next step is to train a model that can discriminate between the two states. A common approach is to use a supervised machine‑learning classifier such as Random Forest, Support Vector Machine or a shallow neural network. The following Python snippet demonstrates how to prepare the data and train a Random Forest classifier using the scikit‑learn library.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix

1. Load spectral dataset
Assume 'spectra.csv' contains columns: wavelength_450, wavelength_550, ..., wavelength_1100, label
df = pd.read_csv('spectra.csv')

2. Separate features and target
X = df.drop(columns=['label'])
y = df['label']  # 0 = healthy, 1 = diseased

3. Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

4. Initialise Random Forest
rf = RandomForestClassifier(
    n_estimators=200,
    max_depth=None,
    random_state=42,
    n_jobs=-1  # utilise all available cores
)

5. Train the model
rf.fit(X_train, y_train)

6. Predict on test set
y_pred = rf.predict(X_test)

7. Evaluate performance
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

Line‑by‑line explanation:

1. Import libraries: NumPy and Pandas for data handling; scikit‑learn modules for model training and evaluation.
2. Load dataset: The CSV is expected to contain one column per wavelength (e.g., wavelength_450) and a label column indicating health status.
3. Separate features and target: Features are all spectral bands; the target is the health label.
4. Train‑test split: 80 % of the data for training, 20 % for testing; stratify=y preserves class proportions.
5. Initialise Random Forest: 200 trees are sufficient for most small to medium datasets; n_jobs=-1 makes use of all CPU cores for speed.
6. Train the model: The fit method learns decision rules from the training data.
7. Predict: Generate predictions for the unseen test set.
8. Evaluate: The confusion matrix shows true positives, false positives, etc.; the classification report provides precision, recall and F1‑score for each class.

In a production setting, one would typically perform cross‑validation, hyper‑parameter tuning and feature importance analysis to refine the model further.

5. Interpreting Results in the Context of Disease Severity

Beyond binary classification, it is often useful to quantify disease severity. One approach is to compute a continuous severity score by regressing the spectral data onto a ground‑truth severity scale (e.g., 0–100 %). A simple linear regression model can serve as a proof of concept:

from sklearn.linear_model import LinearRegression

Assume 'severity' column holds ground‑truth severity percentages
y_severity = df['severity']

Train regression model
reg = LinearRegression()
reg.fit(X, y_severity)

Predict severity on new sample
pred_severity = reg.predict(new_sample_features)

Such severity estimates can be visualised as heat‑maps over field plots, enabling precision management decisions like targeted pesticide application.

6. Common Pitfalls and Mitigation Strategies

• Spectral overlap: Some diseases produce spectral changes that are subtle and may overlap with abiotic stress signals (e.g., drought). Incorporating ancillary data such as temperature or soil moisture can help disambiguate.
• Calibration drift: Over long campaigns, sensor calibration may shift. Regular calibration checks against a reference panel are mandatory.
• Class imbalance: Diseased samples are often rarer than healthy ones. Techniques such as SMOTE or class‑weight adjustments in the loss function can alleviate bias.
• Spatial autocorrelation: Adjacent pixels are not independent. Employ spatially aware sampling strategies or incorporate spatial features into the model.

7. Future Directions

Emerging techniques such as deep learning with convolutional neural networks (CNNs) are beginning to exploit the full hyperspectral cube, capturing both spectral and spatial context simultaneously. Transfer learning from large plant image datasets may reduce the annotation burden. Additionally, integrating hyperspectral data with UAV‑based thermal imaging and LiDAR can provide a multi‑modal view of plant health, further improving diagnostic accuracy.

In summary, recognising and quantifying the spectral manifestations of healthy and diseased crop states is a multidisciplinary endeavour that blends plant physiology, optics, data science and agronomy. Mastery of these concepts enables the design of robust, scalable diagnostic pipelines that can be deployed in real‑world farming operations.

# Principal Component Analysis (PCA) for dimensionality reduction before classification
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

Build a pipeline: scaling -> PCA -> RandomForest
pipeline = Pipeline([
    ('scaler', StandardScaler()),          # centre and scale data
    ('pca', PCA(n_components=0.95)),       # retain 95% variance
    ('rf', RandomForestClassifier(n_estimators=200, random_state=42))
])

Fit pipeline on training data
pipeline.fit(X_train, y_train)

Predict and evaluate
y_pred_pca = pipeline.predict(X_test)
print(confusion_matrix(y_test, y_pred_pca))
print(classification_report(y_test, y_pred_pca))

Implementation/Code: Automating Label Assignment

In this section we walk through a concrete, reproducible workflow that takes a raw CSV of crop images, assigns labels indicating whether a plant is healthy or diseased, and writes the enriched data back to disk. The code is intentionally written in a way that novices can follow, yet it remains robust enough for production use. We assume you have a baseline dataset with a column called image_path that points to the location of each image file. The goal is to add a label column that contains either healthy or diseased, possibly with an intermediate confidence score if you are using a simple thresholding approach.

Step 1: Importing the required libraries

We rely on the standard data manipulation stack: pandas for tabular data, os for path handling, and numpy for numerical operations. If you plan to use a pre‑trained image classifier, you would also import torch or tensorflow; however, for the scope of this lesson we keep the classification logic simple.

import pandas as pd
import numpy as np
import os
from pathlib import Path

Each line performs a straightforward task:

• import pandas as pd brings in the pandas library under the alias pd, the conventional shortcut used across the data science community.
• import numpy as np imports NumPy, which we use for vectorised operations such as calculating a simple score from image metadata.
• import os gives us access to operating‑system utilities, notably os.path for joining paths in a platform‑independent way.
• from pathlib import Path provides an object‑oriented interface to paths, which is often more readable than raw string manipulation.

Step 2: Loading the raw CSV into a DataFrame

We use pandas.read_csv to read the file. The dtype parameter ensures that the image_path column is treated as a string, which can prevent subtle bugs when the CSV contains numeric values that are actually part of a path.

raw_df = pd.read_csv(
    'data/raw_crop_images.csv',
    dtype={'image_path': str}
)

Explanation of each component:

• 'data/raw_crop_images.csv' is the relative path to the source file. Adjust this path if your data lives elsewhere.
• dtype={'image_path': str} forces the image_path column to be parsed as text.

Step 3: Defining a simple rule‑based classifier

In many practical pipelines, a lightweight rule‑based heuristic can be sufficient to flag obvious cases. For example, if the file name contains the word “virus”, we automatically label the plant as diseased. Otherwise we flag it as healthy. This is a placeholder; you can replace it with a machine‑learning model later.

def rule_based_label(row):
    "
    Assigns a label based on the presence of key substrings in the file name.
    The function returns a tuple (label, confidence), where confidence is a
    float between 0 and 1 indicating the certainty of the assignment.
    "
    filename = Path(row['image_path']).stem.lower()

    # Check for disease keywords
    disease_keywords = ['blight', 'mildew', 'rust', 'virus']
    for keyword in disease_keywords:
        if keyword in filename:
            # If a keyword is found, classify as diseased with high confidence
            return ('diseased', 0.9)

    # Default to healthy with medium confidence
    return ('healthy', 0.7)

Line‑by‑line:

• The function accepts a single row of the DataFrame, which is a pandas Series object.
• Path(row['image_path']).stem.lower() extracts the file name without extension and normalises it to lower case to avoid case‑sensitive mismatches.
• A list of disease_keywords defines the substrings that trigger a diseased label.
• The for loop iterates over each keyword and checks if it is a substring of the file name.
• Upon a match, the function returns a tuple with the label and a confidence score of 0.9.
• If no keyword matches, the function returns ('healthy', 0.7), signalling a default healthy assignment.

Step 4: Applying the classifier to the entire DataFrame

We use DataFrame.apply to vectorise the operation across all rows. The result is two new columns: label and confidence.

label_results = raw_df.apply(rule_based_label, axis=1, result_type='expand')
raw_df['label'] = label_results[0]
raw_df['confidence'] = label_results[1]

Explanation:

• axis=1 tells apply to operate row‑wise.
• result_type='expand' ensures that the tuple returned by rule_based_label is split into separate columns.
• The left hand side, label_results, is a DataFrame with two columns: the first containing the labels, the second the confidences.
• These columns are then added to the original raw_df under the names label and confidence.

Step 5: Quality assurance – flagging uncertain predictions

Even with a simple heuristic, it is prudent to surface predictions that fall below a chosen confidence threshold. We create a boolean column needs_review that marks such rows.

confidence_threshold = 0.75
raw_df['needs_review'] = raw_df['confidence'] < confidence_threshold

Here we set the threshold to 0.75. Any row with a confidence lower than this will have needs_review set to True, signalling that a human expert should verify the label.

Step 6: Exporting the enriched dataset

Once the labels are assigned, we write the DataFrame back to a new CSV file. The index=False argument removes the default integer index column, producing a cleaner file that matches the structure expected by downstream tools.

output_path = Path('data/processed_crop_labels.csv')
raw_df.to_csv(output_path, index=False)
print(f'Label assignment complete. Results written to {output_path}')

Breakdown:

• output_path = Path('data/processed_crop_labels.csv') defines the destination path.
• raw_df.to_csv(output_path, index=False) writes the DataFrame.
• The final print statement provides a quick runtime confirmation.

Step 7: Packaging the workflow into a reusable function

For maintainability, wrap the entire pipeline in a single function that accepts the input and output paths. This encapsulation facilitates unit testing and future extension.

def automate_label_assignment(input_csv: str, output_csv: str, threshold: float = 0.75):
    "
    Reads raw crop image metadata, assigns healthy/diseased labels,
    flags low‑confidence predictions, and writes the enriched data.

    Parameters
    ----------
    input_csv : str
        Path to the CSV containing at least an 'image_path' column.
    output_csv : str
        Destination path for the processed CSV.
    threshold : float, optional
        Confidence threshold below which rows are marked for review.
    "
    df = pd.read_csv(input_csv, dtype={'image_path': str})
    label_results = df.apply(rule_based_label, axis=1, result_type='expand')
    df['label'] = label_results[0]
    df['confidence'] = label_results[1]
    df['needs_review'] = df['confidence'] < threshold
    df.to_csv(output_csv, index=False)
    print(f'Processed {len(df)} records. Output written to {output_csv}')

Key aspects:

• Type annotations (e.g., input_csv: str) improve readability and aid static analysis tools.
• The function is self‑contained: it imports only what it needs and does not rely on global variables.
• Docstrings follow the NumPy style, providing clear guidance for users.

Step 8: Running the pipeline

Invoke the function with the paths to your raw data and desired output location. If you are working in an interactive environment (e.g., Jupyter), simply call the function in a cell.

automate_label_assignment(
    input_csv='data/raw_crop_images.csv',
    output_csv='data/processed_crop_labels.csv',
    threshold=0.75
)

Upon execution, you should see a console message confirming the number of processed records and the location of the output file.

Step 9: Extending to a machine‑learning classifier

When you replace the rule‑based function with a trained model, the only change required is the rule_based_label function. For instance, if you have a PyTorch model that outputs a probability for the diseased class, you could rewrite the function as follows:

def ml_based_label(row, model, transform):
    "
    Uses a PyTorch model to predict the disease probability.

    Parameters
    ----------
    row : pandas.Series
        Row containing the image path.
    model : torch.nn.Module
        Pre‑trained classification model.
    transform : torchvision.transforms.Compose
        Image preprocessing pipeline.

    Returns
    -------
    tuple: (label, confidence)
    "
    from PIL import Image
    import torch

    image = Image.open(row['image_path']).convert('RGB')
    tensor = transform(image).unsqueeze(0)  # Add batch dimension
    with torch.no_grad():
        logits = model(tensor)
        probs = torch.softmax(logits, dim=1)
    disease_prob = probs[0, 1].item()  # Assuming index 1 is 'diseased'

    label = 'diseased' if disease_prob >= 0.5 else 'healthy'
    return (label, disease_prob)

Notice how the structure of the pipeline remains unchanged: you still call df.apply with the new function, and the rest of the code handles the DataFrame manipulation. This modular design means you can swap out the classification logic without touching the data‑I/O or quality‑check components.

Step 10: Logging and reproducibility

For audit trails, it is good practice to log the version of the algorithm and the random seed used (if any). Append a simple log entry to a text file after processing.

import datetime

log_entry = f"{datetime.datetime.now()}: Processed {len(df)} images using rule_based_label.
"
with open('logs/label_assignment.log', 'a') as log_file:
    log_file.write(log_entry)

Adding a timestamp and record count gives future users a quick snapshot of when the data were labelled and how many entries were involved.

Conclusion

By following the steps outlined above, you obtain a clean, repeatable workflow that transforms raw image metadata into a labelled dataset ready for downstream analysis or model training. The code is deliberately straightforward, prioritising clarity so that students can adapt it to their own projects, whether they choose to stick with rule‑based logic or integrate sophisticated machine‑learning models later on. The modular structure ensures that each component can be tested in isolation, facilitating maintenance and future enhancements.

Additional Example: Vectorised String Matching Without apply

For larger datasets, the apply method can become a bottleneck. A fully vectorised approach uses pandas string methods to avoid Python loops entirely.

def vectorised_label(df: pd.DataFrame) -> pd.DataFrame:
    ""
    Assigns labels using vectorised string operations.
    Returns a new DataFrame with 'label' and 'confidence' columns.
    ""
    # Normalise file names to lowercase
    filenames = df['image_path'].apply(lambda p: Path(p).stem.lower())

    # Build a mask for each disease keyword
    disease_keywords = ['blight', 'mildew', 'rust', 'virus']
    disease_mask = filenames.str.contains('|'.join(disease_keywords), na=False)

    # Apply labels and confidences
    df = df.copy()
    df['label'] = np.where(disease_mask, 'diseased', 'healthy')
    df['confidence'] = np.where(disease_mask, 0.9, 0.7)
    return df

This version eliminates the explicit loop over rows and can offer a noticeable speedup for very large CSV files.

Comparison Table: Rule‑Based Apply vs Vectorised Approach

Worked Example: A Complete Annotation Workflow

In this section we follow a single crop image from acquisition to a fully‑labelled dataset that can be fed directly into a convolutional neural network. The example uses a fruit‑tree orchard, where the goal is to distinguish healthy leaves, leaves with fungal infection, and leaves with insect damage. The workflow is organised into five stages: Image Capture, Pre‑processing, Annotation, Quality Assurance, and Dataset Export. Each stage is broken down into actionable steps and illustrated with code where appropriate.

1. Image Capture

High‑resolution images are essential because the model must learn subtle colour changes and texture differences. In practice, a DSLR camera or a high‑resolution smartphone camera mounted on a tripod is used. For consistency, the following parameters should be recorded:

• ISO setting
• Exposure time
• Focal length
• Lighting condition (natural, flash, diffused)
• Camera model and firmware version

These metadata are stored in the image’s EXIF header and can be extracted later with libraries such as Pillow or ExifRead. Example code to read EXIF tags is shown below.

from PIL import Image
import PIL.ExifTags as ExifTags

def get_exif(filepath):
    "Return a dictionary of EXIF tags for a given image file."
    with Image.open(filepath) as img:
        exif = img._getexif() or {}
    exif_dict = {
        ExifTags.TAGS.get(tag, tag): value
        for tag, value in exif.items()
    }
    return exif_dict

<h1>Example usage</h1>
exif_info = get_exif('leaf_001.jpg')
print(exif_info['ExposureTime'])

Line‑by‑line explanation:

1. Import the Pillow Image module and the ExifTags mapping.
2. Define get_exif which takes a file path and opens the image.
3. Use _getexif() to obtain the raw EXIF dictionary; fallback to an empty dict if none.
4. Transform the numeric tag keys into human‑readable names using ExifTags.TAGS.
5. Return the cleaned dictionary.
6. Demonstrate the function on a sample image and print the exposure time.

Storing this information in a CSV alongside the image path allows later verification that all images were taken under similar conditions.

2. Pre‑processing

Raw images often contain background noise, varying illumination, and colour casts. Pre‑processing normalises these factors and can improve annotation accuracy:

1. Colour Normalisation: Use histogram matching to a reference image to reduce colour variance.
2. Contrast Enhancement: Apply CLAHE (Contrast Limited Adaptive Histogram Equalisation) to bring out fine details.
3. Noise Reduction: A bilateral filter preserves edges while removing speckle.
4. ROI Extraction: Crop the image to the bounding box of the leaf if the leaf occupies only a portion of the frame.

Below is a short snippet that implements CLAHE and a bilateral filter using OpenCV.

import cv2
import numpy as np

def preprocess(image_path, clip_limit=2.0, tile_grid_size=(8, 8)):
    "Apply CLAHE and bilateral filtering to an image."
    # Load image in BGR
    img = cv2.imread(image_path)
    # Convert to LAB colour space for CLAHE on lightness channel
    lab = cv2.cvtColor(img, cv2.COLOR_BGR2LAB)
    l, a, b = cv2.split(lab)
    clahe = cv2.createCLAHE(clipLimit=clip_limit, tileGridSize=tile_grid_size)
    cl = clahe.apply(l)
    merged = cv2.merge((cl, a, b))
    # Convert back to BGR
    enhanced = cv2.cvtColor(merged, cv2.COLOR_LAB2BGR)
    # Bilateral filter: diameter 9, sigmaColor 75, sigmaSpace 75
    denoised = cv2.bilateralFilter(enhanced, d=9, sigmaColor=75, sigmaSpace=75)
    return denoised

<h1>Example usage</h1>
processed_img = preprocess('leaf_001.jpg')
cv2.imwrite('leaf_001_processed.jpg', processed_img)

Line‑by‑line explanation:

1. Import OpenCV and NumPy.
2. Define preprocess with default CLAHE parameters.
3. Read the image in BGR format.
4. Convert to LAB colour space to isolate the lightness channel.
5. Split LAB into L, A, B components.
6. Instantiate a CLAHE object with the clip limit and tile grid size.
7. Apply CLAHE to the L channel.
8. Merge the enhanced L channel back with the original A and B channels.
9. Convert the LAB image back to BGR.
10. Apply a bilateral filter to reduce noise while preserving edges.
11. Return the processed image.
12. Save the processed image for later annotation.

3. Annotation

Annotation is the most labour‑intensive part of the pipeline. We use a polygonal annotation tool that allows the annotator to trace leaf boundaries and assign one of three classes. Two key concepts make the workflow efficient:

• Template‑Based Annotation: If the leaves are of similar shape, a single reference polygon can be scaled and rotated to fit each leaf, reducing manual effort.
• Colour‑Based Segmentation Assistance: An initial automatic segmentation using k‑means clustering can provide a rough mask that the annotator refines.

Below is a minimalistic script that performs k‑means clustering on the pre‑processed image and outputs a binary mask. This mask can be imported into the annotation tool.

import cv2
import numpy as np

def kmeans_segmentation(image_path, k=2):
    "Segment the image into k clusters using k‑means."
    img = cv2.imread(image_path)
    # Reshape to a 2D array of pixels
    pixel_values = img.reshape((-1, 3))
    pixel_values = np.float32(pixel_values)
    # Define criteria and apply kmeans
    criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 0.2)
    _, labels, centres = cv2.kmeans(pixel_values, k, None, criteria, 10, cv2.KMEANS_RANDOM_CENTERS)
    # Convert centres to 8-bit values
    centres = np.uint8(centres)
    # Flatten labels
    labels = labels.flatten()
    # Create segmented image
    segmented = centres[labels]
    segmented_image = segmented.reshape(img.shape)
    return segmented_image

<h1>Generate a binary mask for the leaf region</h1>
segmented = kmeans_segmentation('leaf_001_processed.jpg', k=3)
cv2.imwrite('leaf_001_mask.png', segmented)

Once the mask is loaded into the annotation tool, the annotator can quickly adjust the polygon boundaries. Each polygon is stored in COCO JSON format, which includes the following fields:

4. Quality Assurance

To maintain dataset integrity, a multi‑level review process is implemented:

1. Automated Consistency Checks: Scripts verify that every polygon has a valid area (> 0) and that bounding boxes fall within image boundaries.
2. Cross‑Annotator Validation: At least two annotators label each image independently. Discrepancies are highlighted using an overlay tool and resolved by a senior annotator.
3. Statistical Summary: Compute class distribution, average polygon area, and inter‑annotator agreement (Cohen’s Kappa). This helps identify bias or systematic errors.

Example of a simple consistency check:

import json

def validate_annotations(json_path, image_width, image_height):
    "Check that all polygons are within image bounds."
    with open(json_path) as f:
        data = json.load(f)
    errors = 0
    for ann in data['annotations']:
        bbox = ann['bbox']
        x, y, w, h = bbox
        if x < 0 or y < 0 or x + w > image_width or y + h > image_height:
            errors += 1
            print(f"Annotation {ann['id']} out of bounds.")
    if errors == 0:
        print("All annotations valid.")
    else:
        print(f"{errors} out of bounds annotations found.")

<h1>Run validation</h1>
validate_annotations('annotations.json', 4000, 3000)

5. Dataset Export

After QA, the final dataset is split into training, validation, and test sets. A common split is 70/15/15, but for small datasets, k‑fold cross‑validation may be preferable. The export script generates the following structure:

• train/, val/, test/ – directories containing images.
• annotations_train.json, annotations_val.json, annotations_test.json – COCO‑formatted annotation files.
• Optional: per‑split metadata CSVs summarising class counts.

Example export snippet:

import os
import shutil
import json
import random

def split_dataset(image_dir, ann_file, split=(0.7, 0.15, 0.15)):
    "Split dataset into train/val/test and copy files."
    images = [f for f in os.listdir(image_dir) if f.lower().endswith('.jpg')]
    random.shuffle(images)
    n = len(images)
    train_cut = int(n * split[0])
    val_cut = int(n * (split[0] + split[1]))
    splits = {
        'train': images[:train_cut],
        'val': images[train_cut:val_cut],
        'test': images[val_cut:]
    }
    # Load annotations
    with open(ann_file) as f:
        ann_data = json.load(f)
    # Map image filename to id
    id_map = {ann['file_name']: ann['image_id'] for ann in ann_data['images']}
    for split_name, split_imgs in splits.items():
        os.makedirs(f'output/{split_name}', exist_ok=True)
        split_ann = {
            'images': [],
            'annotations': [],
            'categories': ann_data['categories']
        }
        for img_name in split_imgs:
            shutil.copy(os.path.join(image_dir, img_name), f'output/{split_name}/{img_name}')
            split_ann['images'].append(ann_data['images'][id_map[img_name]-1])
        # Filter annotations
        for ann in ann_data['annotations']:
            if ann['image_id'] in [id_map[img] for img in split_imgs]:
                split_ann['annotations'].append(ann)
        with open(f'output/{split_name}/annotations.json', 'w') as out_f:
            json.dump(split_ann, out_f)
    print("Dataset split complete.")

<h1>Run the split</h1>
split_dataset('images', 'annotations.json')

Line‑by‑line explanation:

1. Import required modules.
2. Define split_dataset with default 70/15/15 split.
3. List all JPEG files in the image directory.
4. Shuffle the list to randomise distribution.
5. Calculate cut‑points for each split.
6. Create a dictionary of split subsets.
7. Load the COCO annotation JSON.
8. Create a mapping from filename to image ID for quick lookup.
9. Iterate over each split, create destination directories, copy images, and build split‑specific annotation objects.
10. Write the split annotation file.
11. Indicate completion.

At this stage, the dataset is ready for ingestion by a deep‑learning framework such as PyTorch or TensorFlow. The annotated polygons, combined with the pre‑processing steps, provide a robust foundation for training a model that can accurately classify leaf health status in real‑time orchard monitoring systems.

Additional Enrichment: Data Augmentation Script

To increase the diversity of the training set, a simple augmentation pipeline is often applied. Below is an example that randomly flips, rotates and colour‑shifts images and synchronises the polygon annotations accordingly.

import albumentations as A
from albumentations.pytorch import ToTensorV2
import cv2
import json

def augment_image_and_mask(img_path, mask_path, aug_name='aug'):
    # Load image and mask
    img = cv2.imread(img_path)
    mask = cv2.imread(mask_path, cv2.IMREAD_GRAYSCALE)

    # Define augmentation
    aug = A.Compose([
        A.RandomRotate90(p=0.5),
        A.HorizontalFlip(p=0.5),
        A.VerticalFlip(p=0.5),
        A.RandomBrightnessContrast(p=0.2),
        A.HueSaturationValue(p=0.2),
        A.Resize(512, 512),
        ToTensorV2()
    ], additional_targets={'mask': 'mask'})

    augmented = aug(image=img, mask=mask)
    aug_img = augmented['image']
    aug_mask = augmented['mask']

    # Save augmented files
    cv2.imwrite(f'aug/{aug_name}_img.png', aug_img.numpy().transpose(1, 2, 0))
    cv2.imwrite(f'aug/{aug_name}_mask.png', aug_mask.numpy())

    # Update COCO annotation (simplified example)
    # Here we assume mask is a single channel binary mask
    # and we re‑extract contours to create new polygons
    contours, _ = cv2.findContours(aug_mask.numpy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    new_polygons = []
    for cnt in contours:
        cnt = cnt.squeeze()
        if cnt.ndim == 2:
            new_polygons.append(cnt.tolist())
    # Append to a new JSON (omitted for brevity)
    return new_polygons

Example usage
augment_image_and_mask('leaf_001_processed.jpg', 'leaf_001_mask.png', 'leaf_001_aug')

Pitfalls: Common Labelling Errors and Mitigation

Accurate labelling of crop samples is the foundation of any reliable plant pathology study. A single mis‑labelled specimen can cascade into erroneous conclusions, wasted resources, and compromised data integrity. This section dissects the most frequent mistakes encountered in field and laboratory settings, then presents pragmatic countermeasures. The discussion is grounded in real‑world scenarios and reinforced with a concise Python snippet that demonstrates how to programme a quick sanity check for label consistency.

1. Ambiguous or Incomplete Identifiers

One of the most common pitfalls is the use of vague labels such as “Sample A” or “Crop 1”. While convenient initially, these identifiers obliterate essential contextual information. Without a unique, descriptive tag, you cannot reliably trace a sample back to its source, harvest time, or treatment condition.

• Consequence: Data cannot be cross‑referenced with environmental records or treatment logs.
• Mitigation: Adopt a structured naming convention that encodes critical metadata. For example: TX2024-07-15-LOC1-REP2 could represent a treatment (TX), date (2024‑07‑15), location (LOC1), and replicate (REP2).

2. Inconsistent Colour‑Code Usage

Colour‑coding is a popular visual aid for quick identification. However, inconsistencies arise when the same colour is assigned to different categories across batches, or when colours are mixed with other visual cues.

• Consequence: Misinterpretation during sample sorting or data entry.
• Mitigation: Standardise colour palettes and document them in a master reference sheet. Use colour‑blind friendly palettes and ensure printed labels include both colour and textual identifiers.

3. Human Transcription Errors

Manual transcription from field notes to lab notebooks or digital spreadsheets is fraught with typographical mistakes. A missing hyphen or an inverted digit can render a label unreadable.

• Consequence: Duplicate entries, loss of traceability.
• Mitigation: Employ barcode or QR‑code scanners to minimise manual entry. When manual input is unavoidable, double‑entry verification or a third‑party check can catch errors early.

4. Overwriting or Rerunning Labels

During sample preparation, labels may be inadvertently overwritten with new information. This is especially problematic when a sample undergoes multiple stages (e.g., surface sterilisation, inoculation, incubation).

• Consequence: Loss of original provenance data.
• Mitigation: Use permanent, tamper‑evident labels (e.g., laminated stickers) and maintain a change‑log that records every modification, including the date, reason, and author.

5. Mixing Sample Types on the Same Plate

In high‑throughput studies, different species or cultivars may be processed on the same microtitre plate or slide. Without clear demarcation, one sample can be mistaken for another during imaging or analysis.

• Consequence: Mixed data sets, contaminating statistical analyses.
• Mitigation: Physically segregate sample types and label each container explicitly. Use a colour‑coded system that is distinct from the sample identifier.

6. Failure to Log Environmental Conditions

Environmental variables such as temperature, humidity, and light exposure are often recorded separately from the sample label. When these conditions are not synchronised with the sample identifier, it becomes difficult to attribute phenotypic differences to environmental causes.

• Consequence: Confounded experimental results.
• Mitigation: Integrate environmental metadata into the sample ID or attach a secondary tag that references a master environmental log.

7. Inadequate Training and SOPs

Even the most robust labelling protocol is only as good as its implementation. Workers who are not adequately trained may overlook subtle guidelines, leading to inconsistent practices.

• Consequence: Systematic bias in data collection.
• Mitigation: Develop comprehensive Standard Operating Procedures (SOPs) and conduct regular refresher training sessions. Include a quick‑reference cheat sheet next to the labelling station.

8. Software and Database Mismatches

When sample metadata is entered into a laboratory information management system (LIMS), mismatches between the database schema and the physical labelling format can cause data loss or corruption.

• Consequence: Unreliable data, difficulty in exporting for analysis.
• Mitigation: Design the database schema in parallel with the labelling convention. Use field validation rules to enforce consistency (e.g., regex patterns for ID formats).

9. Neglecting Batch Identification

Samples from different batches (e.g., different harvests or experimental runs) are often labelled identically, leading to cross‑batch contamination in data analysis.

• Consequence: Inaccurate assessment of batch effects.
• Mitigation: Include a batch number or date stamp in the label. A master batch register should be maintained to track all samples belonging to each batch.

10. Ignoring Regulatory and Ethical Requirements

In some jurisdictions, crop samples may be subject to biosecurity or intellectual property regulations. Failure to label samples in accordance with these laws can result in legal ramifications.

• Consequence: Legal penalties, loss of funding.
• Mitigation: Familiarise yourself with local regulations and embed compliance checks into the labelling workflow. Use a compliance checklist that must be ticked before sample release.

Practical Implementation: A Python Sanity‑Check Script

Below is a small Python snippet that demonstrates how to validate a batch of sample labels against a predefined naming scheme. The script checks for the presence of all required components, verifies date formats, and flags duplicate IDs.

import re
from collections import Counter

Sample list – in practice this would be read from a CSV or database
sample_ids = [
    "TX2024-07-15-LOC1-REP1",
    "TX2024-07-15-LOC1-REP2",
    "TX2024-07-15-LOC2-REP1",
    "TX2024-07-15-LOC1-REP1",  # duplicate
    "TX2024-07-32-LOC1-REP3",  # invalid date
    "TX2024-07-15-LOC1"        # missing replicate
]

Define a regex pattern for the expected ID format
pattern = re.compile(r"^TX\d{4}-\d{2}-\d{2}-LOC\d+-REP\d+$")

def validate_ids(ids):
    errors = []
    seen = Counter()
    for id_ in ids:
        # Check format
        if not pattern.match(id_):
            errors.append(f"Format error: {id_}")
            continue
        # Check date validity (rudimentary)
        year, month, day = map(int, id_.split("-")[1:4])
        try:
            import datetime
            datetime.date(year, month, day)
        except ValueError:
            errors.append(f"Invalid date: {id_}")
        seen[id_] += 1
    # Detect duplicates
    duplicates = [id_ for id_, count in seen.items() if count > 1]
    if duplicates:
        errors.extend([f"Duplicate ID: {id_}" for id_ in duplicates])
    return errors

Run validation
validation_errors = validate_ids(sample_ids)
for err in validation_errors:
    print(err)

Line‑by‑line explanation:

1. import re – imports the regular expressions module for pattern matching.
2. from collections import Counter – provides a convenient way to count occurrences of each ID.
3. sample_ids = [...] – a placeholder list of sample identifiers; in real use, data would be read from an external source.
4. pattern = re.compile(r"^TX\d{4}-\d{2}-\d{2}-LOC\d+-REP\d+$") – compiles a regex that enforces the structure: TX prefix, four‑digit year, two‑digit month, two‑digit day, location code, and replicate number.
5. def validate_ids(ids): – defines a function to perform validation.
6. errors = [] – initialises a list to collect error messages.
7. seen = Counter() – initialises a counter to track duplicates.
8. for id_ in ids: – iterates over each sample ID.
9. if not pattern.match(id_): – checks if the ID matches the expected format.
10. errors.append(f"Format error: {id_}") – records a format error if the match fails.
11. continue – skips further checks for this ID.
12. year, month, day = map(int, id_.split("-")[1:4]) – extracts numeric components of the date.
13. try: import datetime; datetime.date(year, month, day) – attempts to construct a valid date object; raises ValueError if the date is invalid.
14. except ValueError: errors.append(f"Invalid date: {id_}") – records an error if the date is not valid.
15. seen[id_] += 1 – increments the count for this ID.
16. duplicates = [id_ for id_, count in seen.items() if count > 1] – identifies IDs that appear more than once.
17. if duplicates: errors.extend([f"Duplicate ID: {id_}" for id_ in duplicates]) – appends duplicate errors to the list.
18. return errors – returns the collected errors.
19. validation_errors = validate_ids(sample_ids) – runs validation on the sample list.
20. for err in validation_errors: print(err) – outputs each error message.

This script can be integrated into a laboratory workflow to automatically flag problematic labels before samples are processed further. By catching format violations, date errors, and duplicates early, the laboratory mitigates the risk of downstream analytical failures.

Alternative Validation with Pandas

For larger datasets, a vectorised approach using pandas can be more efficient and easier to integrate with existing data pipelines.

import pandas as pd
import re

df = pd.DataFrame({
    'sample_id': [
        "TX2024-07-15-LOC1-REP1",
        "TX2024-07-15-LOC1-REP2",
        "TX2024-07-15-LOC2-REP1",
        "TX2024-07-15-LOC1-REP1",
        "TX2024-07-32-LOC1-REP3",
        "TX2024-07-15-LOC1"
    ]
})

Regex pattern
pattern = r"^TX\d{4}-\d{2}-\d{2}-LOC\d+-REP\d+$"

df['format_ok'] = df['sample_id'].str.match(pattern)

Date validation
def valid_date(id_):
    try:
        year, month, day = map(int, id_.split('-')[1:4])
        pd.Timestamp(year, month, day)
        return True
    except Exception:
        return False

df['date_ok'] = df['sample_id'].apply(lambda x: valid_date(x) if df.loc[df['sample_id']==x, 'format_ok'].iloc[0] else False)

Duplicate check
df['duplicate'] = df.duplicated(subset='sample_id', keep=False)

print(df)

Compared to the regular‑expression approach, the Pandas method scales better for thousands of rows, offers cleaner reporting, and integrates directly with data frames used for downstream statistical analysis.

Approach Comparison Table

Conclusion

Label integrity is non‑negotiable for credible crop pathology research. By recognising the most prevalent missteps and embedding systematic safeguards—structured conventions, colour‑coding protocols, barcode scanning, rigorous training, and automated validation—research teams can preserve data fidelity. The simple Python example illustrates that even modest computational tools can reinforce human diligence, ensuring that every sample carries a trustworthy, traceable identity from field to final analysis.

Further Reading

• Rao, R. (2019) — Hyperspectral Remote Sensing of Agriculture. Comprehensive guide to data acquisition, preprocessing, and interpretation in agricultural contexts. Highlights key steps for preparing crop spectra for AI models.
• Wang, Y., & Zhang, L. (2021) — Machine Learning Applications in Plant Disease Diagnosis. Peer‑reviewed review detailing spectral feature extraction and classification techniques used for diseased vs healthy crop identification.
• ISO/IEC 17025 (2023) — General Requirements for the Competence of Testing and Calibration Laboratories. Provides standards for data quality, calibration, and validation essential for reliable hyperspectral crop analysis.
• European Space Agency (ESA) (2022) — ESA Handbook on Hyperspectral Data Processing. Official documentation covering radiometric correction, noise reduction, and spatial‑spectral fusion for agricultural surveillance.
• Lee, J., & Kim, H. (2020) — Deep Learning for Crop Health Monitoring. Journal article presenting state‑of‑the‑art CNN architectures and preprocessing pipelines to distinguish healthy and diseased samples.
• National Agricultural Library (NAL) (2018) — Hyperspectral Imaging in Agriculture: Applications and Techniques. Government‑issued resource outlining best practices for sample preparation, data labelling, and AI integration.