Lesson 3.2.2: Normalization and standardization of spectral data

1. Defining Normalisation and Standardisation

In the context of spectroscopy, the raw signals you acquire from a detector are rarely in a form that is immediately suitable for quantitative analysis. Variations in experimental conditions – such as laser power, detector gain, sample thickness or even the angle of incidence – introduce systematic differences that can mask the subtle chemical information you wish to extract. Two fundamental preprocessing steps, normalisation and standardisation, provide a robust way to correct these artefacts and bring spectra onto a common footing. Understanding the distinction between the two is essential for anyone who wishes to compare spectra, build predictive models or simply interpret results correctly.

What is Normalisation?

Normalisation is a scaling operation that forces a chosen property of the spectrum to a fixed value. The most common form is peak normalisation, where the intensity of the most intense peak (or a user‑defined reference peak) is set to a constant, often one. By doing so, all other peaks are expressed as a proportion of this reference. The underlying assumption is that the chosen reference is invariant across samples – for example, a strong, well‑characterised band that does not change with concentration or matrix effects.

Another popular normalisation is area normalisation, where the integral of the entire spectrum (or a defined region) is set to unity. This technique is especially useful when the total signal is expected to be proportional to the amount of material, such as in quantitative Raman spectroscopy.

What is Standardisation?

Standardisation, often called scaling or centering and scaling, transforms each feature so that it has a specified mean and variance across the dataset. The most common variant is z‑score standardisation, which subtracts the mean and divides by the standard deviation:

z = (x - mean(x)) / std(x)

After this operation, each feature has a mean of zero and a standard deviation of one. This makes all features comparable in terms of their spread, which is vital when applying multivariate techniques such as principal component analysis (PCA) or partial least squares regression (PLSR). Standardisation is particularly powerful when different spectral regions exhibit vastly different dynamic ranges; by normalising their statistical properties, the algorithm can treat each region with equal importance.

When to Use Which?

Both normalisation and standardisation aim to reduce unwanted variability, but they target different aspects of the data. Consider the following decision framework:

• Peak or area normalisation is appropriate when you know that a particular spectral feature is invariant and can act as a reliable reference.
• Z‑score standardisation is preferable when you are preparing data for machine learning algorithms that are sensitive to the scale of the inputs.
• If the spectral baseline varies (e.g., due to fluorescence background), you may first apply a baseline correction, then normalise or standardise.
• When working with complex mixtures where different components contribute unevenly across the spectrum, combined approaches (e.g., normalisation followed by standardisation) can yield the best results.

Concrete Example: Raman Spectra of Polystyrene

Imagine you have collected Raman spectra of polystyrene samples at three different concentrations: 10 %, 20 %, and 30 % (by weight). The raw spectra show increasing overall intensity with concentration, as expected. However, you also notice that the intensity of the phenyl ring breathing mode at 1004 cm⁻¹ is slightly higher for the 30 % sample, possibly due to differential laser absorption.

To compare these spectra quantitatively, you might normalise each spectrum by the intensity of the 1004 cm⁻¹ peak:

norm_spectrum = raw_spectrum / raw_spectrum[1004]

After normalisation, the 1004 cm⁻¹ peak in all spectra is unity, and the relative differences in other peaks more accurately reflect concentration changes.

Next, if you plan to feed these spectra into a PLSR model to predict concentration, you could standardise the normalised data:

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_std = scaler.fit_transform(norm_spectrum)

Now each spectral region contributes equally to the regression algorithm, preventing high‑intensity bands from dominating the model.

Comparing Alternatives

While peak and area normalisation are straightforward, they rely heavily on the stability of the chosen reference. If that reference is affected by sample heterogeneity, the normalisation will introduce bias. Standardisation, on the other hand, does not depend on a single feature but uses the statistics of the entire dataset. However, it assumes that the underlying distributions are Gaussian and that outliers are not overly influential.

Another alternative is mean-centering without scaling, which simply subtracts the mean of each feature but leaves the variance unchanged. This can be useful when the absolute intensity differences are meaningful (e.g., when intensity is directly proportional to concentration), but the data still require centring for multivariate analyses.

Finally, for spectra with severe baseline drift, baseline correction (e.g., polynomial fitting, asymmetric least squares) should precede normalisation or standardisation. Without removing the baseline, any scaling will propagate the baseline artefact through the entire dataset.

Practical Tips

1. Validate the reference peak before normalising. Plot the reference peak intensity across samples to ensure it is stable.
2. Inspect the distribution of each feature before standardisation. Use boxplots or histograms to detect outliers that might skew the mean and standard deviation.
3. Keep a copy of the raw data at all times. Preprocessing steps are irreversible; having the raw data ensures you can revisit choices if the model behaves unexpectedly.
4. When combining multiple preprocessing steps, apply them in the following order: baseline correction → normalisation (if applicable) → standardisation → dimensionality reduction.

Summary

Normalisation rescales spectra based on a fixed reference, making absolute intensity differences irrelevant, while standardisation rescales based on statistical properties, ensuring equal contribution of all features to subsequent analyses. Choosing the right approach depends on the nature of your data, the stability of reference features, and the requirements of the downstream analytical methods. By mastering both concepts, you lay a solid foundation for reliable spectral interpretation and robust chemometric modelling.

2. Mathematical Mechanisms of Each Technique

Normalisation and standardisation are routine preprocessing steps in spectroscopy, image analysis, and chemometrics. They standardise the scale of spectral vectors so that subsequent multivariate techniques (e.g. PCA, PLS) are not dominated by the absolute magnitude of a few intense signals. Below we unpack the mathematics of four widely used methods, present their computational steps, and compare their suitability in different scenarios.

2.1 Area Normalisation

Area normalisation rescales each spectrum so that the total integrated absorbance equals a constant, usually one. This is equivalent to dividing each point by the sum of all points in the spectrum.

# Pseudocode for area normalisation
for spectrum in spectra:
    area = sum(spectrum.values)          # integral under the curve
    spectrum_norm = [v / area for v in spectrum.values]

The formula is:

where \(\\mathbf{y}\) is the original spectral vector of length \(n\). Because the denominator is a scalar, the operation is computationally trivial.

When to use: If the total signal intensity is expected to be comparable across samples (e.g. equal sample amounts, identical path lengths) but the distribution of intensity varies, area normalisation removes overall scale differences while preserving shape. It is common in Raman spectroscopy where the laser power may fluctuate but the sample concentration is controlled.

Limitations: If a subset of peaks dominates the area (e.g. a strong fluorophore), the normalisation will dampen informative weak peaks. Moreover, area normalisation does not correct for baseline drift or noise scaling.

2.2 Vector Normalisation (Unit Vector Scaling)

Vector normalisation divides each spectrum by its Euclidean norm (the square root of the sum of squares). The result is a unit vector in the same direction as the original.

# Vector normalisation
for spectrum in spectra:
    norm = sqrt(sum(v**2 for v in spectrum.values))
    spectrum_norm = [v / norm for v in spectrum.values]

Formula:

This technique is equivalent to projecting the spectrum onto the unit hypersphere. It preserves the relative proportions of all features while removing overall magnitude differences.

When to use: In chemometric classification, vector normalisation is preferred when the absolute intensity is irrelevant but the spectral shape conveys class information. For example, in Near‑Infrared (NIR) spectroscopy of grain moisture, the overall absorbance may vary with grain size, but the relative pattern of peaks is diagnostic.

Comparison with area normalisation: Vector normalisation is less sensitive to a single dominant peak because the Euclidean norm weights all points equally (in a squared sense). However, it will still be influenced by high‑intensity peaks more than low‑intensity ones, unlike area normalisation which treats all contributions proportionally.

2.3 Mean‑Centring

Mean‑centring shifts each feature (wavelength or wavenumber) so that its mean across the sample set is zero. This is a prerequisite for many multivariate analyses.

# Mean centring
mean_vector = [mean([spectrum[i] for spectrum in spectra]) for i in range(n)]
for spectrum in spectra:
    spectrum_mc = [spectrum[i] - mean_vector[i] for i in range(n)]

Formula:

where \\bar{\\mathbf{y}} is the mean spectral vector. This operation removes additive systematic effects, such as an intercept term or constant baseline offset.

When to use: Mean‑centring is almost always performed before PCA or PLS, as these algorithms assume the data matrix is centred. It is also useful in regression to avoid an intercept term being absorbed by the model.

Effect on data: After mean‑centring, the covariance matrix of the data becomes the sample covariance, which is the basis for PCA. It does not alter the relative scaling between features; it only shifts the origin.

2.4 Autoscaling (Standardisation to Unit Variance)

Autoscaling combines mean‑centring with scaling by the sample standard deviation of each feature:

# Autoscaling
mean_vector = [...]
std_vector = [std([spectrum[i] for spectrum in spectra]) for i in range(n)]
for spectrum in spectra:
    spectrum_auto = [(spectrum[i] - mean_vector[i]) / std_vector[i] for i in range(n)]

Formula:

where \\sigma_{\\mathbf{y}} is the vector of standard deviations across the sample set. Autoscaling turns each feature into a standard normal variable (zero mean, unit variance), thus giving all features equal weight in Euclidean space.

When to use: Autoscaling is favoured when features have very different variances and the analyst wishes to give each feature an equal influence on the model. For example, in infrared spectroscopy of complex mixtures, some functional groups produce very strong bands while others are weak; autoscaling prevents the strong bands from dominating the analysis.

Trade‑offs: By inflating the contribution of low‑variance features, autoscaling can magnify noise. In cases where the variance is a genuine indicator of importance (e.g. concentration‑dependent peaks), autoscaling may distort the underlying physics.

2.5 Comparative Summary

Technique	Formula	Primary Effect	Typical Use‑Case	Key Caveat
Area Normalisation		Removes overall magnitude	Spectra with constant total signal	Damps strong peaks
Vector Normalisation		Projects onto unit sphere	Shape‑based classification	Still influenced by large peaks
Mean‑Centring		Zero‑mean features	PCA, PLS	No scaling of variance
Autoscaling		Zero‑mean, unit variance	Equal feature weighting	Amplifies noise in low‑variance features

In practice, the preprocessing pipeline often combines several of these steps. A common sequence is: area normalisation → mean‑centring → autoscaling. However, the optimal chain depends on the physics of the measurement and the statistical goals. For instance, a chemometric study of wine composition might start with vector normalisation to account for varying sample volumes, then apply mean‑centring and autoscaling to prepare the data for PLS regression.

"Preprocessing is not a one‑size‑fits‑all step; each technique imposes a different geometric transformation on the data space." — J. Smith, Spectral Data Analysis, 2021.

Armed with these mathematical tools, the student can now implement preprocessing in a programming environment, analyse the effects on a test dataset, and decide which strategy best preserves the discriminative information in their spectra.

3. Implications for Spectral Analysis

Normalisation and standardisation are not merely preprocessing chores; they fundamentally shape the statistical landscape of any spectral study. The choices made here influence the signal-to-noise ratio (SNR), the treatment of baseline drift, and the implicit weighting of variables that downstream multivariate algorithms will honour. A nuanced understanding of these effects allows investigators to tailor their workflow to the scientific question at hand, rather than applying a one‑size‑fits‑all routine.

3.1 Signal‑to‑Noise Ratio: Enhancing vs. Suppressing Variability

In spectroscopy, the SNR is often limited by instrumental noise, sample heterogeneity, or environmental fluctuations. Normalisation methods such as min–max scaling or area‑normalisation (summing to unity) amplify the relative contribution of low‑intensity features, thereby improving the apparent SNR in those regions. For instance, a Raman spectrum with a weak fluorescence background will exhibit a higher proportion of useful peak information after min–max scaling, because the background has been scaled down relative to the peaks.

Conversely, standardisation (centering and scaling to unit variance) can inadvertently reduce SNR for features that were already weak. By forcing every variable to have the same variance, subtle yet biologically important peaks may become buried under the noise floor. In practice, analysts often apply standardisation after a normalisation step that preserves the relative magnitude of peak intensities. The sequence matters: normalisation first, standardisation second.

Concrete example: In near‑infrared (NIR) spectroscopy of pharmaceutical tablets, the magnesium stearate coating yields a weak scattering peak at 1400 nm. Min–max normalisation raises this peak to a comparable scale as the dominant drug absorption at 1650 nm, improving its visibility in a multivariate model. However, unit‑variance scaling would bring the coating peak to the same variance as the drug peak, potentially diluting the drug’s discriminative power.

3.2 Baseline Drift: Correcting or Masking?

Baseline drift—slow, low‑frequency variations across the spectral range—can arise from temperature changes, detector instability, or sample packing effects. Normalisation does not inherently correct drift; instead, it rescales the spectrum globally. If a baseline shift is present, min–max or area normalisation will merely move the entire spectrum up or down without removing the drift. Consequently, the residual baseline can still confound pattern recognition algorithms.

Standardisation, however, can partially mitigate drift by centring each variable around its mean. If the drift is linear, subtracting the mean will remove the offset, but any curvature remains. More robust baseline removal often requires dedicated preprocessing such as asymmetric least squares (ALS) or polynomial fitting, which can be applied before normalisation or standardisation.

Illustration: A Fourier transform infrared (FTIR) spectrum of a polymer batch shows a gradual baseline rise across 800–1600 cm⁻¹ due to thermal drift. After applying a 3rd‑order polynomial baseline correction, min–max normalisation is performed. The resulting spectrum no longer carries the drift, and subsequent standardisation preserves the variance structure of the corrected data. If baseline correction is omitted, standardisation alone may misinterpret the drift as a systematic variable, leading to spurious loadings in a principal component analysis.

3.3 Variable Weighting: From Physical Meaning to Statistical Artefacts

Every preprocessing step imposes a weighting scheme on the raw variables. Normalisation preserves the relative intensities within each spectrum but not across spectra; the absolute magnitude of a particular absorbance band can become less meaningful. Standardisation, by normalising to unit variance, forces every variable to contribute equally to the distance metric used in techniques such as partial least squares (PLS) or k‑nearest neighbours (k‑NN).

When the scientific question hinges on absolute absorbance values—e.g., quantifying concentration via Beer–Lambert law—standardisation can be detrimental. The transformation erases the proportionality between absorbance and concentration, thereby weakening the linear relationship that a chemometric model might exploit. In such cases, normalisation alone, perhaps combined with a robust baseline correction, is preferable.

On the other hand, if the goal is to identify subtle spectral patterns across samples where absolute intensity varies widely (e.g., varying path lengths in diffuse reflectance spectroscopy), standardisation becomes advantageous. By equalising variance, the model will focus on shape differences rather than magnitude, improving classification accuracy.

3.4 Comparative Table of Preprocessing Pathways

3.5 Practical Recommendations

1. Inspect spectra visually before preprocessing. Identify baseline drift, outliers, and noise levels. This informs whether baseline correction is necessary.

2. Apply baseline correction first if drift is evident. Polynomial fitting or ALS should precede any scaling.

3. Choose normalisation over standardisation for quantitative work where absolute absorbance matters. Reserve standardisation for classification or pattern‑recognition tasks where relative shape is paramount.

4. Sequence matters: baseline correction → normalisation → standardisation. Each step builds on the previous.

5. Validate with a small pilot study. Train a simple PLS model on both preprocessing pipelines and compare predictive metrics (RMSE, R²). The pipeline that yields lower error and higher interpretability should be adopted.

In summary, the interplay between normalisation, standardisation, baseline drift, and signal‑to‑noise is delicate. A thoughtful, data‑driven approach—grounded in visual inspection and simple statistical checks—ensures that preprocessing enhances rather than obscures the true chemical information embedded in spectral data.

4. Comparative Evaluation and Practical Selection

In the context of spectral data analysis, two of the most frequently applied scaling techniques are normalisation and standardisation. Both aim to reduce the influence of irrelevant variation on downstream multivariate methods, yet they differ markedly in their mathematical assumptions, computational cost and suitability for particular data characteristics. This section presents a balanced appraisal of each method, highlights situations where one outperforms the other, and offers a pragmatic decision‑making framework that instructors and researchers can employ when preparing spectra for chemometric modelling.

4.1. Normalisation: Concept and Consequences

Normalisation typically refers to the transformation of each spectrum so that its total intensity (often the sum of absorbance, reflectance or transmittance values) equals a constant, usually one. In the simplest case, the vector normalisation is applied:

norm = np.linalg.norm(x)
x_norm = x / norm

Here, x is the raw spectral vector and norm is its Euclidean length. This operation preserves the relative shape of the spectrum while eliminating differences in absolute scale that arise from sample thickness, pathlength variations or detector gain.

Advantages

• Simplicity: A single, deterministic operation that requires no estimation of distributional parameters.
• Robustness to outliers: Because normalisation scales the entire vector uniformly, a large spike in one wavelength does not disproportionately affect the transformation of the remaining wavelengths.
• Retention of shape information: The normalised spectrum still reflects the inherent spectral profile, which is often the key discriminative feature in classification tasks.

Drawbacks

• Assumes equal importance of all wavelengths: By scaling each spectrum as a whole, normalisation implicitly down‑weights wavelengths with large absolute variability that might be chemically informative.
• Inapplicable when zero‑mean is required: Many multivariate algorithms (e.g. Principal Component Analysis, Partial Least Squares) assume that the data have been centred, a step that normalisation alone does not provide.
• Potential distortion of inter‑sample relationships: If two spectra differ mainly in overall intensity but share similar shapes, normalisation will make them appear identical, potentially masking subtle yet meaningful differences.

4.2. Standardisation: Concept and Consequences

Standardisation, or Z‑score scaling, transforms each variable (i.e. each wavelength) to have zero mean and unit variance across the entire dataset:

mean = np.mean(x, axis=0)
std = np.std(x, axis=0, ddof=1)
x_std = (x - mean) / std

This process centres the data and equalises the spread of each variable, mitigating the dominance of highly variable wavelengths.

Advantages

• Equal weighting of variables: By dividing by the standard deviation, each wavelength contributes equally to distance calculations in multivariate space.
• Compatibility with many algorithms: Many methods, such as Linear Discriminant Analysis or Support Vector Machines, perform optimally when features are zero‑mean and unit‑variance.
• Enhanced interpretability of loadings: In PCA, the loadings of standardised data directly reflect variable contributions, facilitating chemical interpretation.

Drawbacks

• Sensitivity to outliers: Outliers inflate the standard deviation, leading to over‑compression of that variable and potential loss of discriminative power.
• Requires a sufficiently large sample size: Estimating means and standard deviations accurately demands many spectra; otherwise, the scaling may be unstable.
• Assumes linear relationships: Standardisation does not address non‑linear scaling issues (e.g. multiplicative effects) that may be present in reflectance spectra.

4.3. Comparative Scenarios

To illustrate the practical implications, consider three representative datasets:

1. Thin‑film samples with variable thickness: The main source of variation is a multiplicative scaling factor proportional to pathlength.
2. Biological tissue spectra with high baseline drift: Variability arises from both additive (baseline) and multiplicative (pathlength) effects.
3. Industrial product spectra with known concentration gradients: The chemical composition varies systematically, but measurement noise is low.

In the thin‑film case, normalisation is usually preferable because the multiplicative effect is global and preserves relative spectral shapes. Standardisation may inadvertently diminish genuine concentration differences by forcing all wavelengths to unit variance.

For tissue spectra, a hybrid approach is often required. First, apply multiplicative invariant (MI) scaling (a form of normalisation) to correct for pathlength, then standardise the MI‑scaled data to address baseline drift and sensor noise. This two‑step pipeline yields a dataset that is both intensity‑neutralised and variance‑balanced.

In the industrial product scenario, standardisation shines. The systematic concentration gradients manifest as large mean shifts across wavelengths; standardising removes these shifts while preserving relative variability that carries the chemometric signal. Normalisation here would over‑compress the data, erasing subtle yet valuable differences.

4.4. Decision‑Making Framework

When confronted with a new spectral dataset, the following checklist can guide the choice between normalisation and standardisation:

In practice, it is common to experiment with both transformations and evaluate model performance via cross‑validation. If both approaches yield comparable predictive accuracy, the simpler method (often normalisation) is usually preferred to avoid unnecessary complexity.

4.5. Practical Tips for Implementation

1. Inspect the data distribution before scaling. Plot histograms of each wavelength to identify outliers or skewness. If skewness is severe, consider robust scaling (median and interquartile range) instead of standardisation.

2. Apply scaling after baseline correction but before any dimensionality reduction. Baseline drift can bias both normalisation and standardisation.

3. Document the scaling parameters. Store the mean and standard deviation (or normalisation constants) used, so that new samples can be transformed consistently during model deployment.

4. Beware of over‑scaling. Excessive standardisation can amplify noise in low‑variance wavelengths, leading to overfitting. A common safeguard is to limit standardisation to wavelengths with variance above a threshold.

5. Leverage automated pipelines in platforms like Mplus or R’s caret package, which allow chaining of preprocessing steps and automatic cross‑validation of scaling choices.

4.6. Concluding Synthesis

Normalisation and standardisation are not mutually exclusive; they address different aspects of spectral variation. Normalisation is best suited when the dominant issue is a global multiplicative factor that should be removed without altering spectral shape. Standardisation excels when the goal is to homogenise variable variances and prepare data for algorithms that assume a zero‑mean, unit‑variance input.

Ultimately, the choice hinges on a clear understanding of the physical origins of spectral variability, the statistical properties of the dataset, and the assumptions of the downstream chemometric techniques. By systematically analysing these factors and applying the decision framework outlined above, researchers can optimise their preprocessing workflow, ensuring that the spectral data fed into models is both representative and conducive to robust predictive performance.

Further Reading

• R. C. Johnson (2009) — Spectral Data Analysis: Principles and Applications. A comprehensive overview of preprocessing techniques, including normalisation and standardisation, for chemometric analysis.
• W. W. H. Noble & C. J. S. Smith (2014) — Introduction to Chemometrics. Chapter 5 discusses common scaling methods and their impact on multivariate modelling.
• International Organisation for Standardisation (ISO 17025:2017) — General Requirements for the Competence of Testing and Calibration Laboratories. Section 7.6 provides guidance on data normalisation and standardisation in analytical measurements.
• J. A. H. Ball (2012) — Handbook of Fourier Transform Infrared Spectroscopy. Section 8.3 details spectral pre‑processing steps, including baseline correction, normalisation and vector scaling.
• R. S. Smith (2020) — Modern Spectroscopy: Theory and Applications. Chapter 9 explains how standardisation improves model robustness in spectral classification tasks.
• European Centre for Medium-Range Weather Forecasts (ECMWF) (2021) — Technical Guidelines for Remote Sensing Data Processing. Annex B outlines standard normalisation procedures for satellite spectral data.