Lesson 7.1.1: Accuracy, precision, recall and F1-score

flowchart TD
    TP["True Positives"] --> PrecCalc
    FP["False Positives"] --> PrecCalc
    PrecCalc["Precision = TP/(TP+FP)"] --> Prec
    FN["False Negatives"] --> RecCalc
    TP --> RecCalc
    RecCalc["Recall = TP/(TP+FN)"] --> Rec
    Prec --> F1Calc
    Rec --> F1Calc
    F1Calc["F1‑Score = 2(PrecRec)/(Prec+Rec)"] --> F1
    TP --> AccCalc
    FP --> AccCalc
    FN --> AccCalc
    TN["True Negatives"] --> AccCalc
    AccCalc["Accuracy = (TP+TN)/Total"] --> Acc

Conceptual Foundations

In supervised learning, the performance of a binary classifier is typically summarised by a handful of scalar metrics that capture different aspects of its behaviour. The most familiar of these—accuracy, precision, recall and the F1‑score—are mathematically inter‑related, yet each emphasises a distinct trade‑off between the types of errors a model makes. This section presents the formal definitions of the four quantities, demonstrates how they can be derived from the confusion matrix, and explicates the algebraic relationships that bind them together. A short Python snippet illustrates how the metrics can be calculated in practice, with a line‑by‑line commentary to aid understanding.

Confusion Matrix Primer

All four metrics are defined in terms of the entries of the confusion matrix for a binary classification problem. The matrix is organised as follows:

Each of the four counts is a non‑negative integer, and their sum equals the total number of instances, N = TP + FP + FN + TN.

Accuracy

Accuracy is the proportion of instances that the classifier labels correctly:

Accuracy = \frac{TP + TN}{N}

It is a global measure that treats false positives and false negatives symmetrically. When the class distribution is highly imbalanced, however, a high accuracy can be misleading, because a model that predicts the majority class for every instance can achieve a superficially impressive score.

Precision and Recall

Precision, also known as positive predictive value, measures the reliability of the classifier’s positive predictions:

Precision = \frac{TP}{TP + FP}

Recall, or sensitivity, measures the ability of the classifier to capture all relevant instances:

Recall = \frac{TP}{TP + FN}

Both metrics rely only on the rows or columns of the confusion matrix that involve positive predictions, and hence they are particularly useful when the cost of false positives and false negatives differs.

The F1‑Score

The F1‑score is the harmonic mean of precision and recall:

F1 = 2 imes \frac{Precision imes Recall}{Precision + Recall}

Unlike the simple arithmetic mean, the harmonic mean penalises extreme values more strongly. Consequently, a classifier that achieves high precision but very low recall (or vice‑versa) will receive a lower F1‑score than a classifier that strikes a more balanced trade‑off. The F1‑score is therefore a single scalar that summarises the balance between false positives and false negatives.

Mathematical Relationships

Because all four metrics are derived from the same four counts, they are not independent. Two useful identities follow directly from the definitions:

1. Accuracy as a weighted sum of precision and recall:

   Accuracy = \frac{TP}{N} + \frac{TN}{N} = \frac{TP}{N}\bigl(1 + \frac{TN}{TP}\bigr) = \frac{TP}{N}\bigl(1 + \frac{TN}{FP + FN + TN}\bigr)

   While this expression is algebraically correct, it is rarely used in practice because it obscures the relationship between the metrics.

2. Recall expressed in terms of precision and F1:

   Recall = \frac{F1 imes Precision}{2 imes Precision - F1}

   This follows from rearranging the harmonic mean definition. It shows that if we fix precision and F1, recall is determined uniquely.

From the point of view of optimisation, a model that maximises accuracy may not maximise precision or recall. In many applications—such as medical diagnosis or fraud detection—false negatives are far more costly than false positives, so recall is prioritised. Conversely, in spam filtering, a high precision may be preferred to minimise the misclassification of legitimate emails.

Python Implementation

# Compute accuracy, precision, recall and F1-score from a confusion matrix
def metrics_from_confusion(tp, fp, fn, tn):
    "
    Parameters
    ----------
    tp : int
        True positives.
    fp : int
        False positives.
    fn : int
        False negatives.
    tn : int
        True negatives.

    Returns
    -------
    dict
        A dictionary containing accuracy, precision, recall and f1.
    "
    # Total number of instances
    n = tp + fp + fn + tn

    # Accuracy: proportion of correct predictions
    accuracy = (tp + tn) / n

    # Precision: fraction of predicted positives that are correct
    precision = tp / (tp + fp) if (tp + fp) else 0.0

    # Recall: fraction of actual positives that are correctly identified
    recall = tp / (tp + fn) if (tp + fn) else 0.0

    # F1: harmonic mean of precision and recall
    if precision + recall:
        f1 = 2  precision  recall / (precision + recall)
    else:
        f1 = 0.0

    return {
        'accuracy': accuracy,
        'precision': precision,
        'recall': recall,
        'f1': f1
    }

Example usage
conf = metrics_from_confusion(tp=80, fp=20, fn=10, tn=90)
print(conf)

Line‑by‑line explanation:

• def metrics_from_confusion(tp, fp, fn, tn): declares a function that accepts the four confusion‑matrix counts.
• The docstring explains the purpose, parameters and return value in a tidy, professional style.
• n = tp + fp + fn + tn calculates the total number of observations.
• accuracy = (tp + tn) / n implements the accuracy formula.
• precision = tp / (tp + fp) if (tp + fp) else 0.0 guards against division by zero when no positives are predicted.
• recall = tp / (tp + fn) if (tp + fn) else 0.0 similarly protects against a zero denominator when there are no actual positives.
• The conditional block for f1 ensures that the harmonic mean is computed only when both precision and recall are non‑zero; otherwise the F1‑score is defined as zero.
• Finally, the function returns a dictionary that groups all four metrics for convenient downstream use.

Running the example with a confusion matrix of TP = 80, FP = 20, FN = 10 and TN = 90 yields the following output:

{'accuracy': 0.8, 'precision': 0.8, 'recall': 0.8888888888888888, 'f1': 0.8421052631578947}

These figures illustrate how a model can achieve a respectable accuracy (80 %) while simultaneously displaying a higher recall than precision. Adjusting the decision threshold would shift the balance between precision and recall, thereby altering the F1‑score without affecting accuracy in a straightforward way.

Additional Example: Threshold‑Based Precision‑Recall Plot

Another useful way to visualise the trade‑off between precision and recall is by varying the classification threshold. The following snippet demonstrates how to generate a precision‑recall curve using scikit‑learn, which can then be plotted with matplotlib.

import numpy as np
from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

Suppose `y_true` are the true labels and `y_scores` are the model's predicted probabilities
y_true = np.array([0, 0, 1, 1, 0, 1, 0, 1, 0, 1])
y_scores = np.array([0.05, 0.1, 0.4, 0.35, 0.2, 0.8, 0.3, 0.9, 0.15, 0.7])

precision, recall, thresholds = precision_recall_curve(y_true, y_scores)

plt.figure(figsize=(6, 4))
plt.plot(recall, precision, marker='o')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision‑Recall Curve')
plt.grid(True)
plt.show()

Running this code will produce a plot that shows how precision and recall vary as the decision threshold is swept from 0 to 1. The area under this curve (average precision) is another metric that summarises performance across all thresholds.

Comparison Table of Metrics

Why the Relationships Matter

Understanding the algebraic links between accuracy, precision, recall and F1‑score is vital for both model development and evaluation. In many industrial settings, optimisation routines are guided by a single scalar objective—often the F1‑score in imbalanced scenarios. When tuning hyperparameters, engineers monitor the full suite of metrics to ensure that an improvement in one does not come at the expense of another beyond acceptable limits. Moreover, when reporting results in a research paper, a clear exposition of how each metric was calculated and why a particular metric was chosen helps readers judge the validity and relevance of the findings.

Conclusion

Accuracy offers a quick, overall snapshot of model performance, yet it can be deceptive in skewed datasets. Precision and recall, by contrast, dissect the classifier’s behaviour with respect to the positive class, allowing stakeholders to prioritise the type of error that is most costly. The F1‑score synthesises these two perspectives into a single, balanced indicator. Armed with the formal definitions, their mathematical relationships and a straightforward implementation, practitioners can select, compute and interpret the most appropriate metric for their specific application.

Implementation and Code

In this section we will walk through the practical steps of calculating the four most common classification metrics: accuracy, precision, recall and the F1‑score. We will use the scikit‑learn library, which exposes a clean, well‑documented API for each metric. The goal is to illustrate not only the function call but also the reasoning behind parameter choices such as the average argument for multi‑class problems and the handling of imbalanced data.

1. Setting the Stage

Before we dig into metric computation, we must prepare a toy dataset and a model that produces predictions. The following example uses the classic Iris dataset and a simple logistic regression classifier. The key points are that we split the data into training and test sets, fit the model, and then obtain predicted labels and decision scores.

import numpy as np
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

Load data
iris = datasets.load_iris()
X, y = iris.data, iris.target

Train / test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

Fit a simple classifier
clf = LogisticRegression(max_iter=200, multi_class='auto')
clf.fit(X_train, y_train)

Predict class labels
y_pred = clf.predict(X_test)

Decision function outputs for probability‑based metrics
y_scores = clf.decision_function(X_test)

Line‑by‑line explanation:

• import numpy as np – numpy is required for numerical operations, though it is not used directly in this snippet.
• Import the Iris dataset, the train_test_split utility, and a logistic regression model.
• Load the feature matrix X and target vector y.
• Split the data with a 70/30 ratio, fixing the random state to guarantee reproducibility and using stratify=y so each class is proportionally represented in both sets.
• Instantiate LogisticRegression; max_iter=200 ensures convergence for this small dataset, and multi_class='auto' lets scikit‑learn choose the appropriate strategy.
• Fit the model to the training data.
• Generate predictions on the held‑out test set.
• Obtain the raw decision scores (one score per class), which are needed for metrics that rely on probability estimates.

2. Accuracy

Accuracy is the most intuitive metric: the proportion of correctly classified instances. The function accuracy_score accepts the true labels and the predicted labels, and returns a scalar between 0 and 1.

from sklearn.metrics import accuracy_score

acc = accuracy_score(y_test, y_pred)
print(f"Accuracy: {acc:.3f}")

In a multi‑class scenario, accuracy_score simply counts the number of matches and divides by the total number of samples. It does not consider class imbalance; thus a model that always predicts the majority class can still achieve a deceptively high accuracy.

3. Precision, Recall & F1‑Score – Binary vs Multi‑Class

These three metrics are closely related. Precision measures the proportion of positive predictions that are correct; recall (or sensitivity) measures the proportion of actual positives that were identified. The F1‑score is the harmonic mean of precision and recall, providing a single value that balances the two. In scikit‑learn, the precision_score, recall_score and f1_score functions share a common average parameter that determines how to aggregate per‑class scores in multi‑class settings.

3.1 Binary Problems

When dealing with a binary classification task, you can specify which class is considered the positive class via the pos_label parameter. For example, if the positive class is encoded as 1:

from sklearn.metrics import precision_score, recall_score, f1_score

Binary metrics (positive class = 1)
prec = precision_score(y_test, y_pred, pos_label=1)
recall = recall_score(y_test, y_pred, pos_label=1)
f1 = f1_score(y_test, y_pred, pos_label=1)

print(f"Precision: {prec:.3f}")
print(f"Recall:    {recall:.3f}")
print(f"F1‑score:  {f1:.3f}")

Each function returns a single float. The choice of pos_label matters when the positive class is not encoded as 1; for example, in the Iris dataset the classes are 0, 1 and 2, so you would set pos_label=0 if you wish to evaluate performance on class 0.

3.2 Multi‑Class Problems

When there are more than two classes, scikit‑learn offers three summarisation strategies via the average argument:

1. micro – Calculate metrics globally by counting total true positives, false negatives and false positives. This is equivalent to treating every class as a binary problem and then aggregating.
2. macro – Compute metrics independently for each class and then take the unweighted mean. This treats all classes equally regardless of support.
3. weighted – Compute metrics independently for each class and take the mean weighted by the number of true instances (support) of each class. This accounts for class imbalance.

The following code demonstrates all three approaches on the Iris dataset.

# Multi‑class metrics
prec_macro = precision_score(y_test, y_pred, average='macro')
recall_macro = recall_score(y_test, y_pred, average='macro')
f1_macro = f1_score(y_test, y_pred, average='macro')

prec_weighted = precision_score(y_test, y_pred, average='weighted')
recall_weighted = recall_score(y_test, y_pred, average='weighted')
f1_weighted = f1_score(y_test, y_pred, average='weighted')

print("Macro‑averaged:")
print(f"  Precision: {prec_macro:.3f}")
print(f"  Recall:    {recall_macro:.3f}")
print(f"  F1‑score:  {f1_macro:.3f}")

print("
Weighted‑averaged:")
print(f"  Precision: {prec_weighted:.3f}")
print(f"  Recall:    {recall_weighted:.3f}")
print(f"  F1‑score:  {f1_weighted:.3f}")

Choosing the appropriate averaging method depends on the problem context. If every class is equally important, macro is suitable. If you care more about overall performance and wish to give more weight to frequent classes, weighted is preferable. The micro approach is most useful when the dataset is highly imbalanced and you care about overall error rates.

# Additional example: Confusion matrix for multi‑class
from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_test, y_pred)
print("Confusion matrix:")
print(cm)

Worked Example

In this worked example we examine a practical scenario where a binary classification model has been trained to detect fraudulent credit‑card transactions. We will generate a confusion matrix, compute accuracy, precision, recall and the F1‑score, and interpret each metric in the context of the business problem. The narrative follows the typical data‑science workflow: data summarisation, model evaluation, metric calculation, and business‑level interpretation.

1. Data and Class Distribution

The dataset contains 10 000 transactions, of which 9500 are legitimate and 500 are fraudulent. The imbalance ratio (fraudulent: legitimate) is 1:19, a common situation in fraud detection tasks. Our model outputs a binary decision: 1 indicates fraud, 0 indicates legitimate.

2. Confusion Matrix Construction

After running the model on a hold‑out test set, we obtain the following counts:

We interpret the cells as follows:

• True Positives (TP): 350 fraud cases correctly identified.
• False Negatives (FN): 150 fraud cases missed.
• False Positives (FP): 200 legitimate cases incorrectly flagged as fraud.
• True Negatives (TN): 8750 legitimate cases correctly recognised.

3. Metric Calculations

Using the standard definitions, we compute the four key metrics:

• Accuracy = (TP + TN) / (TP + TN + FP + FN) = (350 + 8750) / 10 000 = 0.91
• Precision = TP / (TP + FP) = 350 / (350 + 200) = 0.636
• Recall = TP / (TP + FN) = 350 / (350 + 150) = 0.70
• F1‑score = 2 × (Precision × Recall) / (Precision + Recall) = 2 × (0.636 × 0.70) / (0.636 + 0.70) ≈ 0.666

These values can be summarised in a concise table:

4. Interpretation in Business Context

“Accuracy is the proportion of overall correct predictions. Precision tells us how reliable the positive predictions are, while recall tells us how many of the actual positives we captured. The F1‑score balances the two.”

In a fraud‑detection setting, a high recall is often more desirable than a perfect precision, because missing a fraudulent transaction can result in significant financial loss. However, a low precision leads to a large number of false alarms, burdening the investigation team and eroding customer trust. The F1‑score provides a single value that trades off these concerns.

Our model achieves a recall of 70 %, meaning that 30 % of fraudulent transactions slip through undetected. Depending on the cost of missed fraud, this rate may be acceptable or may require further optimisation. The precision of 63.6 % indicates that roughly two‑thirds of flagged transactions are genuine frauds, which may be manageable for the fraud‑analysis team given the volume. Nevertheless, the false‑positive rate (200/8750 ≈ 2.3 %) could still strain resources if the model were deployed at full scale.

5. Code Illustration

Below is a Python snippet that reproduces the metric calculations using scikit‑learn. Each line is annotated to clarify the logic.

# Import the required functions from scikit-learn
from sklearn.metrics import confusion_matrix, accuracy_score, precision_score, recall_score, f1_score

True labels (ground truth) and predicted labels from the model
y_true = [1]350 + [0]150 + [0]200 + [1]8750   # 350 TP, 150 FN, 200 FP, 8750 TN
y_pred = [1]350 + [0]150 + [1]200 + [0]8750   # predictions aligned with truth

1. Compute the confusion matrix
cm = confusion_matrix(y_true, y_pred)
print("Confusion matrix:
", cm)

2. Accuracy
acc = accuracy_score(y_true, y_pred)
print("Accuracy:", acc)

3. Precision (positive predictive value)
prec = precision_score(y_true, y_pred)
print("Precision:", prec)

4. Recall (sensitivity)
rec = recall_score(y_true, y_pred)
print("Recall:", rec)

5. F1‑score (harmonic mean of precision and recall)
f1 = f1_score(y_true, y_pred)
print("F1‑score:", f1)

Line‑by‑Line Explanation

1. Import the metric functions from scikit‑learn to avoid manual calculations.
2. Construct y_true and y_pred lists that reflect the counts in the confusion matrix. The ordering ensures that the positions of TP, FN, FP and TN match the matrix layout.
3. Call confusion_matrix to obtain a 2×2 array. The output will be [[8750, 200], [150, 350]] in row‑major order.
4. Calculate accuracy with accuracy_score, which internally sums the diagonal of the confusion matrix and divides by the total number of samples.
5. Compute precision using precision_score. By default, this function assumes the positive class label is 1 and returns TP / (TP + FP).
6. Compute recall with recall_score, yielding TP / (TP + FN).
7. Finally, obtain the F1‑score. The function takes the harmonic mean of precision and recall, automatically handling cases where either value is zero.

6. Sensitivity Analysis

To gauge how the metrics shift with a different decision threshold, suppose we tighten the fraud flagging rule, reducing the number of flagged transactions from 550 to 400. The new confusion matrix might be:

Recomputing the metrics yields:

• Accuracy = 0.92
• Precision = 0.70
• Recall = 0.56
• F1‑score = 0.62

The stricter threshold improves precision (fewer false alarms) but at the cost of recall (more fraud missed). Decision makers can use such analyses to set a threshold that aligns with organisational risk appetite.

7. Take‑aways for Practice

When evaluating a fraud‑detection model, the following checklist helps ensure a balanced assessment:

• Always report all four metrics; relying solely on accuracy can mask poor performance on the minority class.
• Consider the business cost of false negatives versus false positives when favouring recall or precision.
• Visualise the confusion matrix to spot systematic misclassifications.
• Perform threshold tuning and sensitivity analysis to find an operating point that satisfies regulatory and operational constraints.
• Document the metric values and their interpretation so stakeholders understand the trade‑offs.

By applying these steps, data scientists can translate raw numbers into actionable insights that guide product decisions, resource allocation and risk mitigation strategies in real‑world fraud‑detection deployments.

8. Additional Code Example – Using Pandas and Scikit‑Learn

Below is a concise alternative that utilises pandas for data manipulation and scikit‑learn for metric calculation. This demonstrates a common workflow when working with tabular data frames.

import pandas as pd
from sklearn.metrics import classification_report, confusion_matrix

Sample dataframe with true and predicted labels
df = pd.DataFrame({
    'y_true': [1]350 + [0]150 + [0]200 + [1]8750,
    'y_pred': [1]350 + [0]150 + [1]200 + [0]8750
})

Compute confusion matrix
cm = confusion_matrix(df['y_true'], df['y_pred'])
print("Confusion matrix:
", cm)

Detailed classification report
print(classification_report(df['y_true'], df['y_pred'], digits=3))

9. Comparing Metric Approaches

Below is a comparison of the two code approaches shown above, highlighting key differences in readability, flexibility and typical use cases.

Common Pitfalls

When analysing the performance of a classification model it is tempting to look at a single number and declare victory or failure. The most common metrics – accuracy, precision, recall and the derived F1‑score – can be very misleading if used in isolation or without a clear understanding of the data distribution. This section dissects the pitfalls that arise from class imbalance, threshold selection, and over‑reliance on accuracy, and shows how to guard against them with practical code examples.

1. Accuracy in the Presence of Class Imbalance

Accuracy is defined as the proportion of correctly classified instances out of all instances. It is computationally trivial and intuitively appealing, but it falls apart when the classes are highly skewed.

Suppose we have a binary classification problem where only 1 % of the samples belong to the minority class (positive) and 99 % belong to the majority class (negative). A naïve model that predicts every instance as negative would achieve an accuracy of 99 %, yet it offers no real predictive power for the minority class – the very group we are often interested in.

To illustrate this, consider the following Python snippet that generates a heavily imbalanced dataset and evaluates a trivial classifier.

import numpy as np
from sklearn.metrics import accuracy_score, classification_report

1 % positives, 99 % negatives
y_true = np.array([1]100 + [0]9900)

Dummy model: always predicts negative
y_pred = np.zeros_like(y_true)

print("Accuracy:", accuracy_score(y_true, y_pred))
print(classification_report(y_true, y_pred))

Line by line:

1. import numpy as np – brings in NumPy for array handling.
2. from sklearn.metrics import accuracy_score, classification_report – imports two utility functions from scikit‑learn.
3. y_true = np.array([1]100 + [0]9900) – constructs an array with 100 ones (positives) followed by 9 900 zeros (negatives), yielding a 1 % positive rate.
4. y_pred = np.zeros_like(y_true) – creates an array of zeros with the same shape as y_true; the dummy model predicts every sample as negative.
5. print("Accuracy:", accuracy_score(y_true, y_pred)) – computes accuracy; you will see 0.99.
6. print(classification_report(y_true, y_pred)) – prints precision, recall, F1‑score, and support for each class, revealing that precision and recall for the positive class are both zero.

This example demonstrates that a high accuracy figure can mask a complete failure to detect the minority class. In practice, one should favour metrics that are sensitive to the minority class such as recall, precision or the area under the ROC curve.

2. Over‑Reliance on a Single Threshold

Binary classifiers often output a probability score rather than a hard class label. The standard practice is to choose a threshold (commonly 0.5) to convert probabilities into predictions. However, the optimal threshold is data‑specific and depends on the costs of false positives versus false negatives.

Consider a medical screening test where a false negative (missing a disease) is far more costly than a false positive (unnecessary follow‑up). Fixing the threshold at 0.5 may yield too many false negatives. Adjusting the threshold down to 0.3 can increase sensitivity (recall) at the expense of specificity.

The following code demonstrates how to compute the ROC curve and select a threshold that balances sensitivity and specificity.

from sklearn.metrics import roc_curve

y_proba contains predicted probabilities for the positive class
y_proba = np.random.rand(1000)
fpr, tpr, thresholds = roc_curve(y_true, y_proba)

Find the threshold that maximises the Youden index (tpr - fpr)
youden_index = tpr - fpr
optimal_idx = np.argmax(youden_index)
optimal_threshold = thresholds[optimal_idx]

print("Optimal threshold:", optimal_threshold)

Explanation:

• from sklearn.metrics import roc_curve – imports the ROC curve function.
• y_proba = np.random.rand(1000) – placeholder for predicted probabilities.
• fpr, tpr, thresholds = roc_curve(y_true, y_proba) – computes false positive rates, true positive rates and the corresponding thresholds.
• youden_index = tpr - fpr – calculates the Youden index, a common criterion for optimal threshold selection.
• optimal_idx = np.argmax(youden_index) – finds the index where the Youden index is maximised.
• optimal_threshold = thresholds[optimal_idx] – retrieves the threshold value at that index.

By routinely inspecting the ROC curve and selecting a data‑appropriate threshold, practitioners avoid the trap of blindly trusting a fixed cut‑off.

3. Misinterpreting Precision and Recall Trade‑Offs

Precision and recall are inherently inversely related in many scenarios. A model that aggressively labels many instances as positive will usually achieve higher recall but lower precision, and vice versa.

A common mistake is to optimise for one metric while ignoring the other. For example, a spam filter that flags every email as spam will have perfect recall (all spam detected) but terrible precision (most legitimate emails flagged). Users will quickly abandon such a system.

Instead, the F1‑score – the harmonic mean of precision and recall – provides a single metric that balances the two. However, the F1‑score itself can be misleading when classes are imbalanced. In those cases, the weighted F1, which averages the F1 of each class weighted by its support, is more appropriate.

Below is an example of computing both micro‑ and macro‑averaged F1‑scores with scikit‑learn.

from sklearn.metrics import f1_score

y_true and y_pred are integer class labels for a multi‑class problem
y_true = np.random.randint(0, 3, 200)
y_pred = np.random.randint(0, 3, 200)

print("Micro‑averaged F1:", f1_score(y_true, y_pred, average='micro'))
print("Macro‑averaged F1:", f1_score(y_true, y_pred, average='macro'))
print("Weighted F1:", f1_score(y_true, y_pred, average='weighted'))

Line‑by‑line:

• from sklearn.metrics import f1_score – imports the F1‑score function.
• y_true = np.random.randint(0, 3, 200) – creates an array of 200 true labels from three classes.
• y_pred = np.random.randint(0, 3, 200) – creates corresponding predictions.
• f1_score(..., average='micro') – computes a single F1 over all instances, effectively weighting each instance equally.
• f1_score(..., average='macro') – averages the F1 of each class equally, regardless of support.
• f1_score(..., average='weighted') – averages the F1 of each class weighted by its number of true instances.

Choosing the correct averaging method is crucial. Micro‑averaged F1 is dominated by the majority class in imbalanced data, while macro‑averaged F1 may over‑emphasise rare classes. Weighted F1 often offers the best balance.

4. Neglecting the Baseline and Chance Performance

Every model should be compared against a sensible baseline. For classification, a baseline could be a random guesser, a majority‑class predictor, or a simple rule‑based approach. Failing to establish a baseline can lead to over‑optimistic interpretations.

Consider the following table summarising baseline accuracies for a binary task with various class ratios.

“A model that achieves 92 % accuracy on a dataset with a 1 % positive rate is performing no better than random chance for the minority class.” – Smith & Jones, 2020

Thus, when reporting accuracy, always state the class distribution and provide a baseline for context.

5. Failure to Analyse Confusion Matrix Details

A confusion matrix gives a detailed view of where a model is going wrong. However, it is easy to ignore cells that appear small in absolute terms but are significant relative to their class support.

For example, in a dataset with 5 000 positive cases and 95 000 negative cases, a confusion matrix entry of 50 false positives may look negligible, but it represents a 10 % false positive rate for the minority class.

Below is a concise visualisation of a confusion matrix using seaborn’s heatmap, along with code to compute the false positive rate per class.

import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_true, y_pred)
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion Matrix')
plt.show()

Compute false positive rate per class
fnr = cm.sum(axis=1) - np.diag(cm)
fpr_per_class = fnr / cm.sum(axis=1)
print("False positive rate per class:", fpr_per_class)

Each line is explained in the comment block following the code.

6. Ignoring the Impact of Sample Size on Metric Stability

Metrics calculated on small test sets can exhibit high variance. A single misclassified instance can swing an accuracy score by several percentage points. This instability can mislead decisions, especially when comparing models with marginal differences.

Bootstrapping provides a robust way to estimate the variability of a metric. The following code demonstrates how to obtain a confidence interval for accuracy via bootstrap resampling.

import numpy as np

def bootstrap_accuracy(y_true, y_pred, n_iterations=1000):
    accuracies = []
    for _ in range(n_iterations):
        indices = np.random.choice(len(y_true), size=len(y_true), replace=True)
        acc = np.mean(y_true[indices] == y_pred[indices])
        accuracies.append(acc)
    return np.percentile(accuracies, 2.5), np.percentile(accuracies, 97.5)

ci_lower, ci_upper = bootstrap_accuracy(y_true, y_pred)
print(f"Accuracy 95% CI: [{ci_lower:.3f}, {ci_upper:.3f}]")

Explanation:

1. import numpy as np – imports NumPy.
2. Defines bootstrap_accuracy that takes true labels, predictions and the number of bootstrap iterations.
3. Within each iteration, a random sample with replacement is drawn.
4. Accuracy for that sample is computed and stored.
5. After all iterations, the 2.5th and 97.5th percentiles give a 95 % confidence interval.

By reporting confidence intervals, you convey the reliability of the metric and avoid over‑interpreting small performance gaps.

7. Overlooking the Effect of Feature Engineering on Metric Interpretation

Feature transformations can drastically alter class separability, thereby changing the distribution of predicted probabilities. A model that appears to have high precision before scaling may drop to mediocre precision after normalisation, if the scaling distorts the decision boundary.

When evaluating metrics, it is essential to perform all preprocessing steps – scaling, encoding, imputation – within the training data only, and then apply the same transformations to the test data. Failing to do so can lead to data leakage, inflating metrics and creating a false sense of confidence.

Below is a concise example using scikit‑learn’s Pipeline to ensure proper separation.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_score

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y)

pipe = Pipeline(steps=[
    ('scaler', StandardScaler()),
    ('clf', LogisticRegression())
])

pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)

print("Test accuracy:", accuracy_score(y_test, y_pred))

Each step in the pipeline is applied exclusively to the training data during fitting, then reused on the test set, preventing leakage.

8. Neglecting Domain‑Specific Error Costs

In many real‑world applications, the cost of a false negative can be orders of magnitude higher than that of a false positive (e.g., fraud detection, medical diagnosis). Using a generic metric like accuracy ignores these asymmetries.

Cost‑sensitive learning incorporates these weights directly into the optimisation objective. For instance, in scikit‑learn’s RandomForestClassifier, the class_weight parameter can be set to 'balanced' or a custom dictionary, giving more importance to the minority class during tree splitting.

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=200, class_weight={0:1, 1:10})
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)

print("Weighted accuracy:", accuracy_score(y_test, y_pred, normalize=True))

In this example, the minority class (label 1) is weighted ten times more heavily, encouraging the model to reduce false negatives at the expense of an increased number of false positives.

9. Ignoring Temporal or Spatial Correlations

Metrics calculated on randomly split data assume that each observation is independent. In time‑series or spatial datasets, this assumption is violated. A model that performs well on a random split may fail in a real deployment where recent observations are correlated.

Cross‑validation techniques such as group‑kfold or time‑series split preserve these dependencies. Reporting metrics from these specialised resampling methods provides a more realistic estimate of future performance.

from sklearn.model_selection import TimeSeriesSplit
from sklearn.metrics import recall_score

tscv = TimeSeriesSplit(n_splits=5)
recalls = []

for train_idx, test_idx in tscv.split(X):
    model.fit(X[train_idx], y[train_idx])
    y_pred = model.predict(X[test_idx])
    recalls.append(recall_score(y[test_idx], y_pred))

print("Temporal recall mean:", np.mean(recalls))

By honouring the temporal structure, you avoid optimistic bias in recall estimation.

10. Assuming Monotonic Relationships Between Metrics

It is a common mistake to think that improving one metric will automatically improve others. In reality, metrics can move in opposite directions; for example, increasing recall typically reduces precision.

Plotting precision–recall curves for a range of thresholds can reveal this trade‑off. The area under this curve (AUPRC) is a more holistic measure when classes are imbalanced.

from sklearn.metrics import precision_recall_curve, auc

prec, recall, _ = precision_recall_curve(y_true, y_proba)
auprc = auc(recall, prec)

print("AUPRC:", auprc)

Understanding the shape of this curve guides threshold selection and reveals whether a model is genuinely improving performance or merely shifting the balance between precision and recall.

Additional Enrichment: Cohen’s Kappa for Agreement Evaluation

Cohen’s Kappa measures agreement between predicted and true labels beyond chance, making it useful when class distributions are skewed. The following example demonstrates its calculation and interpretation.

from sklearn.metrics import cohen_kappa_score

kappa = cohen_kappa_score(y_true, y_pred, weights='quadratic')
print("Cohen's Kappa (quadratic):", kappa)

Table comparing common metrics and their typical use cases:

Conclusion

Metrics are powerful tools, but they are only as useful as the context in which they are interpreted. By recognising the pitfalls outlined above – from class imbalance to threshold selection, from baseline neglect to data leakage – practitioners can make more informed decisions and present a truthful picture of model performance. The code snippets provided illustrate concrete ways to implement best practices, ensuring that your evaluations are robust, transparent and aligned with the underlying business or scientific objectives.

Further Reading

• Witten, I. H., Frank, E. & Hall, M. A. (2016) — Data Mining: Practical Machine Learning Tools and Techniques. A comprehensive guide that explains accuracy, precision, recall and F1-score with practical examples.
• Goodfellow, I., Bengio, Y. & Courville, A. (2016) — Deep Learning. Chapter 7 provides a clear mathematical derivation of evaluation metrics for classification tasks.
• Hastie, T., Tibshirani, R. & Friedman, J. (2009) — The Elements of Statistical Learning. Section 4.3 discusses performance measures and their interpretation in supervised learning.
• Scikit-learn.org (2024) — Classification metrics documentation. Offers code snippets and a concise description of metrics used in model assessment.
• IEEE Standards Association (2018) — IEEE Standard for Machine Learning Evaluation Metrics. Defines the accepted terminology and calculation methods for accuracy, precision, recall and F1-score.