Benchmarking and real-world evaluation strategies

1) Why benchmarking is different from “just running PGD once”

Robustness evaluation fails most often because teams:

• test against a weak attack,
• test only one threat model,
• evaluate only offline (clean datasets),
• ignore deployment realities (latency, queries, drift, human constraints).

A good robustness benchmark answers:

1. How robust is the model under a clearly defined attacker?
2. How does robustness change across ε budgets and data slices?
3. Does robustness hold in production conditions (shift, noise, APIs, monitoring)?
4. Is performance acceptable (accuracy, latency, cost) after defenses?

2) Benchmarking goals (what you’re trying to measure)

A) Comparative robustness (model vs model)

• Compare architectures/training strategies fairly
• Standardize datasets, attacks, and parameters

B) Regression robustness (model version vs version)

• Ensure a new model release didn’t silently reduce robustness

C) Operational robustness (real-world readiness)

• Can this model stay safe when inputs, users, and attackers change?

3) Core benchmarking dimensions (must be defined)

3.1 Threat model matrix

Benchmark across multiple threat assumptions:

• Norms: L∞, L2 (and domain-specific constraints for tabular/text)
• Budgets: ε grid (e.g., 0.05, 0.1, 0.2, 0.3 for scaled images)
• Attacker knowledge: white-box vs black-box
• Objective: untargeted vs targeted
• Query constraints: rate limits, noise in outputs

A single “robust accuracy at ε=0.3” is not a benchmark; it’s a point estimate.

4) Standard robustness benchmark suite (recommended baseline)

4.1 Attack suite (minimum viable)

Run at least:

• FGSM (fast sanity check)
• PGD (strong first-order baseline; include restarts)
• One additional family (transfer or black-box style)

Why restarts matter: PGD can get stuck; multiple random restarts reduce false robustness.

4.2 Metrics to report (benchmark table)

For each ε:

• Clean Accuracy
• Robust Accuracy (per attack)
• Worst-case Robust Accuracy (across suite)
• ASR + conditional ASR
• Robustness gap (Clean − Robust)

Optional but strong:

• Robustness curve AUC
• Per-class robust accuracy
• Calibration under attack

5) Real-world evaluation strategies (beyond academic benchmarks)

Strategy A — Distribution shift + “natural corruptions”

Adversarial attacks are not the only stressor. Real systems face:

• blur, noise, compression
• lighting changes
• sensor drift
• OCR errors, typos
• missing/incorrect tabular fields

Evaluate:

• performance under corruptions (not just adversarial perturbations)
• robustness under combined conditions: corruption + adversarial

Practical takeaway: A model robust to PGD but fragile to real-world noise is still risky.

Strategy B — Slice-based robustness (fairness + fragile subgroups)

Robustness is rarely uniform. Benchmark by slices:

• class label (some classes break first)
• demographic/environment subgroup (if applicable and ethical)
• device/camera type
• input length for NLP
• rare vs common patterns in tabular

Report: worst-case slice performance, not just averages.

Strategy C — API threat model (black-box, query-based)

Many real attackers don’t have gradients—they have an API.

Evaluate:

• query-limited attack success (budgeted queries)
• effect of rate limiting
• effect of output restriction (labels only vs probabilities)
• anomaly detection on query patterns

Goal: measure whether your deployed interface leaks too much.

Strategy D — Adaptive evaluation (defense-aware attacker)

If you use preprocessing, randomness, or gradient obfuscation risk:

• evaluate with stronger/adaptive settings (EOT, transfer checks)
• verify that “robustness” is not just broken gradients

Strategy E — End-to-end system evaluation

Model robustness can be defeated at the pipeline level:

• preprocessing bugs
• normalization mismatch
• feature leakage
• postprocessing thresholds that can be manipulated

Benchmark the whole pipeline, not just the model.

6) Benchmarking workflow (ELMS-ready playbook)

Step 1 — Lock the benchmark configuration

• dataset version + split
• preprocessing version hash
• model version hash
• attack configs (ε grid, steps, restarts, norms)
• compute environment (seed, framework version)

Step 2 — Run baseline report

• clean accuracy
• robust accuracy under suite
• worst-case robust accuracy
• robustness curve

Step 3 — Stress for realism

• add corruptions / noise
• add slice metrics
• add API constraints simulation (if deployed)

Step 4 — Publish a “Robustness Model Card”

Include:

• threat model scope (what is covered)
• limitations (what is not covered)
• recommended safeguards (rate limits, monitoring, fallbacks)

7) Code: Benchmark PGD robust accuracy across ε + restarts (TF2 + CleverHans)

Assumes:

• model produces logits
• x_test, y_test_oh exist

import numpy as np
import tensorflow as tf
from cleverhans.tf2.attacks.projected_gradient_descent import projected_gradient_descent

def robust_acc_pgd_with_restarts(model, x, y_onehot, eps, eps_iter=0.03, steps=20, restarts=5):
    """
    For each restart, run PGD (starting from random point inside eps-ball).
    Keep the WORST-CASE prediction per sample (attacker succeeds if any restart succeeds).
    Returns robust accuracy under multi-restart PGD.
    """
    y_true = tf.argmax(y_onehot, axis=1)
    # Start by assuming all samples are robust; we will flip to false if any restart breaks them.
    broken = tf.zeros([tf.shape(x)[0]], dtype=tf.bool)

    for _ in range(restarts):
        x_adv = projected_gradient_descent(
            model_fn=model, x=x,
            eps=eps, eps_iter=eps_iter, nb_iter=steps,
            norm=np.inf, targeted=False
        )
        preds_adv = tf.argmax(model(x_adv, training=False), axis=1)
        broken = tf.logical_or(broken, tf.not_equal(preds_adv, y_true))

    robust_acc = 1.0 - float(tf.reduce_mean(tf.cast(broken, tf.float32)))
    return robust_acc

def benchmark_over_eps(model, x, y_onehot, eps_list, eps_iter=0.03, steps=20, restarts=5):
    results = []
    clean_acc = float(model.evaluate(x, y_onehot, verbose=0)[1])
    for eps in eps_list:
        ra = robust_acc_pgd_with_restarts(model, x, y_onehot, eps, eps_iter=eps_iter, steps=steps, restarts=restarts)
        results.append({"eps": eps, "clean_acc": clean_acc, "pgd_robust_acc": ra,
                        "pgd_steps": steps, "pgd_restarts": restarts, "norm": "Linf"})
    return results

# Example:
# eps_grid = [0.0, 0.05, 0.1, 0.2, 0.3]
# report = benchmark_over_eps(model, x_test, y_test_oh, eps_grid, eps_iter=0.03, steps=20, restarts=5)
# for row in report:
#     print(row)

Why this matters: multi-restart PGD is closer to a real attacker than PGD with a single start.

8) Code: Add “natural corruption” benchmarking (simple noise/blur examples)

These are not full benchmark suites like ImageNet-C, but show how to incorporate corruption testing.

def add_gaussian_noise(x, sigma=0.1):
    noise = tf.random.normal(shape=tf.shape(x), stddev=sigma)
    return tf.clip_by_value(x + noise, 0.0, 1.0)

def gaussian_blur(x, ksize=3, sigma=1.0):
    ax = tf.range(-ksize//2 + 1, ksize//2 + 1, dtype=tf.float32)
    xx, yy = tf.meshgrid(ax, ax)
    kernel = tf.exp(-(xx**2 + yy**2) / (2.0 * sigma**2))
    kernel = kernel / tf.reduce_sum(kernel)
    kernel = kernel[:, :, tf.newaxis, tf.newaxis]
    return tf.nn.depthwise_conv2d(x, kernel, strides=[1,1,1,1], padding="SAME")

def corruption_benchmark(model, x, y_onehot):
    clean = float(model.evaluate(x, y_onehot, verbose=0)[1])
    noisy = float(model.evaluate(add_gaussian_noise(tf.constant(x), sigma=0.15), y_onehot, verbose=0)[1])
    blurred = float(model.evaluate(gaussian_blur(tf.constant(x), ksize=3, sigma=1.0), y_onehot, verbose=0)[1])
    return {"clean_acc": clean, "noise_acc": noisy, "blur_acc": blurred}

# Example:
# corr_report = corruption_benchmark(model, x_test, y_test_oh)
# print(corr_report)

9) ELMS-ready reporting format (benchmark card)

Robustness Benchmark Summary

• Dataset: __, split: __
• Threat model: norm=, ε grid=
• Attacks: FGSM, PGD-20 (restarts=__), transfer/black-box: __
• Clean accuracy: __
• Worst-case robust accuracy at ε=__ : __
• Robustness curve AUC (optional): __
• Corruption results: noise=, blur=
• Slice worst-case robust acc (optional): __
• Limitations: (patch attacks, semantic changes, distribution shift not fully covered, etc.)