Methods for identifying potential threats

1) Why “threat identification” is the first real security step

Before you can defend an ML system, you must know:

• What you’re protecting (assets)
• Who might attack (adversaries)
• How they can attack (attack paths)
• What success looks like for them (impact)

In ML, threats include not only classic software security issues (auth bypass, injection, DoS), but also ML-specific threats like evasion, poisoning, extraction, and privacy attacks.

2) A practical threat identification workflow (recommended)

Step 1 — Asset inventory (what matters)

Identify all assets an attacker would care about:

Model assets

• model weights, architecture, hyperparameters
• prompts/system prompts (for LLM systems)
• decision thresholds and business rules

Data assets

• training data, labels, feature store
• evaluation datasets and test harness
• user feedback signals (retraining inputs)

System assets

• inference API, model server, CI/CD
• secrets (API keys, DB creds), IAM roles
• logs/telemetry (often contain sensitive info)

Outcome assets

• business decisions (loan approvals, fraud flags)
• safety outcomes (autonomy actions, medical decisions)
• reputation and compliance posture

Output of Step 1: a list of assets + owner + location + sensitivity level.

Step 2 — Dataflow diagram (DFD) of the ML system

Draw the system end-to-end:

• data sources → ingestion → preprocessing → feature store
• training pipeline → model registry → deployment
• inference API → post-processing → downstream decisions
• monitoring → feedback → retraining

Threat identification becomes easier when you can point to where data enters, transforms, and exits.

Step 3 — Identify trust boundaries

Mark boundaries where control changes:

• public internet → API gateway
• third-party data source → your ingestion pipeline
• human labelers → label store
• production logs → analytics warehouse
• model registry → deployment environment

Most attacks occur at trust boundaries.

Step 4 — Apply a threat taxonomy (use a checklist)

Use structured methods so you don’t miss whole categories.

3) Method 1: STRIDE (adapted for ML)

STRIDE is a classic threat model framework. Here’s how it maps to ML:

S — Spoofing

• fake users/devices sending inputs
• replay attacks (same input reused)
• fake labeler identities

Controls: AuthN, device attestation, nonces/timestamps, labeler verification

T — Tampering

• training data poisoning
• label flipping
• feature store injection
• model artifact replacement

Controls: signed artifacts, RBAC, immutability, checksums, provenance

R — Repudiation

• attackers deny actions due to poor logging/audits

Controls: audit logs, immutable logs, trace IDs, retention policies

I — Information disclosure

• membership inference, model inversion
• leaking confidence scores/logits
• leaking PII via logs

Controls: output shaping, privacy controls, PII redaction, access control

D — Denial of service

• query floods, expensive inputs (GPU exhaustion)
• parser crashes, oversized payloads

Controls: WAF, rate limiting, payload limits, circuit breakers

E — Elevation of privilege

• container escape, IAM over-permission
• CI/CD compromise to deploy malicious model

Controls: least privilege, secrets management, hardened containers, approvals

4) Method 2: Attack surface mapping by lifecycle stage (ML-specific)

Identify threats at each lifecycle stage:

Training-time threats

• poisoning (data/labels)
• backdoors
• supply-chain compromise (datasets, dependencies)

Inference-time threats

• evasion attacks (adversarial examples)
• black-box probing
• model extraction

Post-deployment threats

• drift exploitation
• feedback loop poisoning
• monitoring evasion (staying below thresholds)

This method is especially useful because ML attacks differ by stage.

5) Method 3: Adversary modeling (who attacks and why)

Classify adversaries by capability:

• Opportunistic: tries easy exploits (basic evasion, weak auth)
• Insider: has access to data/labels/model artifacts
• Competitor: wants model extraction or sabotage
• Organized attacker: patient, adaptive, uses black-box strategies
• Supply-chain attacker: compromises dependencies, datasets, CI/CD

For each adversary, define:

• access level (API only? feature store access? label tool access?)
• resources (compute, time, team)
• motivation (financial, disruption, political, reputational)

6) Method 4: Misuse cases and abuse stories (highly practical)

Write “misuse cases” like user stories, but for attackers.

Examples:

• “As an attacker, I want to submit near-duplicate inputs repeatedly to infer the decision boundary and bypass fraud detection.”
• “As a malicious labeler, I want to flip labels for a subset of cases to degrade model accuracy without being noticed.”

This method works well with product and ops teams because it’s concrete.

7) Method 5: Risk scoring and prioritization

After listing threats, score them:

Simple scoring model:

• Impact (1–5): harm if exploited (safety, money, privacy)
• Likelihood (1–5): exposure + feasibility
• Detectability (1–5): how quickly you’d notice (lower detectability = higher risk)

Risk score example:

Use this to decide:

• what to fix now (high risk)
• what to monitor (medium)
• what to accept (low)

8) ELMS-ready Threat Identification Worksheet