Skip to main content

Understanding threats across the ML Lifecycle

Instruction and application
Complete

Every phase is an attack surface​

Machine learning systems may look seamless on the surface, but under the hood, each phase of the ML lifecycle is a potential attack surface waiting to be exploited.

From collecting raw data to maintaining models in production, every stage introduces unique vulnerabilities. In this section, you'll explore the most common threats faced during each phase of an ML workflow.

Lifecycle threats illustration

1. Data collection and ingestion threats​

Security begins at the data source. Vulnerabilities in early-stage data handling can silently corrupt performance or expose sensitive info.

  • Data poisoning: Malicious data is inserted into training sets to manipulate outputs. (Impact: Integrity)
  • Data leakage: Sensitive or PII is accidentally included in features. (Impact: Confidentiality)
  • Insecure pipelines: Data is transferred without encryption or validation. (Impact: Confidentiality, Authentication)

Example: Leakage​

A churn model includes raw customer notes containing health details, violating privacy policies and exposing the company to legal risk.

2. Model training threats​

The training phase is a high-value target for external and insider attacks due to privileged access and large datasets.

  • Insider threats: Authorized individuals inject malicious code or exfiltrate data. (Impact: Authentication, Non-repudiation)
  • Model inversion: Attackers reconstruct training data by analyzing outputs. (Impact: Confidentiality)
  • Insecure environments: Weakly configured infrastructure allows access to model artifacts. (Impact: Confidentiality, Integrity)

3. Model evaluation threats​

Failing to sanitize outputs or protect logs can give attackers the tools they need to reverse-engineer your system.

  • Unintended exposure: Reports or tools reveal sensitive data. (Impact: Confidentiality)
  • Evaluation-time attacks: Carefully crafted inputs reveal model structure or vulnerabilities. (Impact: Service Integrity)

4. Deployment threats​

Deployment opens the door to adversarial manipulation and model theft.

  • Adversarial input attacks: Modified inputs trick the model into wrong predictions. (Impact: Service Integrity)
  • Model theft (Extraction): Replicating your model by repeatedly querying it. (Impact: Confidentiality)
  • Unauthorised access: Exploiting unsecured APIs or endpoints. (Impact: Authentication)

5. Monitoring and maintenance threats​

Without active security monitoring, systems are vulnerable to subtle, long-term threats.

  • Membership inference: Determining if specific points were in the training set. (Impact: Confidentiality)
  • Drift exploitation: Exploiting behavior changes over time to find weaknesses. (Impact: Service Integrity)
  • Lack of audit trails: Missing logs make it impossible to attribute changes. (Impact: Non-repudiation)
Action item: Mini-scenario reflection
You’re at a fintech company. After retraining, fraud performance declines, a report contains exposed data, and API calls spike. What two types of security threats are described here?

Type your reflection here...

Which stages of the ML lifecycle do these threats occur in?

Type your reflection here...