Understanding model drift and its impact

Instruction and application

In Progress

When models go off track, they don’t raise their hand — they drift. You won’t hear an alarm. You won’t see an error message. But one day, your ML model may start making poorer predictions, increasing business risk while everything looks fine on the surface. This subtle performance degradation is often the result ofmodel drift— and recognising it early is one of the most important skills in managing ML systems in production.

In this section, you’ll learn what model drift is, why it happens, how to measure it and how to interpret its impact on model performance. Let’s break it down step by step.

What is model drift?

Model drift refers to the phenomenon where an ML model’s performance deteriorates over time because the data it encounters in production changes from the data it was trained on.

There are several flavors of model drift:

• Data drift (or input drift): Changes in the distribution of input data over time.
• Concept drift: Changes in the underlying relationship between features and the target variable.
• Feature drift: Shifts in the significance or role of specific features in making predictions.
• Label drift: Changes in the distribution or definition of the target variable.

Drift example

Imagine you're managing an e-commerce platform's ML model that predicts which products users are likely to click on. Over time, you notice different types of drift affecting the model: Data drift (input drift) Initially, most customers browse on desktop devices, but production data shows a growing shift to mobile users. The input feature distribution (device type) has changed.Concept drift Originally, users who clicked on budget tech accessories were more likely to purchase. Recently, user preferences shifted — now, premium gadgets are driving purchases. The model’s learned relationship between product category and user clicks is no longer valid.Feature drift The discount_percentage feature was a strong predictor during training. However, as discounts become less common, its influence on predictions decreases, reducing its predictive power in production. Label drift The proportion of users who click on recommended products (the target variable) increases significantly during a holiday sale period, altering the baseline click-through rate (CTR) compared to training.

Each of these drift types affects model performance differently — recognising which type is occurring is essential for selecting the right remediation strategy.

Why does model drift occur?

Model drift is inevitable in most real-world systems because the environments in which models operate are dynamic. Here are some common causes:

• Shifting user behaviour: User patterns change over time, making old inputs less relevant (e.g. customers shopping differently due to economic trends).
• Seasonal changes: Models can struggle to adjust to time-based fluctuations (e.g. retail patterns during holiday seasons).
• Upstream data changes: Changes in data structure or format can break model inputs (e.g. new data pipeline formats or schema changes).
• External events: Sudden external shifts can disrupt learned relationships (e.g. a global event alters business dynamics).
• Concept evolution: The meaning of prediction targets may shift gradually (e.g. what qualifies as ‘spam’ evolves over time).

Why is drift a problem?

If unaddressed, drift can lead to:

• Reduced accuracy and performance.
• Incorrect or biased predictions.
• Bad business decisions based on flawed outputs.
• Loss of trust in your ML systems.

Types of data drift

Not all drift is the same — and treating it that way can lead to poor diagnosis and ineffective fixes. To respond appropriately, you need to understand what is changing — the inputs, the relationships or the outcomes. Select the tabs to learn about three common types of data drift you’re likely to encounter in production ML systems, each with distinct causes and implications. Covariate shift:What it is: A change in thedistribution of input features P(X), while therelationship to the target P(Y|X) remains the same.

• Why it matters: The model still applies the correct logic, but struggles because the input data no longer resembles the training data.
• Common causes: New user segments, geographic expansion and changing customer behaviours.
• Example: A churn prediction model trained on users aged 25-40 begins receiving input from younger users (18-24). Even if churn behaviour hasn't changed, the model may perform poorly because it wasn’t trained on this demographic.Concept drift:What it is: A change in therelationship between inputs and the target variable P(Y|X)— the same inputs now lead to different outcomes.
• Why it matters: The model’s logic becomes outdated, and predictions no longer reflect real-world behaviour.
• Common causes: Evolving business rules, shifting market behaviour and adversarial adaptation.
• Example: A fraud detection model flags certain transaction patterns as risky. Over time, fraudsters adapt, those same patterns become legitimate, and new fraud patterns emerge.Label shift:What it is: A change in thedistribution of the target variable P(Y), even though input features stay relatively stable.
• Why it matters: The model misestimates the frequency or importance of certain outcomes, leading to misclassification or skewed predictions.
• Common causes: Pricing changes, market shifts, product launches or seasonal effects.
• Example: A model trained during a stable period predicts low churn. After a subscription price increase, churn rises — but the model underpredicts it because the class distribution has shifted.

How do we measure data drift?

You can’t fix what you don’t measure. Detecting drift means comparing the statistical characteristics of the data your model sees in production with those it was trained on.

Below are the most common techniques used to detect and quantify drift across different types of features.

For numerical features:

• Population stability index (PSI): Compares how the distribution of a feature shifts between two time periods — easy to interpret and widely used in production settings.
• Kullback-Leibler (KL) divergence: Measures how much one probability distribution diverges from another. Sensitive to small changes but unstable if any bucket has a value of zero.
• Jensen-Shannon (JS) divergence: A symmetric and smoothed version of KL — more stable and interpretable for comparing probability distributions.
• Wasserstein distance: Also known as ‘earth mover’s distance,’ it calculates how much effort is needed to transform one distribution into another — especially useful when the shape of the distribution matters.

For categorical features:

• Chi-squared test: Tests whether the frequency distribution of categories differs significantly between two datasets — a common choice for categorical drift detection.
• PSI: Also applicable to categorical features in categories grouped into discrete bins. Offers a consistent way to compare both numerical and categorical drift.

Beyond statistics: Practical monitoring techniques

Statistical methods are valuable, but in fast-moving environments, additional techniques such as A/B testing and visual inspection help catch drift earlier and provide clearer insights into real-world model behaviour.

• A/B testing with new vs. existing model versions: Compare performance over time or across segments to detect behaviour change before full rollout.
• Visual inspection (e.g. histograms, distribution plots): Often, the quickest way to spot noticeable shifts in feature distributions or outliers.
• Rolling performance metrics (e.g. moving averages of accuracy or loss): Help highlight slow performance degradation that might go unnoticed in static evaluations.

Challenges in measuring drift

Even with the right tools, detecting drift is not always straightforward. Some common challenges include:

• High dimensionality: The more features you have, the more opportunities for drift —and false alarms.
• Mixed data types: Working with combinations of numerical, categorical and textual features adds complexity to detection.
• Streaming data: Continuous data requires near-real-time monitoring systems and efficient drift checks.
• False signals: Not every statistical difference leads to real-world performance impact — overreacting can be just as harmful as underreacting.

Impact assessment: Connecting drift to performance

Once you've detected drift, the next question is, does it matter? Not all drift leads to model failure, so it's essential to link changes in data to changes in outcomes.

Drift is a warning signal — but is it actually hurting your model?

To answer that, correlate drift measurements with performance metrics over time:

• Has a spike in PSI for a key feature coincided with a drop in F1-score or accuracy?
• Is a shift in label distribution tied to a rise in false positives?
• Is there a threshold of distributional change that consistently signals when performance declines? This kind of analysis helps you prioritise which drift signals require action and which are harmless noise.

Tip

Always establish baseline data distributions from your training set. These serve as the reference for evaluating incoming production data.

Practical scenario: Drift in a customer churn model

Your company has deployed a model that predicts which users are likely to cancel their subscriptions. The model was trained six months ago, primarily on mid-career professionals in the 30-45 age range.

Recently, you have noticed that your churn rate has climbed unexpectedly. You run your monitoring checks and find the following:

• Covariate shift: PSI for customer_age has increased significantly because there's a growing segment of users aged 18-24 entering the platform.
• Label shift: The overall churn rate (P(Y)) has jumped after a recent subscription price increase.
• Performance drop: F1-score has decreased by 15%, with a noticeable rise in false negatives. You run a chi-squared test on the categorical feature subscription_type and detect statistically significant changes in user distribution across plan tiers.Impact? The model is no longer generalising effectively to the new customer base and is underestimating churn among younger users. Based on these findings, you prepare to retrain the model with updated data and adjust alert thresholds for drift going forward.

Action item: Lab — detecting data drift

In this lab, you will step into the role of an ML practitioner responsible for monitoring a churn prediction model. You’ll compare training and production datasets to identify potential data drift using statistical methods.

This exercise will help you practice key monitoring skills, including:

• Quantifying numerical feature drift using the PSI.
• Detecting categorical feature distribution changes using a chi-squared test.
• Interpreting drift indicators to decide when intervention is necessary. To begin the lab, download the Jupyter Notebook and the datasets you need from the materials section below.

Your task

Open the notebook and follow the steps to:

• Load and inspect the datasets.
• Calculate PSI and chi-squared statistics.
• Interpret whether data drift has occurred. Your notebook contains all the code and instructions you need to perform these tasks.

Materials

Download the Jupyter Notebook:

M11_U1_Detecting_Data_Drift_Lab.ipynb

Download the datasets for this lab:

M11_U1_Training_data_drift_activity.csv

M11_U1_Production_data_drift_activity.csv