Async review
Instruction and application
In ProgressRecap core topics:
- Unit 1: Model monitoring and adaptation.
Unit 1: Model monitoring and adaptation
In Unit 1, you explored…
Model drift and its impact:
This topic discusses how data drift, concept drift, feature drift and label drift quietly degrade performance in production ML systems.
Causes of drift:
These include shifting user behaviour, seasonal patterns, upstream data changes, external events and concept evolution.
Measuring drift:
This topic demonstrates statistical methods such as PSI, KL divergence, chi-squared, Wasserstein distance and complementary approaches such as A/B testing, visual inspection and rolling performance metrics.
Linking drift to performance:
This action connects drift signals with business-critical metrics (accuracy, precision, recall, F1-score, RMSE, AUC-ROC) to prioritise responses.
Monitoring infrastructure:
This section covers** **how to design effective dashboards, logging inputs/outputs, feature stats, latency and building pipelines for timely, auditable monitoring.
Model maintenance strategies:
These include balancing reactive vs proactive maintenance, retraining triggers, versioning and using feature adaptation, ensembles and human-in-the-loop feedback.
Automated testing workflows:
These include shadow, canary, regression and integration testing to ensure reliable model updates.
Types of drift
Data drift:
There is a shift in input features (e.g. customer age distribution changes).
Concept drift:
This shows the relationship between input and target shifts (e.g. age no longer predicts churn the same way).
Feature drift:
One feature changes disproportionately to others (e.g. subscription type categories).
Label drift:
There is a shift in the target variable distribution (e.g. churn rate spikes).
Statistical methods for detecting drift
Population stability index (PSI):
Widely used in industry to flag distribution shifts in features over time.
KL divergence:
Measures how one probability distribution diverges from another.
Chi-squared test:
Compares observed vs expected frequencies for categorical variables.
Wasserstein distance:
Captures differences in continuous feature distributions.
Action item: Poll — how well do you know drift?
It's time for a quick drift-focused poll. This will help you check your understanding of the main types of model drift and how you detect them in practice. No pressure — just go with your best judgement!