Skip to main content

Async review

Instruction and application
In Progress

Recap core topics:​

  • Unit 1: Model monitoring and adaptation.

Unit 1: Model monitoring and adaptation​

In Unit 1, you explored…

  • Model drift and its impact: This topic discusses how data drift, concept drift, feature drift and label drift quietly degrade performance in production ML systems.
  • Causes of drift: These include shifting user behaviour, seasonal patterns, upstream data changes, external events and concept evolution.
  • Measuring drift: This topic demonstrates statistical methods such as PSI, KL divergence, chi-squared, Wasserstein distance and complementary approaches such as A/B testing, visual inspection and rolling performance metrics.
  • Linking drift to performance: This action connects drift signals with business-critical metrics (accuracy, precision, recall, F1-score, RMSE, AUC-ROC) to prioritise responses.
  • **Monitoring infrastructure:This section covers **how to design effective dashboards, logging inputs/outputs, feature stats, latency and building pipelines for timely, auditable monitoring.
  • Model maintenance strategies: These include balancing reactive vs proactive maintenance, retraining triggers, versioning and using feature adaptation, ensembles and human-in-the-loop feedback.
  • Automated testing workflows: These include shadow, canary, regression and integration testing to ensure reliable model updates.

Types of drift​

  • Data drift: There is a shift in input features (e.g. customer age distribution changes).
  • Concept drift: This shows the relationship between input and target shifts (e.g. age no longer predicts churn the same way).
  • Feature drift: One feature changes disproportionately to others (e.g. subscription type categories).
  • Label drift: There is a shift in the target variable distribution (e.g. churn rate spikes).

Statistical methods for detecting drift​

  • **Population stability index (PSI):**Widely used in industry to flag distribution shifts in features over time.
  • **KL divergence:**Measures how one probability distribution diverges from another.
  • Chi-squared test: Compares observed vs expected frequencies for categorical variables.
  • Wasserstein distance: Captures differences in continuous feature distributions.

Action item: Poll — how well do you know drift?​

It's time for a quick drift-focused poll. This will help you check your understanding of the main types of model drift and how you detect them in practice. No pressure — just go with your best judgement!