Skip to main content

Practical exercise

Instruction and application
In Progress

ShopSmart drift response plan​

In this activity, you will act as ShopSmart’s data science team to analyse how model drift affects performance. You’ll identify drift sources, link indicators to metrics, and design a monitoring and response plan.

These skills are essential for organisations that rely on ML models in dynamic environments. By practising drift detection, monitoring design and response planning, you’ll learn how to safeguard model performance, reduce business risks and communicate effectively with stakeholders.

Practical exercise context​

Company: ShopSmart, a fast-growing online retailer.Model in use: ShopSmart uses an ML model topredict the likelihood of product returns. The model takes features such as customer age, purchase history, product category, order value and delivery speed. Predictions help the business reduce costs, optimise inventory and identify high-risk orders.Current challenge:

  • Seasonal changes: These changes (e.g. holiday shopping spikes) are shifting purchasing behaviour, making predictions less reliable.

  • New product categories: ShopSmart has added several new categories (e.g. electronics and home décor), changing input distributions.

  • Shifting demographics: Younger customers now make up a growing share of orders, changing key distributions such as customer age.

  • Monitoring results:PSI > 0.2 has been flagged forcustomer_age, and chi-squared tests forproduct_category show significant shifts. These changes coincide with a measurable drop inF1-score andAUC-ROC.

  • Version tracking gap: ShopSmart tracks model versions internally (e.g. V1.2, V1.3), but alerts are not currently tied to versions, making it hard to trace performance issues back to specific updates.

  • **Retraining baseline:**Retraining is currently scheduled quarterly, but this cadence may be too slow to address sudden shifts.

  • Feature adaptation needs: Certain features (e.g.customer_age,product_category) may require reweighting or adaptation, rather than always triggering full retraining.Business impact: Rising return rates are cutting into margins, creating inventory delays and frustrating customers. Leadership wants a clear monitoring and response plan to safeguard model reliability and minimise business risks.As part of the data science team, your challenge is to:

  • Identify likely sources of drift in ShopSmart’s system.

  • Connect drift indicators with performance metrics.

  • Design a monitoring dashboard to track drift and alert the team.

  • Propose testing and maintenance strategies to keep the model reliable in production.

Activity instructions​

Work with your group to complete the following tasks:

Review the context​

Start by reviewing ShopSmart’s situation. Focus on the seasonal shopping behaviour, the introduction of new product categories and shifting customer demographics that may be driving drift. Consider how these changes could impact both model inputs and business performance.

Diagnose drift sources​

Identify three potential sources of drift, and explain how they affect inputs or predictions.

Connect drift indicators with at least one performance metric.

Design a monitoring dashboard​

Describe what your monitoring dashboard would include:

  • Outline time-series plots to track drift indicators alongside performance metrics.
  • Suggest histograms or distribution comparisons for drifted features.
  • Propose an alert panel with thresholds, flagged features and model version details.

Plan triggers and responses​

Propose when and how retraining should occur, and explain the rationale.

Test retrained models safely​

Select one automated testing strategy (shadow, regression or canary) to validate retrained ShopSmart models before rollout, and justify why it best reduces risk.

Regroup and share your insights​

Return to the main session after 20 minutes to discuss key insights and recommendations.

Action item: Activity share out​

  • What is one drift signal your group identified in the ShopSmart model?
  • What is one monitoring or testing strategy you would recommend to manage that drift?
  • How would you explain the business impact of drift to a non-technical stakeholder?