Skills application
Instruction and application
In ProgressDesigning a basic monitoring system
In this skills application, you will design a basic monitoring system for an ML model deployed in production. You will identify key performance metrics, potential data drift indicators, and outline a response strategy to maintain the model’s long-term effectiveness.
This task mirrors real-world monitoring responsibilities for ML engineers and data scientists.
Context
You are responsible for an ML model that predicts customer churn for a subscription-based service. The model uses features such as customer demographics, subscription history, website activity and support ticket interactions.
Recently, leadership has requested a robust monitoring system design to ensure that the model’s performance remains reliable and adaptive as customer behaviour and data patterns evolve.
Your task is to build a basic ML monitoring system mockup, detailing:
- Key metrics to track model performance.
- Potential sources of data or concept drift.
- Dashboard design to visualise model health.
- A maintenance protocol combining proactive and reactive actions.
Success criteria
To successfully complete the skills application, you must:
- Identify and justify at least five key metrics to monitor the model’s performance.
- Identify at least three potential drift sources, along with indicators that would help detect them.
- Describe the layout and key components of a monitoring dashboard mockup.
- Define alert thresholds for at least two metrics or indicators.
- Outline a maintenance protocol, including both proactive measures and reactive responses.
- Explain how you would incorporate automated testing into your monitoring and retraining workflows.
Instructions and materials
Follow the instructions below to complete the skills application. Completing this activity will unlock the solution example on the following page.## Review the scenario Review the scenario and requirements carefully.
Tasks
- Complete the tasks below by summarising your answers in a structured report format:
Section 1: Key metrics
List at least five quantitative metrics you would track to monitor the model’s performance.
- Justify why each metric is important for the business goal (reducing churn).
Section 2: Drift sources and indicators
Identify three types of drift that could impact the model (data drift, concept drift, label drift).
- For each, specify which feature(s) you would monitor and whatindicator (PSI, KL divergence, etc.) would reveal the drift.
Section 3: Dashboard layout and alerts
Describe the key sections/visualisations you would include on a monitoring dashboard.
- Define alert thresholds for at least two metrics/indicators and explain why those thresholds matter.
Section 4: Maintenance protocol and testing
Outline a basic maintenance protocol, combining scheduled checks and drift-triggered actions.
-
Describe your primary response if testing detects significant concept drift.
-
Explain how you would incorporate automated testing (e.g. regression testing, shadow deployments) into your update workflow.Use the form below to submit your report.## Go deeper After completing the activity, consider these questions:
-
How would your monitoring system need to evolve if customer behaviour starts changing rapidly due to external market factors?
-
What trade-offs might you face when deciding how frequently to retrain the model?
-
How can you balance automation with human oversight in monitoring ML systems?