Skip to main content

Skills application

Instruction iconInstruction and application
Complete iconIn Progress

Designing a basic monitoring system​

In this skills application, you will design a basic monitoring system for an ML model deployed in production. You will identify key performance metrics, potential data drift indicators, and outline a response strategy to maintain the model’s long-term effectiveness.

This task mirrors real-world monitoring responsibilities for ML engineers and data scientists.

Skills Application illustration

Context​

You are responsible for an ML model that predicts customer churn for a subscription-based service. The model uses features such as customer demographics, subscription history, website activity and support ticket interactions.

Recently, leadership has requested a robust monitoring system design to ensure that the model’s performance remains reliable and adaptive as customer behaviour and data patterns evolve.

Your task is to build a basic ML monitoring system mockup, detailing:

  • Key metrics to track model performance.
  • Potential sources of data or concept drift.
  • Dashboard design to visualise model health.
  • A maintenance protocol combining proactive and reactive actions.

Success criteria​

To successfully complete the skills application, you must:

  • Identify and justify at least five key metrics to monitor the model’s performance.
  • Identify at least three potential drift sources, along with indicators that would help detect them.
  • Describe the layout and key components of a monitoring dashboard mockup.
  • Define alert thresholds for at least two metrics or indicators.
  • Outline a maintenance protocol, including both proactive measures and reactive responses.
  • Explain how you would incorporate automated testing into your monitoring and retraining workflows.

Instructions and materials​

Follow the instructions below to complete the skills application. Completing this activity will unlock the solution example on the following page.## Review the scenario Review the scenario and requirements carefully.

Tasks​

  • Complete the tasks below by summarising your answers in a structured report format:

Section 1: Key metrics​

List at least five quantitative metrics you would track to monitor the model’s performance.

  • Justify why each metric is important for the business goal (reducing churn).

Section 2: Drift sources and indicators​

Identify three types of drift that could impact the model (data drift, concept drift, label drift).

  • For each, specify which feature(s) you would monitor and whatindicator (PSI, KL divergence, etc.) would reveal the drift.

Section 3: Dashboard layout and alerts​

Describe the key sections/visualisations you would include on a monitoring dashboard.

  • Define alert thresholds for at least two metrics/indicators and explain why those thresholds matter.

Section 4: Maintenance protocol and testing​

Outline a basic maintenance protocol, combining scheduled checks and drift-triggered actions.

  • Describe your primary response if testing detects significant concept drift.

  • Explain how you would incorporate automated testing (e.g. regression testing, shadow deployments) into your update workflow.Use the form below to submit your report.## Go deeper After completing the activity, consider these questions:

  • How would your monitoring system need to evolve if customer behaviour starts changing rapidly due to external market factors?

  • What trade-offs might you face when deciding how frequently to retrain the model?

  • How can you balance automation with human oversight in monitoring ML systems?

Section 1: Key metrics

List at least five quantitative metrics you would track to monitor the model's performance.
0 words
Justify why each metric is important for the business goal (reducing churn).
0 words

Section 2: Drift sources and indicators

Identify three types of drift that could impact the model (data drift, concept drift, label drift).
0 words
For each, specify which feature(s) you would monitor and what indicator (PSI, KL divergence, etc.) would reveal the drift.
0 words

Section 3: Dashboard layout and alerts

Describe the key sections/visualisations you would include on a monitoring dashboard.
0 words
Define alert thresholds for at least two metrics/indicators and explain why those thresholds matter.
0 words

Section 4: Maintenance protocol and testing

Outline a basic maintenance protocol, combining scheduled checks and drift-triggered actions.
0 words
Describe your primary response if testing detects significant concept drift.
0 words
Explain how you would incorporate automated testing (e.g. regression testing, shadow deployments) into your update workflow.
0 words
In progress