Skip to main content

Monitoring and logging deployed ML models

Learning
Complete

Your model may be performing perfectly today — but will it still hold up tomorrow?

After successful deployment, continuous monitoring and robust logging are essential to ensure your model behaves reliably, delivers accurate predictions, and maintains its effectiveness in a changing environment.

Even the best-performing models degrade over time. Why? Real-world data changes. Customer behaviour shifts. Systems evolve. If you're not watching, your model might silently drift away from business value.

In this section, you’ll learn how to design a monitoring strategy that keeps your ML models accountable — tracking not just technical metrics, but also the quality and stability of predictions over time.

Why monitoring matters​

ML models are not static assets— they're dynamic systems that interact with real-world data, and that data doesn’t stand still. Common causes of model degradation include:

Skills Application illustration

  • Data drift: The distribution of incoming data shifts over time (e.g., new customer segments emerge, product trends shift).

  • Model drift: The model’s ability to predict accurately weakens as the relationship between inputs and outputs changes.

  • Concept drift: The underlying definition of what you're predicting changes (e.g., what counts as ‘fraud’ evolves due to regulation or tactics). Left unchecked, these drifts can impact business-critical KPIs, damage user trust, and reduce the return on investment of your ML efforts.

What to monitor: Metrics that matter​

To spot issues early, your monitoring setup should track three key categories of metrics:

1. Operational metrics

These help you monitor the health of your ML system:

  • Latency: Time taken to serve predictions.
  • Throughput: Number of predictions made per second or minute.
  • Error rate: Frequency of failed requests or exceptions.
  • Resource usage: CPU, memory, and GPU utilisation. **Example:**A recommendation system starts showing higher latency during evening traffic spikes, affecting user experience.

2. Model performance metrics​

These help you track how well your model is performing on its predictions:

  • Classification: Accuracy, precision, recall, F1-score, ROC AUC.
  • Regression: RMSE, MAE, R².
  • Comparison to baseline: Is the model still outperforming a rules-based system or the previous version? Example: A fraud model’s recall drops significantly after six months, likely due to emerging fraud patterns.

3. Data quality metrics​

These identify whether the input data is still valid and expected:

  • Schema checks: Are all required features present? Missing values: Are certain inputs becoming unreliable?
  • Distribution shifts: Are key features drifting?
  • Outliers or anomalies: Are you receiving data you didn’t train for? **Example:**An HR attrition model sees a spike in out-of-range values after a new system starts exporting data in a different format.

Logging, alerting, and feedback loops​

Monitoring is only as good as the information you collect and what you do with it. That’s where structured logging, alerting systems, and feedback loops come in.

Logging: Your model’s black box recorder Logging helps you understand exactly what your deployed model is doing. You should log:

  • Inputs received by the model.
  • Predictions and confidence scores.
  • Timestamps and request IDs.
  • Errors, exceptions, or unusual behaviour.

Key points​

Use structured logging (such as JSON) to ensure logs are machine-readable and easily searchable. Send logs to centralised platforms such as theELK stack,Splunk, orcloud-native tools (e.g., AWS CloudWatch, GCP Stackdriver, Azure Monitor) for scalable storage, search, and analysis.

Alerting: Responding when things go wrong​

Logging alone isn’t enough — you also need systems to tell you when action is necessary. Common alerting strategies include:

  • Threshold-based alerts: For example, trigger a notification if model latency exceeds 300ms or accuracy drops below 85%.
  • Drift detection alerts: Flag when incoming data significantly deviates from the training distribution using statistical tests.
  • Operational alerts: Monitor for spikes in error rates or failed predictions.

Feedback loops: Learning from real-world outcomes​

To keep your model relevant and accurate over time, build feedback mechanisms that capture what actually happened after a prediction:

  • Compare predictions to real outcomes (e.g., ‘Was the customer who churned predicted correctly?’).
  • Store new, labeled data for retraining and analysis.
  • Schedule regular retraining cycles based on performance and new data availability. <g></g><defs><clipPath><rect width="24" height="24" fill="white"></rect></clipPath></defs>## Example: Predictive maintenance in manufacturing A factory deploys an ML model to predict equipment failures based on real-time sensor data. Initially, the model performs well, flagging machines likely to require maintenance within the next 48 hours. But over time, the environment changes — the factory introduces new machinery, operational schedules shift, and seasonal temperature variations affect sensor readings.

To keep the model effective, the team implements a full monitoring, logging, and feedback loop:

  • Structured logs capture every inference request, including machine ID, sensor readings, predicted failure probability, timestamp, and actual maintenance outcome.
  • Alert configurations trigger if the model's precision or recall drops below a defined threshold or if input data starts to drift (e.g., sudden changes in vibration sensor distributions).
  • Feedback collection links actual equipment failure reports from the maintenance system back to the model’s predictions, allowing the team to measure false positives and false negatives. Based on this loop, the team re-trains the model every quarter using newly labeled data. This process helps the team maintain high prediction accuracy, reduce unexpected machine downtime, and avoid unnecessary maintenance costs. Over time, they also discover that some sensors contribute more to drift than others, prompting a hardware upgrade strategy.

Action item: Skillable lab — detecting data drift in deployed ML models​

In this skills application, you will complete a hands-on lab guiding you through simulating a data drift scenario and detecting it using statistical divergence measures in Python.

Important access information​

Please set aside 30 uninterrupted minutes to complete the lab before opening it. Your lab attempt will expire after 30 minutes. While you are welcome to restart, please keep in mind that you are limited to a maximum of five attempts at the lab within 90 days from your initial attempt.

Complete the skills application​

Launch this unit's Skillable lab: Detecting data drift in deployed ML models

Note: Clicking ‘Launch’ will open a new window and take you to Skillable's virtual lab environment. To use the lab, you will leave the Multiverse platform and will then need to create an account with Skillable. Skillable's Terms and Conditions and policies will apply to your use of its labs. Using the Skillable lab environment is highly encouraged to help you practise these concepts, but is completely optional. If you or your employer does not wish to proceed with Skillable, then do not click this link and simply continue your learning experience with Multiverse.

Context​

In production, ML models can begin making poor predictions — not because the code changed, but because the data changed. This lab simulates such a situation: You’ll load an original dataset and compare it to a slightly ‘drifted’ version, where one or more feature distributions have shifted.

You’ll write a Python script to calculate statistical divergence (KL divergence or Jensen-Shannon divergence) to quantify the difference, helping detect data drift.

Instructions and materials​

To complete the skills application, you must:

Time​

Set aside 30 minutes of uninterrupted time before beginning.

Accessing Skillable labs​

If this is the first time you are accessing a Skillable lab through Multiverse, follow the registration instructions below.

Open the lab​

Open the guided lab on Skillable: Detecting data drift in deployed ML models.

Log in​

Log in with your Skillable account and launch the lab.

Accept the T&Cs​

Accept Skillable's Terms and Conditions. As a reminder, if you or your employer do not wish to agree to the Skillable Terms and Conditions, then you should exit the lab, return to this page, and continue your learning experience on Multiverse.

Using the lab​

When using Skillable, follow the step-by-step guidance embedded within the activity.

Completion​

Once you have finished on Skillable, return to this page in your browser and complete your learning journey with Multiverse.