Implementing model performance monitoring

Instruction and application

In Progress

What gets monitored gets improved. Once your model is deployed, its predictions become part of real-world decision-making, and small drops in performance can have a big impact. That’s why monitoring isn’t just a technical task — it’s a safeguard for your model’s business value.

In this section, you’ll learn how to monitor the right metrics, set thresholds that matter and design infrastructure that keeps your models under continuous watch.

Key performance metrics for ML models in production

You’ve already seen the standard metrics used for classification and regression. In this section, we’ll quickly recap those, then explore additional metrics used in other domains like natural language processing (NLP), computer vision and recommendation systems, so you can monitor any model type with confidence. Model typeMetricWhat it measuresClassification AccuracyOverall correctness of predictionsPrecision Correctness of positive predictionsRecallCoverage of actual positive casesF1-scoreBalance of precision and recallAUC-ROCClass separation across thresholdsLog lossConfidence in probabilistic outputsRegression MSEAverage squared errorRMSERoot of MSE — easier to interpretMAEAverage absolute errorR-squaredHow well predictions explain variance

Beyond standard models: Metrics by domain

When you move beyond classification and regression, evaluation needs to match the complexity of your model’s task. In language, vision or recommendation systems, you’ll need metrics tailored to how those models behave in the real world. Select the tabs to learn more. NLP NLP models generate or interpret human language, so we need metrics that compare how close their output is to something a human might write.BLEU (bilingual evaluation understudy):

• Compares the overlap of words and phrases between the model’s output and a human-written reference.
• Think of it like grading a student’s translation — did they choose similar words and structure?
• High BLEU means the translation is close to the original.
• Used in machine translation, chatbot responses.ROUGE (recall-oriented understudy for gisting evaluation):
• Measures how much of the important content from the reference text is present in the model’s summary.
• Focuses on recall — how much key information did the model keep?
• High ROUGE means the summary captures core ideas.
• Used in summarisation, headline generation.Example: You build a model to summarise customer support tickets. If BLEU is low but ROUGE is high, your summary may not use the same words but still captures the right ideas.Computer vision Vision models predict where and what something is in an image. Metrics measure how well the predicted locations and classes match reality.IoU (intersection over union):
• Measures how much the predicted box overlaps with the true box for an object.
• Imagine two rectangles — how well do they line up?
• Closer to 1 means a well-localised object.
• Used in object detection, facial recognition.mAP (mean average precision):
• Looks at precision across multiple thresholds — does the model consistently detect the right object classes with confidence?
• It balances how many correct detections you made and how confident the model was.
• High mAP equals better precision across different object types.
• Used in real-time detection systems (e.g. drones, self-driving cars).Example: A smart traffic camera needs high IoU for accurate vehicle boxes and high mAP to reliably detect trucks, bikes and pedestrians.Recommendation systems Recommendation models rank items by what they think a user will like. These metrics check both what was recommended and where it appeared in the list.NDCG (normalised discounted cumulative gain):
• Evaluates how well the model ranks relevant items near the top of the list.
• Rewards the model for showing useful stuff early — because users usually don’t scroll forever.
• Used in streaming platforms, shopping feeds.Recall@K:
• Measures if relevant items appeared anywhere in the top K suggestions.
• It’s like asking: 'Did the top 10 suggestions include anything I wanted?'
• Used in product recommendations, news feeds.Example: If your e-commerce model has high Recall@10 but low NDCG, users might see relevant items — but buried halfway down the list.

Establishing baselines and thresholds

Once your model is live, it’s not enough to track metrics — you need to know what 'good' looks like. That starts with setting baselines, alert thresholds and clear performance targets. Let’s break it down.

Establishing baselines

Start by creating reference points for each metric using past performance:

• Use historical results from your training, validation and test datasets to understand the natural range of metric variation.
• Analyse how your model performs over time in staging or early production environments.
• Consider building separate baselines by segment (e.g. different customer groups or geographies), especially for large or heterogeneous datasets.Why this matters: A baseline helps distinguish normal performance fluctuations from real issues. If your F1-score naturally fluctuates between 0.88 and 0.91, a drop to 0.89 isn’t necessarily a problem — but a drop to 0.80 might be.

Setting alert thresholds

Once you’ve established what normal looks like, define when alerts trigger:

• Use absolute thresholds (e.g, ‘trigger an alert if precision < 0.80’).
• Or relative thresholds (e.g. ‘alert if RMSE increases by more than 10% week-over-week’).
• You can also create tiered thresholds— soft alerts for small deviations and critical alerts for larger ones.
• Make sure these thresholds are tuned to the business impact.
• For example, in fraud detection, a 0.05 drop in recall could result in thousands of missed cases — set a tight threshold.
• In ad targeting, a small accuracy dip might be tolerable if CTRs remain stable — set a looser threshold.

SLOs and SLIs: Bringing structure to monitoring

Borrowed from DevOps, service level indicators (SLIs) andservice level objectives (SLOs) help formalise your model’s performance expectations:

• SLIs: The actual metrics you track, such as accuracy, latency or precision.
• SLOs: The targets you set for those indicators — for example, 'Maintain an F1-score ≥ 0.85 at least 95% of the time'.
• Together, these help ensure your model meets reliability goals just like any other software service.
• Example: In a loan approval model, your SLI might be approval accuracy, and your SLO could be '≥ 90% accuracy 99% of the time during business hours'.

Monitoring infrastructure and tools

Monitoring isn’t just about metrics — it’s about building the right infrastructure to track, surface and act on issues as they arise. Without the proper tools, even the best models can quietly drift into irrelevance.

Popular tools for monitoring ML models

To support reliable model performance in production, many teams rely on a combination of open-source frameworks and cloud-native services that make monitoring scalable and actionable. Let’s explore some of the most commonly used tools and how they fit into real-world ML workflows. Open-source tools:Evidently AI — purpose-built for ML monitoring. Tracks data drift, target drift and model performance with prebuilt visualisations.

• MLflow tracking— logs key metrics, model parameters and artifacts, helping with experiment tracking and reproducibility.
• Prometheus and Grafana— widely used for monitoring system metrics. Combine them to track time-series data like latency, throughput or performance degradation.
• Sagemaker Clarify— adds bias detection and model explainability capabilities to your monitoring stack.Cloud-native services:AWS Sagemaker model monitor— automates drift detection and quality monitoring for deployed models.
• Azure ML monitoring— tracks data quality, prediction drift and system health across endpoints.
• Google Cloud AI platform unified monitoring— integrates model performance, drift tracking, and system logs into one dashboard.

These tools help automate detection, reduce manual checks and create alerts tied to production behaviour.

Logging strategies: What to capture

Your monitoring is only as good as your logs. Here’s what you should consistently capture for every production model:

• Inputs and outputs: What went into the model and what it predicted.
• Predictions vs actuals: Crucial for calculating accuracy, error or drift over time (when ground truth is available).
• Feature statistics: Track summary stats like mean, standard deviation or outliers to spot unexpected shifts.
• Model metadata: Version number, hyperparameters, training date and other deployment context.
• Latency and error rates: Helps detect performance bottlenecks or infrastructure issues.

Data pipelines for monitoring

To make monitoring operationally effective, you need data pipelines that can move monitoring data from production to analysis tools:

• Timely: Set up streaming or batch jobs to capture metrics at regular intervals (e.g. hourly or daily).
• Consistent: Use the same preprocessing steps applied during model training to ensure valid comparisons.Secure and auditable: Especially critical in regulated sectors like health care or finance.

Tip

Schedule a daily batch job to compute PSI and accuracy for the latest batch of predictions. If drift exceeds a defined threshold, trigger an alert in your dashboard.

Designing effective monitoring dashboards

A well-designed dashboard gives you a real-time pulse on your model’s health and makes it easier to spot issues before they become critical. But not all dashboards are equally useful. Great dashboards are:

• Clear: Avoid clutter and show key metrics up front.
• Concise: Focus on meaningful, actionable insights.
• Interactive: Let users filter by time, model version or segments.

Key components to include:

• Time-series plots of key performance metrics (e.g. F1-score, MAE).
• Data drift indicators (e.g. PSI, KL divergence over time).
• Feature distribution comparisons (e.g. histograms for key inputs).
• Alert status and history.
• Model version and deployment details.

Real-world dashboard examples1.

Churn prediction dashboard (telecom use case): Used by a customer retention team to monitor an ML model predicting churn risk for mobile users.

• Key metrics: F1-score, recall (to prioritise catching at-risk customers).

• Drift signals: PSI for features like contract_type, monthly_charge and customer_age.

• Alerts: Triggered if recall drops below 0.82 or PSI exceeds 0.2 for key features.

• Actionable insight: If drift is detected in monthly_charge, the team investigates recent pricing changes. 2. Product recommendation dashboard (e-commerce use case): Used by the data science team to evaluate a collaborative filtering model for personalised product recommendations.

• Key metrics: NDCG and Recall@10 to assess ranking quality.

• Interaction filters: Dashboard can segment by region, device type or customer tenure.

• Feature comparisons: Shift in embeddings or changes in item_category distributions.

• Drift triggers: Alert if NDCG drops more than 10% compared to baseline week. 3. Disease risk prediction dashboard (health care use case): Used by clinicians and data engineers to monitor a model that predicts patient risk for developing Type 2 diabetes.

• Key metrics: AUC-ROC, precision, false negative rate (FN rate).

• Drift signals: PSI and KL divergence for features like blood_glucose, BMI and age_group.

• Compliance checks: Alerts triggered if FN rate rises above 5%, due to high clinical risk of missed diagnoses.

• Actionable insight: If drift is detected in lab test data, the team checks for changes in data entry protocols or lab device calibration. These examples show how dashboards must be tailored to the model’s purpose, business context and risk level.

Action item: ML monitoring checkpoint Test your understanding of performance metrics, thresholds and monitoring strategies for ML models in production.