Knowledge check

Instruction and application

In Progress

What is the primary benefit of calculating the PSI when monitoring ML models in production?

A) It compares predicted vs. actual labels to measure classification accuracy.B) It quantifies changes in feature distributions between training and production data.C) It detects changes in model hyperparameters during retraining cycles.D) It measures the variance of prediction confidence scores across datasets.

Which scenario best illustrates concept drift in a production ML model?

A) A change in the distribution of customer age groups using the platformB) The same user behaviours now correlate with different churn outcomes due to a new pricing model.C) A misalignment in feature column names between data pipelinesD) Seasonal increases in website traffic affecting transaction volume

Why is it important to correlate drift indicators with performance metrics in ML monitoring?

A) Because any statistical drift automatically means the model will failB) To prioritise alerts based on business impact, distinguishing between harmful and harmless driftC) Because drift metrics provide insights into data pipeline latencyD) To ensure the training dataset remains unchanged after deployment

What visualisation is most appropriate to track feature distribution shifts in a monitoring dashboard?

A) Confusion matrixB) Histogram comparison between training and production dataC) Precision-recall curveD) Heatmap of model parameters

Which statement describes a proactive maintenance approach for ML models?

A) Retraining a model only after performance drops significantlyB) Scheduling periodic model validations and testing new versions in shadow deploymentsC) Deploying untested models to production to monitor real-time user feedbackD) Ignoring data drift alerts unless business metrics decline

Which automated testing method allows evaluation of a new model version without affecting live predictions?

A) Shadow deploymentB) A/B testingC) Canary deploymentD) Integration testing

What is the primary purpose of setting alert thresholds on performance metrics in a monitoring dashboard?

A) To automate the retraining of models after every predictionB) To detect when metrics deviate from acceptable ranges and require human interventionC) To compare current model accuracy with previous model versionsD) To ensure model parameters are updated in real-time

Which of the following best describes label shift in a deployed ML system?

A) Changes in input feature distributions while the target distribution remains stableB) Shifts in the distribution of the target variable itself (e.g. churn rate increases)C) The introduction of new categories in a categorical input featureD) Model hyperparameters shifting during training

Why is automated regression testing important before deploying new model versions?

A) It reduces model inference time in production environments.B) It ensures that performance hasn't degraded on historical edge cases.C) It automatically retrains the model on new production data.D) It generates alerts for data pipeline failures.

Which of the following best describes a canary deployment in ML model operations?

A) Testing new model components in isolated development environmentsB) Rolling out a new model version to a small percentage of users to monitor performance before full deploymentC) Running a model in parallel with live data but not using its predictionsD) Replacing the production model entirely with a new version in a single deployment

Evaluate your understanding of this unit by completing the

Knowledge check.

Action item: Knowledge check

In progress