Skills application
Skills application
In ProgressSkills application
Continue building your understanding with the content below.
Statistical evaluation
In this skills application, you’ll apply statistical evaluation methods to compare two machine learning models and propose a refinement strategy grounded in performance evidence.
This will help you practice selecting appropriate metrics, applying statistical tests, and justifying business-relevant decisions.
Context
You’re working as an ML engineer at a health tech company. Your team is developing a model to predict hospital readmissions within 30 days of discharge. You’ve been asked to compare two model candidates:
- ** Model A**: A high-recall random forest model tuned to maximise sensitivity.
- ** Model B**: A compact, calibrated logistic regression model optimised for interpretability and real-time inference. Both models have been evaluated using 10-fold cross-validation and report similar AUC scores.** Fold** ** Model A AUC**** Model B AUC** 10.850.8820.820.8730.800.8440.830.8650.810.8360.840.8870.850.8780.790.8490.830.85100.820.86Additionally, you have the following** disagreement table** from both models’ predictions on a test set:** Model B correct** ** Model B incorrect**** Model A correct** 30040** Model A incorrect** 20140
However, leadership needs a recommendation backed by evidence—one that clearly explains the trade-offs in false positives, false negatives, model transparency, and deployment complexity, especially given limited clinician time and the high cost of missed readmissions.** Success criteria**
To successfully complete the skills application, you must:
- Use the provided data to compare the two models fairly.
- Apply ** two statistical tests** .
- Clearly ** interpret the results** and their implications.
- Propose a ** recommendation and refinement plan** grounded in the evidence. Completing this activity will “unlock” the solution example on the following page.
Instructions and materials
Use the prompts below to structure your analysis. These steps will guide you through comparing models and developing a refinement strategy.
Please answer each question in the form provided. You may reference statistical formulas or diagrams if helpful.
Comparison setup
Briefly describe how you ensured a fair comparison between Model A and Model B (e.g., same dataset splits, consistent metrics, cross-validation).
Statistical testing
Apply the following statistical tests:
-
Paired t-test
-
McNemar’s test For each:
-
Explain why you selected it.
-
Share a brief interpretation of the result.
Interpretation of findings
Summarise what the test results tell you. Do the models perform similarly overall? Where do they differ? What matters most for this use case?
Recommendation and refinement plan
Which model do you recommend, and why?
If neither model is perfect, what further refinement steps would you suggest (e.g., calibration, feature updates, threshold tuning)?
Go deeper
After completing the activity, consider these questions:
- How did statistical testing influence your confidence in selecting between a more complex model and a simpler, more interpretable one?
- What trade-offs became clearer as you evaluated precision, recall, and model deployment feasibility?
- How would you explain your recommendation to clinical or operational stakeholders who may not be familiar with machine learning?
Paired t-test McNemar’s test
For each:
Explain why you selected it.
Share a brief interpretation of the result.
If neither model is perfect, what further refinement steps would you suggest (e.g., calibration, feature updates, threshold tuning)?