Skip to main content
Skills application
In Progress

Skills application

Continue building your understanding with the content below.

Skills application illustration

Statistical evaluation​

In this skills application, you’ll apply statistical evaluation methods to compare two machine learning models and propose a refinement strategy grounded in performance evidence.

This will help you practice selecting appropriate metrics, applying statistical tests, and justifying business-relevant decisions.

Context​

You’re working as an ML engineer at a health tech company. Your team is developing a model to predict hospital readmissions within 30 days of discharge. You’ve been asked to compare two model candidates:

✨

Model A

A high-recall random forest model tuned to maximise sensitivity.

✨

Model B

A compact, calibrated logistic regression model optimised for interpretability and real-time inference.

Both models have been evaluated using 10-fold cross-validation and report similar AUC scores.** Fold** ** Model A AUC**** Model B AUC** 10.850.8820.820.8730.800.8440.830.8650.810.8360.840.8870.850.8780.790.8490.830.85100.820.86Additionally, you have the following** disagreement table** from both models’ predictions on a test set:** Model B correct** ** Model B incorrect**** Model A correct** 30040** Model A incorrect** 20140

However, leadership needs a recommendation backed by evidence—one that clearly explains the trade-offs in false positives, false negatives, model transparency, and deployment complexity, especially given limited clinician time and the high cost of missed readmissions.** Success criteria**

To successfully complete the skills application, you must:

  • Use the provided data to compare the two models fairly.
  • Apply ** two statistical tests** .
  • Clearly ** interpret the results** and their implications.
  • Propose a ** recommendation and refinement plan** grounded in the evidence. Completing this activity will “unlock” the solution example on the following page.

Instructions and materials​

Use the prompts below to structure your analysis. These steps will guide you through comparing models and developing a refinement strategy.

Please answer each question in the form provided. You may reference statistical formulas or diagrams if helpful.

Comparison setup​

Briefly describe how you ensured a fair comparison between Model A and Model B (e.g., same dataset splits, consistent metrics, cross-validation).

Statistical testing​

Apply the following statistical tests:

  • Paired t-test

  • McNemar’s test For each:

  • Explain why you selected it.

  • Share a brief interpretation of the result.

Interpretation of findings​

Summarise what the test results tell you. Do the models perform similarly overall? Where do they differ? What matters most for this use case?

Recommendation and refinement plan​

Which model do you recommend, and why?

If neither model is perfect, what further refinement steps would you suggest (e.g., calibration, feature updates, threshold tuning)?

Go deeper​

After completing the activity, consider these questions:

  • How did statistical testing influence your confidence in selecting between a more complex model and a simpler, more interpretable one?
  • What trade-offs became clearer as you evaluated precision, recall, and model deployment feasibility?
  • How would you explain your recommendation to clinical or operational stakeholders who may not be familiar with machine learning?
Briefly describe how you ensured a fair comparison between Model A and Model B (e.g., same dataset splits, consistent metrics, cross-validation).
0 words
Apply the following statistical tests: • Paired t-test • McNemar's testFor each: • Explain why you selected it. • Share a brief interpretation of the result.
0 words
Summarise what the test results tell you. Do the models perform similarly overall? Where do they differ? What matters most for this use case?
0 words
Which model do you recommend, and why? If neither model is perfect, what further refinement steps would you suggest (e.g., calibration, feature updates, threshold tuning)?
0 words
In progress