Skills application solution

Worked example

In Progress

Continue building your understanding with the content below.

Compare your skills application output to the solution example below provided by Multiverse subject matter experts.

** Solution**## 1. Comparison setup** To ensure a fair comparison, both Model A (Random Forest) and Model B (Logistic Regression) were:

Evaluated on the same dataset using 10-fold cross-validation.
Compared using consistent metrics (AUC).
Assessed on the same test set for classification disagreements. This setup controls for data variance and allows valid statistical testing.** 2. Statistical testing**** Test 1: Paired t-test**** Why used:** The t-test is ideal here to compare the mean AUC of the two models across the same folds.** Formula:** t = (mean₁ − mean₂) / (std_diff / √n)** Result:**- Model A Mean AUC = 0.824
Model B Mean AUC = 0.858
t = −9.16
** p-value ≈ 7.39e-6**→ statistically significant difference.** Interpretation:** Model B’s higher AUC is statistically significant, not due to random variance.** Test 2: McNemar’s test**** Why used:** To assess if the models differ meaningfully in their classification errors on the test set.** Disagreement matrix:**** Model B correct**** Model B incorrect** ** Model A correct** 300b = 40** Model A incorrect** c = 20140** Formula:**

χ² = (|b − c| − 1)² / (b + c) χ² = (|40 − 20| − 1)² / (40 + 20) = 6.02 ** Interpretation:** The χ² value of 6.02 exceeds the critical value at 95% confidence → significant difference in error patterns. Model A and B do not make the same types of mistakes.

3. Interpretation of findings

Model B performs significantly better than Model A on AUC.
McNemar’s test confirms that their error patterns differ: Model A likely trades false positives for higher recall.
Model B is more calibrated, compact, and interpretable, while Model A favors sensitivity.** 4. Recommendation and refinement plan**

** Recommendation:** Deploy** Model B** , as it offers:

Statistically better AUC.
Simpler architecture suited to real-time clinical deployment.
Calibrated output and interpretability—critical for clinician trust.** Refinement Plan:**
Further tune Model B’s threshold to optimise recall while minimising clinician alert fatigue.
Use post-deployment monitoring to assess model drift and trigger recalibration if needed.** What this example does well**
Uses multiple statistical tests tailored to both performance scores and classification disagreement.
Grounds the recommendation in metrics aligned with clinical priorities (recall, trust, usability).
Offers practical next steps that balance model improvement with real-world constraints.

Always test whether improvements are real—not just visible.
Use different statistical tests for different comparison goals (scores vs. predictions).
Communicate results in stakeholder-friendly language, especially when decisions affect patient outcomes or customer experience.** Action item: reflection**

Compare your output to the solution example provided.

What did you do well?
Where could you improve?
How would you adjust your explanation for a stakeholder with limited technical background?