Skip to main content

Skills application solution

Worked example
In Progress

Skills application solution

Continue building your understanding with the content below.

Skills application solution illustration

Skills application solution​

Compare your skills application output to the solution example below provided by Multiverse subject matter experts.

** Solution** ** 1. Diagnosis** The model shows signs of high variance. While training accuracy is high (92%), validation accuracy drops to 78%, indicating overfitting. The model also performs inconsistently across different data splits, reinforcing that it may be too sensitive to training noise. There is no strong evidence of high bias (e.g., underfitting or consistently poor predictions across all cases).** 2. Fairness analysis** Two fairness concerns emerge from the scenario:

  • Age-based disparity: The model has disproportionately high false negatives for applicants under 25. This suggests the model may not generalise well to younger individuals—possibly due to underrepresentation in the training data or overreliance on features that correlate with age.

  • Zip code sensitivity: Applicants from rural zip codes consistently receive lower predicted scores than urban applicants with similar financial profiles. This suggests zip code may be acting as a proxy for socio-economic status or regional bias, potentially disadvantaging certain groups unfairly.** 3. Performance evaluation** The model’s poor validation performance and instability across data splits suggest that it does not generalise well. Its behavior points to a model that is overly complex for the dataset and prone to overfitting. At the same time, fairness risks reveal that generalisation is uneven—some subgroups receive poorer performance, which undermines both reliability and trustworthiness. These issues could lead to poor user outcomes, reduced business value, or reputational risk if left unaddressed.** 4. Recommended adjustments** Based on the above diagnosis, the following strategies are appropriate:

  • ** To reduce variance**: Apply** L2 regularisation** to constrain model complexity. This helps reduce the model’s tendency to fit training noise and improves generalisation across new samples.

  • ** To support generalisation**: Increase training data diversity—specifically by including more data points from younger applicants and rural areas. This reduces the chance of overfitting to overrepresented patterns and improves consistency across subgroups.

  • ** To address fairness risks**:** Reassess feature importance** to detect overreliance on zip code or other correlated proxy features.

  • ** Use ensemble methods**(e.g., bagging) to smooth prediction variability and reduce group-level instability.

  • ** Consider transforming or removing zip code** if it contributes significantly to unfair outcomes and adds minimal performance value. Each of these strategies supports improved generalisation while also helping the model behave more equitably across different user groups—an essential consideration for real-world deployment.** What this example does well**

  • ** Connects model behaviour to variance and generalisation issues** , using evidence from training vs. validation performance and stability across splits.

  • ** Identifies fairness risks** clearly and ties them to specific features and prediction outcomes.

  • ** Applies targeted strategies**(regularisation, more data, ensemble methods) with strong justification tied to tradeoffs and real-world impact.

Tips for applying this skill in your role.​

  • Always examine ** performance across subgroups** , not just overall metrics—this reveals where generalisation breaks down.
  • When using sensitive or proxy features (like zip code or age), consider the fairness impact, not just predictive power.
  • Strategies like regularisation or data expansion can help you solve more than one problem at once—balancing ** performance** ,** stability** , and** fairness** .

Action item: Reflection​

  • How do your proposed adjustments balance performance, generalisation, and fairness?
  • What are the risks of ignoring group-level disparities, even if your model performs well overall?
  • What tradeoffs did you consider when selecting your strategies?