Skills application
Skills application
In ProgressSkills application
Continue building your understanding with the content below.
Bias–variance–fairness audit
In this skills application, you will ** conduct a bias–variance–fairness audit** on an existing model. Your task is to assess the model’s behavior in terms of overfitting, underfitting, and fairness risks—then propose targeted adjustments to improve both performance and responsible deployment.
Context
Your team has inherited a ** gradient boosted decision tree model** used to predict whether a loan applicant will** default** . The model was trained on five years of historical data using the following features:
-
income
-
employment_status
-
credit_score
-
zip_code
-
loan_amount
-
age The model was originally evaluated using ** accuracy** and** AUC** . While performance appeared solid in development, a recent internal review raised concerns:
-
** Training accuracy is 92%, but validation accuracy has dropped to 78%.**
-
Performance is ** unstable across different data splits** , suggesting sensitivity to training samples
-
** False negative rates are disproportionately high for applicants under age 25.**
-
Applicants from ** rural zip codes** tend to receive lower predicted scores than similar urban applicants. You are part of the audit team responsible for identifying performance and fairness risks, and recommending adjustments that improve generalisation and promote more equitable outcomes.** Success criteria**
To successfully complete the skills application, you must:
- Diagnose the model’s performance issues in terms of bias and variance.
- Identify fairness concerns in the model’s features or prediction outcomes.
- Propose one or more strategies to improve both model generalisation and fairness, using techniques discussed in this unit (e.g., regularisation, ensemble methods, expanding training data).
- Justify your choices using the language of model complexity, generalisation, and fairness tradeoffs. Completing this activity will “unlock” the solution example on the following page.
Instructions and materials
Use the form below to respond to the following prompts:
Diagnosis
What are the key signs of high bias, high variance, or both in the model’s behaviour?
Fairness analysis
Which aspects of the model’s inputs or predictions raise fairness concerns? Who might be negatively impacted, and how?
Performance evaluation
Based on your diagnosis, what does the model’s behavior suggest about its ability to generalise to unseen data? How do bias, variance, or fairness risks affect the model’s reliability or stability in real-world use?
Recommended adjustments
Based on the issues you identified, propose one or more strategies to improve the model. Your response should include:
- At least one strategy to address ** high bias** ,** high variance** , or both—depending on your diagnosis.
- At least one strategy to mitigate any ** fairness risks** you identified.
- Explain why each proposed adjustment is appropriate, using concepts such as ** model complexity** ,** generalisation** , and** tradeoffs** . Once you complete and submit your responses, the ** solution example will unlock** .
Go deeper
After completing the activity, consider these questions:
- How do your proposed adjustments balance model performance, generalisation, and fairness?
- What are the risks if a model performs well overall but fails to generalise for specific groups?
- What tradeoffs did you consider when selecting your recommended strategy—for performance, generalisation, or fairness?
What are the key signs of high bias, high variance, or both in the model’s behavior?
Which aspects of the model’s inputs or predictions raise fairness concerns? Who might be negatively impacted, and how?
Based on your diagnosis, what does the model’s behaviour suggest about its ability to generalise to unseen data? How do bias, variance, or fairness risks affect the model’s reliability or stability in real-world use?
Based on the issues you identified, propose one or more strategies to improve the model. Your response should include:
At least one strategy to address high bias, high variance, or both—depending on your diagnosis. At least one strategy to mitigate any fairness risks you identified.
Explain why each proposed adjustment is appropriate, using concepts such as model complexity, generalisation, and tradeoffs.