Skip to main content

Skills application

Instruction and application
Complete

Investigating bias in loan approvals​

Illustration

In this skills application, you will be working with financial data to identify if the model's performance with different subgroups is equitable.

You will start by following a tutorial in using Aequitas flow to assess the performance of a Logistic Regression model before trying it with other classification techniques.

Context​

When a bank approves a loan, it takes a lot of information into account, including the applicant's income, credit score and homeowner status. If a bank wants to use an algorithm to automate this process, it is important that it is tested to ensure it does not unfairly disadvantage any particular group.

You might think it is as simple as not taking information like race, gender or nationality into account, but these attributes can still be buried within other features.

In this exercise, we are going to take some historical data and investigate if the performance of a classification model is the same for all subgroups within a population.

We will demonstrate how to use this technique with Logistic Regression and then you will build other types of model to see if the added complexity reduces bias.

This will be used as the basis of a report for a bank, explaining how testing the false positive rate for different subgroups in the data will benefit their business.

Success criteria​

To successfully complete this skills application, you must submit a written report (approximately 1200-1800 words) addressing the following key areas:

  • Why identifying and mitigating model bias is important: Discuss the risk a biased model creates in approving loans and the consequences an unfair model will have for minority groups.
  • Discuss what can be done to mitigate bias: For example, how does lowering or raising a model's decision threshold affect both its overall accuracy and fairness?
  • Which model is the most suited for this data: Compare different classifiers, and with justification, explain which model should be selected in terms of accuracy and False Positive Rate (FPR) disparity. Completing this activity will “unlock” the solution example on the following page.

Instructions and materials​

Completing this activity will “unlock” the solution example on the following page.

Complete the tutorial​

Work through the Jupyter Notebook to learn how to use an Aequitas Flow for testing a classification model for fairness.

Find the best model​

Build different classification models and evaluate both their accuracy and FPR disparity for the two customer subgroups.

Stretch: optimise the models through hyperparameter tuning to see if that improves performance.

Write your report​

Write your report, justifying your decisions in terms of model performance.