Fairness and Ethical Bias
Instruction and application
CompleteBias and variance are important concepts when considering model performance, they describe how accurate and consistent the model is when evaluated on known data.
Before we look at strategies to make our models more fair, we need to understand what these concepts are and where they come from.
Defining bias and variance
Let’s start by defining what bias and variance are in the context of machine learning models.
What are bias and variance?
Bias is the error introduced by approximating a real-world problem (which may be complex) by a simplified model. A model with high bias makes strong assumptions about the underlying data, often leading to underfitting.
An underfit model fails to capture the important patterns in the training data and as a result performs poorly on both the training data and unseen data.
For example, imagine trying to fit a linear regression model to a dataset which has a clearly curved pattern. The model will be too simple and therefore result in high bias.
Mathematically, we can think of bias as the expected difference between the predictions of our model and the true values of the dependent variable. A high bias implies that on average, our model’s predictions are far from the actual values.
For example, a model predicting house prices might show high bias if factors such as location, number of bedrooms or age of property are ignored.
Variance, on the other hand, is the sensitivity of the model’s performance to small fluctuations in the training data. A model with high variance learns the noise in the training data along with the underlying pattern, resulting in overfitting.
An overfit model performs very well on the training data but poorly on new, unseen data because it has memorised the specifics of the training set, including its random noise.
For example, imagine that we use a complex model that accurately captures the curved trend in a dataset. While it passes through most training points, it might take wild swings between them and is unlikely to generalise well to new data.
Mathematically, variance can be understood as the extent a model’s prediction will change if trained on different subsets of the data. High variance indicates that the model’s predictions are highly dependent on the specific training data used.
For example, a very complex neural network with many layers and parameters trained on a small dataset of customer reviews to predict sentiment might show high variance. This is because the network might memorise specific wording and nuances in the reviews, including relevant noise.
Bias-variance tradeoff
High bias results in an underfitting model, while high variance results in an overfitting model.
Can you see how these two problems are the inverse of each other?
- To combat bias, we need to build more complex models- which can result in higher variance.
- To combat variance, we need to use simpler models- which can result in higher bias. The goal when building a ML model is to find a bias-variance tradeoff that results in a model that generalises well to unseen data. This often involves finding a “sweet spot” of model complexity that balances the need to capture the underlying pattern (low bias) with the need to avoid learning the noise (low variance).
Model selection, hyperparameter tuning, regularisation and using larger training sets are useful techniques in managing this tradeoff.
A method for tracking this is to build models of varying complexity and plot the bias and variance for each (both recorded as error). If you plot the total error of each model (bias + variance), you should get a “U” shape. The minimum point of this curve is the optimal model complexity.
Bias in ML vs statistical bias
Bias is a term that is used in multiple contexts, so it is important that you understand how it is being used here.
- Bias in machine learning relates to the simplifying of the learning algorithm.
- Bias in statistics refers to systematic errors in data collection, sampling or measurement. While both can negatively impact the performance and fairness of ML models, they originate from different sources.
Statistical bias in the training data can directly contribute to algorithmic bias in the resulting model. For example, if a facial recognition dataset disproportionately contains images of one demographic group, the model trained on this data will be biased as it is too simple for the complexity of the data population.
Sources of bias in the ML lifecycle
So where does bias come from when building models?
It can be introduced at various stages, from the initial data collection to the final evaluation of the model.
Let’s take a closer look so you are better prepared to spot sources of bias when building your models.
Data collection
Skewed datasets and underrepresentationSkewed datasets occur when the distribution of important features or target variables is not representative of the population the model is intended to serve.
Underrepresentation means certain subgroups or categories are significantly less present in the data than their actual proportion.
A model trained on underrepresentative data may not learn the patterns and nuances relevant to the missing groups. This can lead to poorer performance or even discriminatory predictions for these groups.
The image above shows that the proportion of races used in the training data does not match the distribution in the overall population.
Historical biasesData collected often reflects existing societal inequalities and biases.
For example, historical hiring data might show a gender imbalance in certain roles due to past discriminatory practices.
If a model is trained on such data, it can learn and perpetuate these historical biases, leading to outcomes that reinforce past unfairness.
The image above shows that over time, male proportions have been going down - if historical data is used, it will exhibit bias as the male members will be overrepresented.
Patterns of missing dataMissing values are a common issue in data.
However, the pattern of missingness can be non-random and correlated with certain attributes or groups.
If missing data is handled improperly or if the missingness itself carries information related to a sensitive attribute or outcome, it can introduce bias. For example, if certain demographic groups are less likely to provide certain information, and this missingness is ignored or imputed without careful consideration, it can skew the model's learning.
Class imbalanceThis occurs in classification problems when one or more classes have significantly fewer instances in the training data compared to the majority class(es).
Standard classification algorithms often aim to optimise overall accuracy. In imbalanced datasets, a model can achieve high accuracy by simply predicting the majority class most of the time, without effectively learning to identify the minority class.
This can lead to poor performance and unfair outcomes for the minority class, which might represent a vulnerable or historically disadvantaged group (e.g., in fraud detection, disease diagnosis). The model becomes biased towards the majority class, effectively ignoring the minority class.
The image above shows a confusion matrix of a loan approval classification model, due to a class imbalance, the model has low precision, tending to predict “approved” when it shouldn’t.
Data preprocessing
The steps taken to clean, transform, and prepare data for modelling can inadvertently introduce or amplify biases present in the original data.
ImputationThis is when we replace missing values in our data, commonly using measures of central tendency like the mean, median or mode.
If imputation is done without considering group differences, it can mask or distort underlying biases. For example, imputing missing income values with the overall mean might not accurately reflect the income distribution of different demographic groups, potentially leading the model to learn biased relationships.
The image compares the distribution of incomes before and after missing values are imputed with the mean. Doing so significantly affects the distributions, which can lead to biased outcomes in the model.
Feature engineeringWhen we create new features from existing ones to make our data more usable (e.g. dividing raw salary figures into discrete bands).
If new features are created based on biased assumptions or if they inadvertently encode sensitive information in a way that is not obvious, they can introduce or amplify bias.
For example, creating a "creditworthiness score" based on historical data that reflects past discriminatory lending practices will embed that bias into the new feature.
The image shows the result of adjusting income from data to account for inflation, using a historical formula. The result is that “male” incomes are made to seem higher, while others stay the same.
Data transformationTechniques like scaling and normalisation are used to bring features so they are all on a similar range.
If scaling or normalisation is applied globally to the entire dataset without considering subgroup differences, it can distort the relative importance or variance of features within different groups, potentially impacting model performance unfairly across these groups.
For example, if one group has a naturally smaller range for a certain feature, global scaling might compress their data more significantly.
The image compares the distribution of credit scores for White vs Black customers after scaling the scores. As both show the same shape, we can be more confident the scaling process hasn’t introduced bias.
**Handling missing data (beyond imputation)**This includes strategies like removing rows or columns with missing data or using algorithms that can handle missing values.
If the decision to remove data is correlated with a sensitive attribute, it can lead to a biased training set. Similarly, if an algorithm's handling of missing values implicitly favors certain patterns of completeness over others, it can introduce bias.
The image above compares the average income per race before and after missing records from the employment status have been removed. Notice that for Black customers, the average income drops as a result of this transformation.
Model selection
The choice of machine learning algorithm itself can introduce bias, as different algorithms have inherent inductive biases – assumptions they make about the data.
Choosing algorithms that favour certain groups
Some algorithms might inherently perform better on data distributions that are more common for certain demographic groups or might be more sensitive to features that are correlated with these groups.
For example, a very complex model might overfit to the patterns of the majority group if the minority group has less data.
If the algorithm's inductive bias aligns better with the characteristics of one group over another, it can lead to disparities in performance and potentially unfair predictions.
For instance, if a linear model is chosen for a task where the relationship is non-linear for a specific subgroup, that subgroup will be disadvantaged.
This image shows a scatter chart that demonstrates the relationship between the age of someone’s checking account and their credit score. Male customers are disproportionally represented, so the overall trend seems linear. However, when you look at the non-male customers, the relationship appears to be quadratic.
Evaluation
The metrics used to evaluate the performance of a machine learning model play a crucial role in determining whether it is fair. Using metrics that only focus on overall performance can mask significant disparities in performance across different subgroups.
Using metrics that do not adequately capture fairnessRelying solely on overall accuracy, precision, or recall can be misleading when dealing with imbalanced datasets or when fairness is a concern. A model might achieve high overall accuracy by performing well on the majority group while performing poorly on a minority group.
If fairness is not explicitly measured and considered during evaluation, biased models might be deployed because they appear to perform well according to the chosen (but inadequate) metrics.
Two classification models were used to predict if someone’s credit application would be approved, the bar chart shows the difference in accuracy scores for both when evaluating the performance of specifically White and Black customers.
Impact of bias and variance
While bias and variance are technical concepts related to model performance, their presence in machine learning systems can have significant consequences beyond mere inaccuracies.
Let’s explore the ethical, legal, and business ramifications of deploying biased ML models.
Examples of biased ML systems and their consequences
Bias in ML systems isn’t some theoretical concept, but a real problem that you will need to deal with when developing models.
We'll start by looking at some examples where bias was not addressed that led to harm and unfairness.
**Recidivism prediction (COMPAS)**The Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) algorithm, used in the US to assess the likelihood of a defendant re-offending, has been shown to exhibit racial bias.
ProPublica's investigation revealed that COMPAS incorrectly labeled Black defendants as "high risk" at nearly twice the rate of white defendants, while white defendants were mislabeled as "low risk" more often than Black defendants.
This bias led to harsher sentencing, denial of parole, and other detrimental impacts on individuals' lives, disproportionately affecting minority communities and perpetuating systemic inequalities within the criminal justice system.
Gender bias in hiring algorithmsAmazon reportedly scrapped an AI-powered recruiting tool because it showed significant bias against women. The system was trained on historical hiring data that predominantly featured male applicants for technical roles.
As a result, the AI learned to downgrade resumes containing words commonly associated with women's resumes (e.g., "women's" colleges) and favored male candidates even when equally qualified.
This is an example of how model bias can reinforce gender imbalances in the workplace, limit opportunities for qualified individuals, and hinder diversity and inclusion efforts.
Image recognition and taggingEarly versions of image recognition software have been known to exhibit racial bias, misclassifying or failing to recognise individuals with darker skin tones.
In one notorious example, Google Photos in 2015 tagged Black individuals as "gorillas."
Such errors are not only offensive and dehumanising but also highlight the lack of diverse representation in training data and the potential for technology to perpetuate harmful stereotypes.
Speech recognitionThese have been shown to perform less accurately for individuals with accents or non-standard dialects, which can disproportionately affect certain regional or ethnic groups.
This can lead to frustration and exclusion in accessing voice-activated technologies and services, further marginalising those whose speech patterns are not well-represented in the training data.
Ethical and legal ramifications
Machine learning models are rarely built for fun, but to achieve some business goal for an organisation.
Therefore, there are often significant ethical and legal concerns that arise if the model we deploy is biased.
- Potential discrimination: biased algorithms can lead to direct or indirect discrimination against individuals or groups based on protected characteristics such as race, ethnicity, gender, religion, sexual orientation, disability, or age. This can violate principles of fairness and equality.
- Violation of anti-discrimination laws: the 2010 Equality Act prohibits discrimination based on protected characteristics. If biased ML systems are used in decision-making processes (e.g., hiring, lending, housing), they could be deemed in violation of this law, even if the bias was unintentional.
- Lack of transparency and explainability: complex ML models, especially those with high bias or variance learned from biased data, can be "black boxes," making it difficult to understand why certain decisions are made. This lack of transparency can hinder accountability and make it challenging to identify and rectify discriminatory outcomes.
- Reinforcement of societal biases: if left unchecked, biased ML systems can perpetuate and even amplify existing societal inequalities, creating a feedback loop where biased outputs reinforce biased data collection and further entrench unfairness.
- Emerging regulations: regulatory bodies worldwide are increasingly focusing on the ethical implications of AI and considering or implementing regulations to address bias and ensure fairness in algorithmic decision-making (e.g. the EU AI act). Failure to address bias proactively can lead to legal challenges and non-compliance. Not resolving biases in your model outputs can lead to significant consequences for your organisation. It is important therefore that you identify any cases by thoroughly testing the model in different situations and work to resolve them if they happen.
Business implications
Apart from ethical and legal implications, bias in ML systems can result in negative consequences for our organisation's business goals.
ImplicationConsequences Reputational damagePublic exposure of a biased AI system can severely damage your organisation's reputation, leading to loss of customer trust, negative media coverage, and potentially losing business to rival organisations.Compliance issues and legal penaltiesAs regulations around AI fairness evolve, organisations deploying biased systems face the risk of legal challenges, fines, and other penalties for non-compliance with anti-discrimination laws and AI governance frameworks.Financial risks-Loss of customers: if a system unfairly disadvantages a significant demographic, the business may lose those customers to competitors.
- Inefficient resource allocation: biased models might lead to inefficient targeting of resources or misidentification of opportunities, resulting in financial losses.
- Increased operational costs: addressing bias after deployment can be costly, requiring significant resources for auditing, retraining, and potentially redesigning the system.
- Liability: businesses could face lawsuits and financial liabilities if their biased AI systems cause harm or financial loss to individuals.Reduced innovation and market opportunitiesFocusing on biased data and models can limit an organisation's ability to innovate and cater to diverse markets effectively. Addressing bias can lead to the development of more robust and inclusive products and services, opening up new opportunities.Difficulty in attracting and retaining talentOrganisations known for deploying unethical or biased AI may struggle to attract and retain employees who value fairness and social responsibility.### Key points
In this section, you've seen that deploying a biased model can result in significant consequences for your organisation. Remember, it's your responsibility to ensure your models are fair and mitigate/resolve bias if uncovered.
Action item: Pause and think
Before moving on, try applying what you've learned by answering the questions below.