Using explainable AI (XAI) to identify and mitigate bias

Instruction and application

Complete

Unlocking trustworthy models

Explainable AI (XAI) is a field dedicated to developing techniques that make the decisions and internal workings of AI models transparent and understandable to humans.

In the context of bias, XAI provides valuable tools to inspect model behaviour, understand why certain predictions are made, and ultimately inform strategies to reduce unfairness.

What is XAI?

XAI refers to a suite of methods and techniques that aim to make AI systems more interpretable and transparent to human users.

Unlike "black box" models whose decision-making processes are opaque, XAI seeks to provide insights into how and why a model arrives at a particular prediction. This involves generating explanations that humans can understand, such as feature importance scores, visual explanations of image classifications, or reasons for a specific loan application denial.

The goal of XAI is to build trust in AI systems, facilitate debugging and improvement, ensure accountability, and address ethical concerns, including the identification and mitigation of bias.

Purpose of XAI

XAI serves several important purposes in machine learning, particularly concerning bias:

• Understanding model behaviour: it allows us to dissect how different features influence a model's output, revealing potentially problematic dependencies on sensitive attributes or their proxies.
• Identifying bias: by making the model's reasoning transparent, XAI techniques can highlight instances where the model's predictions are unfairly skewed towards or against specific groups.
• Debugging and improvement: understanding the sources of bias enables data scientists and engineers to make informed decisions about data pre-processing, feature engineering, and model selection to reduce unfairness.
• Building trust and accountability: when users understand how a model works and why it makes certain decisions, they are more likely to trust it. In high-stakes applications, explainability is crucial for accountability and ensuring fairness.
• Regulatory compliance: in some regulated industries (e.g., finance, healthcare), there are increasing demands for transparency and explainability in automated decision-making processes to ensure fairness and prevent discrimination.

Common techniques

Several key XAI techniques are particularly useful for understanding and addressing bias:

**SHapely Additive exPlanations (SHAP)**SHAP is a game-theoretic approach to explain the output of any machine learning model. It calculates the Shapley values, which quantify the contribution of each feature to the prediction for a specific instance, compared to the average prediction across the dataset. Features with positive Shapley values increase the prediction, while those with negative values decrease it.When to use it:

SHAP is powerful for understanding the feature importance for individual predictions and for gaining a global understanding of feature impact across the entire dataset. It's particularly useful when you need to understand the contribution of each feature for a specific decision, such as why a particular loan application was denied.

Benefits:

• Individual and global interpretability: provides explanations for individual predictions and a global overview of feature importance.

• Consistent and theoretically sound: based on game theory, SHAP offers a consistent and theoretically grounded way to attribute feature importance.

• Handles feature dependencies: can account for dependencies between features.

• Visualisations: offers insightful visualisations like summary plots and dependence plots to understand feature effects.Risks:

• Computational cost: calculating exact Shapley values can be computationally expensive, especially for large datasets and complex models. Approximations are often used, which can introduce some level of error.

• Interpretation complexity: while the underlying theory is sound, interpreting Shapley values for highly complex models or interactions can still be challenging for non-experts.**Local Interpretable Model-agnostic Explanations (LIME)**LIME explains the predictions of any classifier or regressor by approximating it locally with an interpretable model (e.g., a linear model or decision tree) around the specific instance being predicted.

It perturbs the input data around the instance, gets predictions from the black-box model, and then learns a simple, interpretable model on this perturbed data to explain the local behavior.

When to use:

LIME is excellent for understanding why a model made a specific prediction for a single instance. It helps to build trust in individual decisions and can highlight if the model's reasoning for similar instances from different groups is consistent or biased.

Benefits:

• Model-agnostic: can be applied to any machine learning model.

• Local fidelity: provides a locally faithful explanation of the model's behavior around a specific prediction.

• Intuitive explanations: often generates explanations that are easy to understand, such as feature weights in a local linear model.

• Highlights local biases: can reveal if the model behaves differently for similar instances belonging to different protected groups.Risks:

• Instability: explanations for slightly different instances can sometimes vary, leading to instability.

• Choice of perturbation: the way the input data is perturbed can significantly influence the resulting explanation. Choosing appropriate perturbation methods can be challenging.

• Local approximation: the local explanation might not accurately reflect the global behavior of the model.**Partial Dependence Plots (PDPs)**Partial Dependence Plots (PDPs) visualise the marginal effect of one or two features on the predicted outcome of a machine learning model. They show how the average predicted response changes as the selected feature(s) vary, while averaging out the effects of all other features in the model.

PDPs help in understanding the overall relationship between a feature and the model's prediction, revealing whether the relationship is linear, monotonic, or more complex.

When to use:

PDPs are particularly useful for gaining a global understanding of how a specific feature influences the model's output on average. In the context of bias, you can use PDPs to examine the relationship between a sensitive feature (or a potential proxy) and the model's predictions.

An unexpected or discriminatory trend in the PDP for a sensitive feature might indicate bias. For example, a PDP showing that loan approval probability decreases more sharply with income for one gender compared to another could suggest a biased model.

Benefits

• Global understanding: provides a clear visualisation of the average effect of a feature on the model's prediction.

• Model-agnostic: can be applied to any trained machine learning model.

• Identifies non-linear relationships: effective at revealing complex, non-monotonic relationships between features and the target variable that simple linear analysis might miss.

• Highlights potential interactions (Two-way PDPs): can show how the effect of one feature on the prediction depends on the value of another feature.Risks:

• Assumption of independence: PDPs assume that the feature(s) being analysed are not correlated with other features. If strong correlations exist, the "average" effect shown in the PDP might not represent realistic scenarios, as it averages over impossible or unlikely combinations of feature values.

• Aggregation hides heterogeneity: by showing the average effect, PDPs can mask significant variations in how a feature affects individual instances. This is where ICE plots can provide additional insight.

• Limited to a few features: typically, PDPs are used to analyse the effect of one or two features at a time. Understanding complex interactions involving many features requires exploring multiple PDPs.**Individual Conditional Expectation plots (ICE)**Individual Conditional Expectation (ICE) plots display the dependence of the predicted outcome on a feature for each individual instance in the dataset.

For each instance, an ICE plot shows a line that represents how the model's prediction changes as the value of the chosen feature varies, while all other features remain constant at their original values for that instance.

When to use

ICE plots are valuable for understanding the heterogeneous effects of a feature across different instances. In the context of bias, you can plot ICE curves for a sensitive feature (or a proxy) and then color or group these curves based on another sensitive attribute (e.g., plotting ICE for income with lines colored by gender).

If the ICE curves for different groups show systematically different responses to the feature, it can indicate bias. For example, if increasing education level leads to a much steeper increase in salary prediction for one gender compared to another across many individuals, it could suggest gender bias in the model.

Benefits

• Reveals heterogeneous effects: shows how a feature impacts different individuals in potentially different ways, uncovering patterns hidden by the average effect in PDPs.

• Identifies subgroups with different sensitivities: by visualising individual curves, you can identify subgroups of the population for whom the model reacts differently to a particular feature.

• Complements PDPs: ICE plots can be used alongside PDPs to understand both the average and the individual-level effects of a feature.Risks

• Can be cluttered: for large datasets, ICE plots can become overcrowded with many individual lines, making it difficult to discern clear patterns. Aggregation or highlighting of representative curves might be necessary.

• Assumption of independence: similar to PDPs, ICE plots assume that changing one feature while holding others constant represents realistic scenarios for individual instances. If features are highly correlated, these hypothetical changes might be unlikely in the real world.

• Interpretation complexity: while showing individual effects is powerful, interpreting the patterns across a large number of ICE curves can still require careful analysis.

Bias detection with XAI

So how can we detect bias using XAI? Here are a few methods:

Feature importance analysisBy examining the importance scores assigned to different features by XAI methods (like SHAP or permutation importance), we can identify if sensitive attributes or their proxies are among the most influential features in the model's predictions. High importance scores for such features can be a red flag, indicating potential discriminatory behavior.When to use:

Feature importance analysis is a good initial step in bias detection after training a model. It provides a global overview of which features the model relies on most heavily. If you suspect bias related to a specific protected attribute, checking its importance (or the importance of its proxies) is a logical first step.

Benefits:

• Global overview: provides a high-level understanding of the model's reliance on different features.

• Easy to implement: many XAI libraries offer straightforward ways to calculate and visualise feature importance.

• Highlights potential bias sources: can quickly point to sensitive features or their proxies as significant drivers of predictions.Risks:

• Correlation vs. causation: high importance doesn't necessarily mean the feature causes bias; it might be correlated with other factors.

• Masking effects: if sensitive attributes are highly correlated with non-sensitive ones, their importance might be masked or distributed among the correlated features.

• Non-linear effects: simple feature importance might not fully capture complex, non-linear ways in which features interact to create bias.Example-based explanationsTechniques like LIME and SHAP provide explanations for individual predictions. By examining explanations for similar instances that belong to different protected groups but receive different outcomes, we can identify if the model's reasoning differs unfairly.

For example, if two loan applicants with similar financial profiles but different genders receive different loan approval decisions, comparing their LIME or SHAP explanations might reveal if gender played a significant role in the discrepancy.

When to use:

Example-based explanations are important for understanding local biases and identifying discriminatory behavior on a case-by-case basis. They are particularly useful when investigating specific instances of potential unfairness or when trying to understand why the model treats similar individuals differently.

Benefits:

• Local bias detection: directly highlights if the model uses sensitive attributes differently for similar individuals from different groups.

• Intuitive understanding: provides concrete examples of how the model's reasoning varies across groups.

• Facilitates debugging: can pinpoint specific instances and feature combinations where bias might be occurring.Risks:

• Scalability: examining explanations for a large number of individual instances can be time-consuming.

• Subjectivity: interpreting and comparing explanations across different instances might involve some level of subjectivity.

• Choosing representative examples: selecting the right examples to analyse is crucial for uncovering meaningful patterns of bias.Rule extractionRule extraction techniques aim to distill the complex logic of a black-box model into a set of human-understandable rules (e.g., "IF feature A > X AND feature B = Y THEN prediction = Z").

By examining these extracted rules, we can directly see if sensitive attributes are used as conditions in the rules or if the rules lead to disparate outcomes for different groups.

When to use:

Rule extraction is valuable when interpretability is paramount, and you want to understand the explicit decision-making logic of the model. If the extracted rules directly involve sensitive attributes or lead to demonstrably unfair outcomes for certain groups, it's a clear indication of bias.

Benefits:

• Highly interpretable: provides a direct and transparent view of the model's decision logic.Direct bias identification: makes it easy to see if sensitive attributes are explicitly used in decision rules.

• Facilitates auditing: extracted rules can be easily audited for fairness and compliance.Risks:

• Fidelity issues: extracting a perfectly faithful set of rules from a complex model can be challenging. The extracted rules might not fully capture the model's behavior in all scenarios.

• Scalability: for very complex models, the number of extracted rules can become very large and difficult to manage and interpret.

• Applicability: rule extraction is not always straightforward or effective for all types of machine learning models, especially highly non-linear ones.Counterfactual explanationsCounterfactual explanations identify the smallest change to an input instance that would change the model's prediction to a desired outcome. In the context of bias detection, we can generate counterfactuals for individuals from different protected groups who received different outcomes.

If the counterfactual changes required for individuals from one group involve changes to their protected attribute (or proxies), while similar individuals from another group could achieve the desired outcome with changes to non-sensitive features, this can indicate bias.

When to use:

Counterfactual explanations are useful for understanding the "what if" scenarios and identifying discriminatory decision boundaries. They can highlight if individuals from different groups are held to different standards or face unfair barriers to achieving a positive outcome.

Benefits:

• Highlights unfair barriers: can reveal if changes to protected attributes are necessary to achieve a desired outcome for one group but not another.

• Actionable insights: provides concrete suggestions on what changes an individual would need to make to receive a different outcome (though these might not always be feasible or fair).

• Identifies discriminatory decision boundaries: can show if the model's decision boundary unfairly separates different groups.Risks:

• Generation complexity: generating meaningful and realistic counterfactuals can be computationally challenging and require careful consideration of feature dependencies and constraints.

• Interpretability of changes: the changes suggested in a counterfactual might not always be easily interpretable or actionable in the real world.

• Ethical considerations: presenting counterfactuals that involve changing sensitive attributes can be ethically problematic.

Bias mitigation with XAI

Beyond detection, XAI can also inform and guide bias mitigation strategies:

Informed feature engineeringBy using XAI to understand which features contribute most to biased predictions, we can make more informed decisions about feature engineering.

This might involve creating new, less biased features, transforming existing features to reduce their correlation with sensitive attributes, or even removing highly influential but problematic features (with caution, considering potential loss of predictive power).

When to use:

Feature engineering informed by XAI is a pre-processing step that can be applied before or during model development. When XAI reveals that certain features are strong drivers of potentially biased outcomes, this approach can help create a more equitable feature space for the model to learn from.

Benefits:

• Addresses bias at the source: by modifying the input data, this approach aims to prevent the model from learning biased relationships in the first place.

• Can improve model fairness and performance: thoughtful feature engineering can sometimes lead to both fairer and more accurate models by focusing on truly predictive and less discriminatory signals.

• Provides justification for feature selection: XAI insights can provide a rationale for including, transforming, or excluding certain features based on their impact on fairness.Risks:

• Potential loss of predictive power: removing or significantly altering influential features might inadvertently reduce the model's overall accuracy.

• Complexity of identifying and engineering fairer features: determining which features to engineer and how to do so effectively can be a complex and iterative process.

• Risk of introducing new biases: poorly designed feature engineering could unintentionally introduce new forms of bias.**Adversarial debiasing (guided by XAI)**Adversarial debiasing involves training a model to make accurate predictions while simultaneously trying to prevent an adversary model from predicting the protected attribute based on the main model's learned representations.

XAI can enhance this process by providing insights into which features the main model relies on that are most informative about the protected attribute. This information can then be used to refine the adversarial training process or the architecture of the debiasing components.

When to use:

When standard adversarial debiasing is not achieving the desired level of fairness, XAI can provide a deeper understanding of the model's internal workings and guide improvements to the debiasing strategy.

Benefits:

• More targeted debiasing: XAI insights can help focus the adversarial training on the specific features or representations that are most strongly correlated with the protected attribute.

• Improved fairness outcomes: by more effectively targeting the sources of bias, XAI-guided adversarial debiasing might lead to more significant reductions in unfairness.

• Better understanding of debiasing effects: XAI can help analyse how adversarial training alters the model's representations and its reliance on different features.Risks:

• Increased complexity: integrating XAI into the adversarial debiasing pipeline can further increase the complexity of the training process.

• Potential for over-debiasing: aggressively removing all information about the protected attribute might lead to a significant loss of predictive accuracy if that attribute (or its correlates) contains some genuinely useful, non-discriminatory information.

• Interpretation challenges: understanding the interplay between the main model, the adversary, and the XAI insights can be challenging.**Fairness-aware training (with explainable constraints)**Fairness-aware training methods directly incorporate fairness constraints into the model's optimisation objective. XAI can play a role here by allowing us to define and understand these fairness constraints in an explainable way.

For example, instead of just optimising for a fairness metric, we might use XAI to understand which features contribute most to violations of that metric and then design constraints that specifically target those features or their impact on different groups.

When to use:

When you want to directly optimise for fairness during training and want to ensure that the fairness constraints are transparent and understandable. XAI can help in designing these constraints based on insights into the model's behavior and potential sources of bias.

Benefits:

• Explainable fairness enforcement: XAI can make the fairness constraints used during training more transparent and justifiable.

• Potentially more effective fairness optimisation: By understanding the drivers of unfairness, we can design more targeted and effective fairness constraints.

• Improved trust and accountability: explainable fairness constraints can increase trust in the fairness-aware training process.Risks:

• Complexity of defining explainable constraints: translating XAI insights into effective and interpretable fairness constraints can be challenging.

• Trade-offs with accuracy and other fairness metrics: Optimising for fairness based on specific XAI insights might still lead to trade-offs with predictive accuracy or other notions of fairness.

• Need for deep understanding of both XAI and fairness metrics: effectively using XAI to inform fairness-aware training requires a strong understanding of both domains.Explainable regularisationRegularisation techniques are used to prevent overfitting by adding penalties to the model's loss function based on the complexity of the model's parameters. Explainable regularisation would involve using XAI insights to design regularisation terms that specifically penalise model behavior that is indicative of bias.

For example, if XAI reveals that a sensitive attribute has a disproportionately large influence on predictions, a regularisation term could be added to reduce the weight or influence of that feature (or its proxies) during training. Similarly, regularisation could be designed to encourage the model to have more consistent feature importance or partial dependence patterns across different protected groups.

When to use:

Explainable regularisation is a form of in-processing bias mitigation that is most applicable during the model training phase. It's particularly useful when you have identified specific features or model behaviors that contribute to bias through XAI and want to directly discourage these patterns during the learning process.

Benefits:

• Directly addresses bias during training: by integrating bias penalties into the loss function, the model is actively discouraged from learning biased relationships.

• Potentially better accuracy-fairness trade-off: compared to post-processing, this approach can lead to a more fundamental reduction in bias without severely compromising model accuracy, as the model learns fairer representations from the outset.

• Transparency in fairness constraints: using XAI insights to design regularisation terms can make the fairness interventions more transparent and justifiable, as they are based on an understanding of how the model is behaving.Risks:

• Complexity of design and implementation: designing effective regularisation terms that target specific forms of bias identified by XAI can be challenging and require a deep understanding of both the model architecture and the XAI techniques used.

• Need for careful tuning: the strength of the regularisation terms needs to be carefully tuned to achieve the desired balance between bias reduction and model performance. Over-regularisation could harm accuracy, while insufficient regularisation might not effectively mitigate bias.

• Potential for unexpected consequences: modifying the loss function with custom regularisation terms could lead to unintended effects on other aspects of the model's behavior or fairness properties. Thorough evaluation across multiple fairness metrics is crucial.

Ongoing monitoring and governance

Achieving fairness in machine learning models is not a one-time task but an ongoing commitment that requires continuous attention and adaptation.

Ongoing monitoring and governance means the establishment of systematic processes, organisational policies, and accountability structures designed to ensure that deployed AI systems consistently adhere to fairness principles over time.

This involves proactively tracking model performance and fairness metrics in production, identifying and addressing any emerging biases or fairness drifts, and embedding fairness considerations into the organisation's AI development lifecycle and decision-making processes.

The goal is to move beyond initial bias mitigation efforts and create a sustainable framework that fosters accountability and ensures the long-term ethical and equitable operation of AI systems.

Here are some strategies your organisation can use:

StrategyDescriptionGetting started Establishing organisational policies for fairnessClear and comprehensive organisational policies are foundational for sustained fairness. These policies should articulate the organisation's commitment to fairness, define what fairness means in the specific context of their AI applications (potentially referencing specific fairness metrics or principles), outline acceptable levels of bias, and establish guidelines for the entire AI lifecycle, from data acquisition to model deployment and monitoring. These policies should be developed collaboratively, involving legal, ethical, and technical stakeholders, and should be regularly reviewed and updated to reflect evolving societal norms and regulatory landscapes.Implementing accountability structuresAccountability structures define roles, responsibilities, and reporting lines for ensuring fairness throughout the AI lifecycle. This includes assigning individuals or teams with specific responsibility for overseeing fairness assessments, implementing mitigation strategies, monitoring deployed models, and addressing fairness concerns. Clear accountability fosters ownership and ensures that there are mechanisms in place to address and rectify instances of bias. This might involve establishing ethics review boards, fairness working groups, or designated AI ethics officers.Establishing ongoing monitoring processesContinuous monitoring of deployed AI systems is crucial for detecting bias drift, performance degradation across different groups, and the emergence of new fairness issues over time. This involves regularly tracking relevant performance metrics (e.g., accuracy, precision, recall) and fairness metrics (e.g., demographic parity, equalised odds) disaggregated by protected attributes. Automated monitoring systems and dashboards can help visualise these metrics and trigger alerts when fairness thresholds are breached.Implementing governance and review mechanismsGovernance and review mechanisms provide a structured approach for overseeing the fairness of AI systems. This includes regular audits of deployed models, reviews of development processes and fairness assessments, and established procedures for addressing and resolving identified fairness issues. Governance structures might involve cross-functional committees responsible for reviewing AI deployments from a fairness perspective and ensuring adherence to organisational policies.### Key points

By establishing robust organisational policies, clear accountability structures, effective ongoing monitoring processes, and comprehensive governance mechanisms, our organisations can build a sustainable framework for ensuring the long-term fairness and ethical operation of their AI systems.

Action item: Pause and think

Before moving on, try applying what you've learned by answering the questions below.