Skip to main content

Model explainability and interpretability

Instruction and application
Complete

As a ML Engineer, you'll often face a critical trade-off: Should you choose a model that delivers superior performance but functions as a ‘black box’ or opt for a more transparent (interpretable) model that might sacrifice some accuracy but provides clear insights into its decision-making process?

Explainability spectrum illustration

As models become more powerful and complex, they often become less interpretable. The explainability dimension of ML models exists on a spectrum, from highly interpretable to completely opaque. This creates a paradox where the most accurate models may be the hardest to explain to stakeholders. Understanding where different models fall on this spectrum is essential for making informed decisions about which algorithms to deploy in different contexts.

Interpretable models​

Interpretable models have internal workings that are transparent and can be directly understood by humans. These models allow practitioners to trace exactly how inputs are transformed into outputs, making it possible to explain predictions in clear, logical terms.

Key characteristics of interpretable models​

  • Transparency: The model's internal logic is visible and understandable.
  • Feature attribution: The contribution of each input feature to the output can be clearly quantified.
  • Monotonicity: The relationship between inputs and outputs follows predictable patterns.
  • Simplicity: The model has a limited number of parameters or rules.

Examples of interpretable models​

  • Linear and logistic regression.
  • Decision trees (especially smaller ones).
  • Rule-based systems.
  • K-Nearest Neighbours.
  • Naive Bayes classifiers.

Black-box models​

At the other end of the spectrum are black-box models — complex algorithms whose internal workings are difficult or impossible to interpret directly. These models often achieve superior performance by capturing intricate patterns in the data, but at the cost of transparency.

A black-box model might be ethically acceptable for recommending movies, but problematic for determining criminal sentencing. Always consider the specific context and stakes of your application.

Key characteristics of black-box models​

  • Complexity: Often involving thousands or millions of parameters.
  • Non-linearity: Capturing complex, non-linear relationships in the data.
  • Feature interactions: Modelling sophisticated interactions between features that may not be obvious.
  • Distributed representation: Information is encoded across many parameters rather than in discrete, interpretable components.

Why use a black-box model?​

  • Protect businesses: Can be harder to reverse-engineer or manipulate.
  • Pattern recognition: Can recognise patterns that may be indiscernible with human analysis.
  • Performance: Often deliver superior predictive performance.

Examples of black-box models​

  • Deep neural networks.
  • Complex ensemble methods (Random Forests, Gradient Boosting Machines).
  • Support Vector Machines with non-linear kernels.
  • Deep reinforcement learning systems.

Ethical trade-offs​

Choosing between black-box and interpretable models often involves weighing ethical priorities against performance requirements. This isn't merely a technical decision, but one with significant ethical implications that should be explicitly considered in the model selection process.