Implications for model performance

Instruction and application

In Progress

Implications for model performance

You’ve diagnosed the problem—now it’s time to treat it. Just like tuning an instrument, getting a machine learning model to perform well in the real world requires subtle, targeted adjustments.

Having quantified the bias-variance tradeoff, we now turn to mitigation techniques that help you fine-tune model performance—boosting generalisation, stabilising predictions, and avoiding the traps of over- or underfitting.

Bias and variance: Impact on model performance

When developing models for real-world use, it's not just about achieving good results on training data—it's about ensuring your model performs ** accurately** ,** consistently** , and** reliably** when it matters most. Bias and variance directly influence these outcomes in distinct ways. Let’s see how.

High bias: Accuracy suffers, generalisation fails

A model with ** high bias** is overly simplistic. It makes broad assumptions about the data and overlooks important relationships between inputs and outputs.

• ** Accuracy impact**: Predictions are consistently off in a similar way—systematically wrong rather than randomly scattered.
• ** Generalisation impact**: The model cannot adapt to new data or subtle shifts in patterns.
• ** Reliability impact**: Users quickly lose confidence in outputs that feel too generic or “one-size-fits-all.”** Example:** A forecasting model that always predicts average demand, ignoring product categories, promotions, or seasonality.

High variance: Accuracy is misleading, reliability breaks down

A model with ** high variance** fits the training data too closely. It reacts to noise and anomalies as if they were meaningful signals.

• ** Accuracy impact**: Performance looks impressive during training but drops sharply on unseen data.
• ** Generalisation impact**: The model performs inconsistently—small shifts in input lead to large shifts in output.
• ** Reliability impact**: Business users may see the model as unpredictable or unstable, especially when deployed in changing environments.** Example:** A fraud detection system that flags benign customer actions because it memorises quirks in historical data.

Key point

To build models that ** generalise well and perform reliably in production** , we must actively balance bias and variance. This becomes even more critical in high-stakes scenarios—where mistakes can mean** lost revenue, compliance risks, or harm to end users** .

Strategies to control bias and variance

Once you’ve identified whether your model is suffering from high bias (underfitting) or high variance (overfitting), the next step is choosing the right mitigation strategy. Let’s explore three widely used techniques, each with its own strengths and tradeoffs.** Regularisation: Simplifying complex models to reduce variance**

** When to use it:** Your model performs well on training data but poorly on validation/test data—classic signs of overfitting and high variance.** What it does:** Regularisation techniques (such as L1 and L2 penalties) add a constraint to the model’s learning process. This discourages it from fitting noise in the training data and helps it generalise better to new inputs.

• L1 regularisation (Lasso): Encourages sparsity by pushing some weights to zero—useful for feature selection.
• L2 regularisation (Ridge): Penalises large weights—helpful for reducing model sensitivity to training data fluctuations. The diagram below compares how L1 and L2 regularisation affect model optimisation. The ** L1 regularisation** forms sharp corners in the error landscape, which makes it more likely for the optimisation process to land on zero-weight solutions—effectively eliminating less important features.The** L2 regularisation** produces smooth, rounded contours that encourage smaller weights without necessarily zeroing them out—reducing overfitting while preserving more features.

** Tradeoff:** Regularisation slightly increases bias by simplifying the model, but significantly reduces variance, improving overall generalisation.** More training data: Strengthening generalisation**

** When to use it:** You observe that your model's performance varies significantly across different subsets of data or validation folds—suggesting high variance.** What it does:** Increasing the amount of high-quality training data helps the model learn stable, generalisable patterns rather than memorising noise. It’s one of the most effective ways to reduce variance and improve robustness—especially in data-hungry models like deep neural networks.** Examples:**

• Collecting more user sessions for behavior prediction.
• Using data augmentation (e.g., image rotation, flipping) in computer vision tasks.
• Expanding synthetic datasets for rare-event detection.** Tradeoff:** Requires more data collection, storage, and processing time. If the model is suffering from high bias, more data won’t necessarily help unless model complexity is also increased.** Ensemble methods: Combining models to reduce variance**

** When to use it:** Individual models are unstable or highly sensitive to training data, and you want more robust, balanced predictions.** What it does:** Ensemble methods combine multiple models to produce a single, aggregated output—reducing the impact of any single model’s errors or noise.

• Bagging (e.g., Random Forests): Trains multiple models on random data samples and averages their predictions—great for reducing variance.
• Boosting (e.g., XGBoost, AdaBoost): Sequentially trains models to correct errors made by previous ones—often improves both bias and variance.
• Stacking: Blends diverse models and learns how to weight their predictions—helps capture complex relationships.** Tradeoff:** Increased computational cost and reduced interpretability—but often leads to higher accuracy and stability in production.

Comparison of common mitigation strategies

The table below summarises when to use each strategy, how it influences bias and variance, and what tradeoffs you should consider when applying it in practice.** Strategy** ** When to use it**** Effect on bias** ** Effect on variance**** Typical tradeoff** ** Regularisation** When model overfits and is too complexSlight increaseSignificant reductionMay slightly lower accuracy but improves generalisation** More training data** When model performs inconsistently across validation/test setsMinimal or noneReductionRequires data availability and computational resources** Ensemble methods** When single models are unstable or overly sensitiveSmall increase (if any)Strong reductionIncreased complexity and reduced interpretability## Bias in model predictions: When performance impacts people** Not all bias is purely technical. In real-world applications, bias in machine learning models can also lead to** unfair outcomes** that affect individuals and groups—especially when the model is used to make decisions about access to loans, jobs, housing, or healthcare.

For example:

• A model that heavily relies on ** zip code** might pick up patterns linked to regional economic data—but it could also act as a proxy for** race or income** , resulting in unfair treatment of applicants from certain neighbourhoods.
• A credit risk model that underpredicts the repayment likelihood for ** younger applicants** may reflect** representation bias**—where that group wasn’t well-represented in the training data. These issues may not affect overall accuracy much, but they can undermine ** fairness** ,** trust** , and** business integrity**—especially if certain groups are consistently disadvantaged by model decisions.

How bias mitigation strategies help

Strategies introduced earlier also help reduce the risk of unfair outcomes:

• ** Regularisation** can reduce overreliance on individual features—especially those that act as proxies for sensitive attributes.
• ** More training data** helps models learn from a broader range of users, improving representation and reducing skewed learning.
• ** Ensemble methods** can stabilise model behavior, reducing erratic predictions that may disproportionately impact specific groups.

Action item: Quick reflection – Choosing your strategy

Based on what you’ve just learned, reflect on your approach to improving a model that’s underperforming in production.** Prompt** Think of a time—or imagine a scenario—where a model delivered inconsistent or poor predictions. How would you determine whether the root cause was high bias or high variance? What strategy would you try first to improve it, and why?