Ensemble learning for performance and robustness

Instruction and application

In Progress

What’s smarter than a ML model? A team of them.

Individually, models can be fragile and inconsistent—but combine them, and you unlock the power of collective intelligence.

Just like a panel of experts reaches better conclusions than a single voice, ensemble methods blend the strengths of multiple models to deliver more accurate, reliable, and robust predictions.

Now that you’ve explored tuning single models, it’s time to see how multiple models can collaborate to outperform any one of them. Ensemble learning takes performance and stability to the next level, helping you build systems that are both powerful and dependable.

Ensemble methods

Sometimes, an individual ML model might not be very powerful on its own. For example, decision trees can produce different results each time they are run on the same data depending on which path the algorithm follows.

An ensemble method is when we combine the predictions of several “base” or “weak” learners to make a final prediction. The idea is that a group of diverse models together can achieve better predictive accuracy and generalisation than on their own.

You can think of it like “crowd-sourcing” a prediction, where each model provides its output and the one which is most common is selected.

Ensemble methods help us:

• Reduce both bias and variance through aggregating the outputs of multiple models, leading to more accurate predictions.
• Enhance model robustness and stability by overcoming errors in individual models i.e. if one model makes an error, the others might still predict correctly.
• Get better generalisation to unseen data by combining diverse perspectives, reducing the risk of overfitting.

Let’s take a look at a couple of methods.

**Bagging (Bootstrap Aggregating)**This is a parallel ensemble method that aims to reduce the variance of a base learner. It involves training multiple instances of the same model (the base learners) on different subsets of the training data.

These subsets are created randomly by sampling the original data with replacement (called bootstrapping). Once each model has been trained, their predictions are aggregated to produce the final prediction.

Random Forest is a popular bagging algorithm that uses decision trees as base learners.

How it works:

• For each base learner B, we create a bootstrap sample of size N from the training data with replacement (this means that some data points might be included multiple times in a single bootstrap sample).
• Each base learner (e.g. decision tree or neural network) is trained independently on its bootstrap sample.
• When we want to make a prediction, we do so with each base learner and aggregate the results.Classification: Each base learner makes a class prediction, with the final prediction being the class that receives the majority of the “votes”.Regression: Each base learner predicts a numerical value which is averaged.

When to use it:

• When the base learner is prone to high variance (overfitting). Decision trees (especially deep learning ones) tend to be sensitive to specific training data, so bagging is a common choice.
• When we want to improve the stability and accuracy of a single model.

Benefits:

• Reduced variance: By training on different bootstrap samples and aggregating the results, bagging smooths out the predictions and reduces the impact of high-variance base learners.

• Improved stability: The ensemble is less likely to be drastically affected by small changes in the training data.

• Relatively easy to implement: The process of bootstrap sampling and independent training of base learners is conceptually straightforward.

• Out-of-Bag (OOB) error estimation: Since each base learner is trained on a subset of the data, the data points not included in that specific bootstrap sample (on average, about 37% of the data) can be used as a validation set to estimate the model's performance without the need for a separate validation set. This is known as the OOB error.Risks:

• Increased bias (potentially): While primarily aimed at reducing variance, if the base learners are already highly biased, bagging might not significantly reduce this bias and could even slightly increase it in some cases.

• Reduced interpretability: Ensembles are generally less interpretable than single models, as you are dealing with the combined predictions of multiple learners.

• Computational cost: Training multiple base learners can be computationally expensive, especially if the base learners themselves are complex or the number of learners in the ensemble is large.BoostingThis is a sequential ensemble method that aims to reduce the bias and variance of a base learner.

Unlike bagging, where base learners are trained independently, in boosting, each new base learner is trained to focus on the mistakes made by the previous learners. The learners are combined using a weighted sum (for regression) or a weighted majority vote (for classification), where more weight is given to the better-performing learners.

AdaBoost, Gradient Boosting (GBM), XGBoost, LightGBM and CatBoost are popular boosting algorithms.

How it works:

• The base learners are trained sequentially.
• Each data point in the training set is assigned a weight (initially all weights are equal).
• In each iteration, the new base learner is trained on the re-weighted data, with higher weights given to the data points that were misclassified (or had larger errors) by the previous learners. This forces the new learner to focus on the difficult instances.
• Each trained base learner is assigned a weight based on its performance. More accurate learners typically receive higher weights in the final aggregation.
• Finally, the predictions from the base learners are aggregated.Classification: Each base learner makes a prediction. The final prediction is determined by a weighted majority vote (i.e. more accurate learners get more votes).
• Regression: Each base learner predicts a value. The final prediction is a weighted average.

When to use it:

• When we want to reduce both bias and variance and achieve higher accuracy than a single model can provide.
• When the base learner is relatively weak (slightly better than random guessing). Simple models like shallow decision trees (stumps) are often used as base learners in boosting.

Benefits:

• High accuracy: Boosting algorithms often achieve very high predictive accuracy by iteratively improving the model's performance on difficult instances.

• Bias reduction: By focusing on the errors of previous learners, boosting can effectively reduce the bias of the overall ensemble.

• Feature importance: Some boosting algorithms (like Gradient Boosting and XGBoost) provide a measure of feature importance, which can be useful for understanding the data and the model's decision-making process.Risks:

• Sensitivity to noisy data and outliers: Boosting algorithms can be sensitive to noisy data and outliers, as they try to correct every single misclassification, which might lead to overfitting the noise.

• Potential for overfitting: If the boosting process continues for too many iterations, the model can start to overfit the training data. Careful tuning of hyperparameters (e.g., the number of boosting rounds, learning rate) and the use of regularisation techniques are crucial.

• Computational cost: Sequential training can be computationally expensive, especially if the base learners are complex or the number of boosting rounds is large.

• Interpretability: Similar to bagging, boosted ensembles are generally less interpretable than single models.StackingFinally stacking (or stacked generalisation) is a method that combines the prediction of multiple different base learners by training a new model called a “meta-learner” or “aggregator” to learn how to best combine the base learner’s predictions.

Unlike bagging or boosting, the base learners in stacking are typically different types of model. For example, you might use a Random Forest, Gradient Boosting machine and a Support Vector Machine as the base learners, with a Logistic Regression as the meta-learner.

The predictions of the three base learners on the validation set (or using cross-validation) would be used as features to train the Logistic Regression model to make the final prediction.

How it works:

• Several different base learners (e.g. Random Forest, SVM and Gradient Boosting) are trained on the original training data (called Level-0 models).
• The trained base learners are then used to make predictions on the original training data. Often out-of-fold predictions from a cross-validation procedure are used to avoid information leakage. These predictions become the “meta-features” or “Level-1” data.
• A meta-learner (e.g. Logistic Regression) is trained on these meta-features, with the original target variable as the output. The meta-learner learns how to weigh the predictions of the base learners to make the final prediction. For a new, unseen data point, the base learners first make their predictions. These predictions are then fed as input to the trained meta-learners, which outputs the final ensemble prediction.

When to use it:

• When we have a diverse set of well-performing base learners and we believe that a meta-learner can learn to effectively combine their strengths.
• When the base learners make different types of errors, and we want a model that can learn to correct these errors based on the other learners' predictions.

Benefits:

• Potential for high accuracy: Stacking can often achieve higher accuracy than individual base learners by optimally combining their predictions.

• Combine the strengths of different models: It allows you to combine models that excel in different aspects of the data or make different types of predictions.

• Flexibility: You can choose different types of base learners and a suitable meta-learner for your specific problem.Risks:

• Increased complexity: Stacking introduces more complexity to the modeling process, with the need to train multiple base learners and a meta-learner.

• Risk of overfitting: If the meta-learner is too complex or the process is not carefully implemented (e.g., without using cross-validation to generate meta-features), there is a risk of overfitting the meta-learner to the predictions of the base learners on the training data.

• Reduced interpretability: Stacked ensembles are generally even less interpretable than bagged or boosted ensembles due to the multiple layers of models involved.

• Computational cost: Training multiple diverse base learners and a meta-learner can be computationally expensive.

Tip: Use ensembles to balance bias and variance

• If your models tend to suffer from high variance or bias, ensemble methods can help you achieve more stable and accurate results.
• Explore ensemble approaches such as bagging, boosting, or stacking, and experiment with them in your next analysis to see how they enhance model performance. <g></g><defs><clipPath><rect width="24" height="24" fill="white"></rect></clipPath></defs>## Example: Reducing variance with Random Forests Imagine you’re developing a credit scoring model using a single decision tree to classify borrowers as low or high risk. The model achieves high accuracy on training data but performs inconsistently when retrained or tested on new applicants — a clear sign ofhigh variance. Small shifts in the dataset, such as a few new loan applicants or slightly different income distributions, drastically change the model’s predictions.

By applying Bagging through aRandom Forest, you create an ensemble of hundreds of decision trees. Each tree is trained on a different bootstrapped sample of the data and considers a random subset of features at each split. The individual predictions are then averaged to produce the final output.The results are clear:

• The ensemble’s predictions become more stable and consistent across training runs.
• The model generalises better to unseen applicants, maintaining accuracy on validation data.
• Noise and anomalies in the dataset have less influence on the outcome.

This demonstrates how Bagging, and specifically Random Forests, can dramatically reduce variance, improve robustness, and produce a more trustworthy model compared to a single, unstable decision tree.

Tip

Hyperparameter tuning and ensemble learning often work hand in hand. Fine-tuning ensemble parameters — such as the number of estimators,learning rate, ormaximum tree depth — can greatly enhance your model’s accuracy, stability, and generalisation performance.

Key points

Ensemble methods help overcome the limitations of individual models by combining their predictions to improve accuracy, stability, and generalisation.

Whether using bagging to reduce variance, boosting to reduce bias, or stacking to combine the strengths of different model types, these approaches allow you to build more powerful solutions.

• By applying ensemble techniques in your role, you can create models that better handle complex datasets, mitigate errors, and provide more dependable insights for decision-making.

Action item: Quiz - Ensemble learning in action

Before moving on, see if you can apply what you've learned by answering the knowledge check questions below.