Skip to main content

Regularisation techniques for better generalisation

Instruction and application
Complete

Don't let your models just "memorise"​

Even the smartest models can fail if they memorise instead of learning. Think of a student who memorises every practice test question but struggles to answer new ones on the real exam — that’s overfitting in action.

Even the best optimisation algorithm can’t save a model that memorises its training data. That’s where regularisation comes in—the secret to models that truly generalise and deliver reliable, real-world results.

Student memorising illustration

Why overfitting happens: low bias, high variance​

A common issue we come across when training models is overfitting. This happens when the model adapts too well to the noise and unique patterns in the training data and cannot generalize to unseen data, making it unreliable for deployment.

In essence, overfitting occurs when the model has low bias (it fits the training data extremely well) buthigh variance (it performs inconsistently on new data). Regularisation techniques help balance this trade-off by discouraging excessive model complexity and encouraging simpler, more stable patterns.

How it works​

Regularisation works by adding a penalty term to the model’s loss function, preventing large or unnecessary parameter weights. This allows the model to maintain predictive power while improving generalisation.

Regularization methods​

Regularisation techniques help prevent overfitting by adding a penalty term to the model’s loss function. Let’s take a look at some common techniques.

L1 Regularisation (Lasso regression)​

If you have a dataset with high-dimensionality (i.e. a large number of features), a model will struggle to find the relationships in the data without overfitting.

Lasso regression penalizes the absolute size of the coefficients, driving some to zero. This effectively results in a form of feature selection, resulting in a more sparse and interpretable model.How it works: L1 regularisation adds the absolute value of the magnitude of the coefficients as a penalty term to the loss function.

Mathematically, we represent the loss function as:

L1 Formula

Where $\lambda$ (lambda) is the regularisation strength (a hyperparameter) and $n$ is the number of features. This penalty encourages the coefficients of less important features to become exactly zero.

Choosing $\lambda$ (Lambda)​

  • Cross-validation: Train and evaluate your model with different values of $\lambda$ and choose the one that yields the best performance.
  • Information criteria: Techniques like AIC or BIC balance model fit and complexity.
  • Grid search: Systematically search through a range of values, often using a logarithmic scale.

L2 Regularisation (Ridge Regression)​

Unlike Lasso which penalises unimportant features, Ridge regression looks to reduce the impact of large coefficients by penalising the squared size of the coefficients, shrinking them all towards zero.

We use Ridge regression when all features are potentially relevant, aiming to improve generalisation and reduce multicollinearity.

How it works: L2 regularisation adds the squared magnitude of the coefficients as a penalty term:

L2 Formula

This penalty discourages large coefficient values, shrinking them but rarely making them exactly zero.

Elastic Net Regularisation​

Why choose between L1 and L2 when you can have both? Elastic Net is a hybrid approach that combines both penalties to get the benefits of both feature selection and handling correlated features.

Elastic Net Formula

If you set $\rho$ = 1 you get Lasso regression and $\rho$ = 0 gives you Ridge regression.

When to use Elastic Net​

Use this if you have both problems of too many features andmulticollinearity. It provides a balanced approach between sparsity and stability.

Dropout (Neural Networks)​

Dropout is a technique used when training deep neural networks to prevent over-reliance on specific neurons.

How it works: During training, dropout randomly selects a subset of neurons to be temporarily "dropped out" with a certain probability $p$. This forces the network to learn redundant representations.

Dropout Visualization

Operational efficiency trade-offs​

Regularisation improves model stability but can introduce extra computational cost, especially when multiple hyperparameters require tuning.

  • Stronger regularisation = slower training but better generalization.
  • Weaker regularisation = faster training but higher risk of overfitting.

Key Takeaways​

  • L1/L2/Elastic Net help control model complexity by penalizing large coefficients.
  • Dropout prevents over-reliance on specific neurons in Deep Learning.
  • Early stopping, batch normalisation, and data augmentation further enhance performance.
Reflection: Strengthening model generalization
1. What signs would alert you that your model is overfitting during training, and how could regularization help address them?

Type your reflection here...

2. If you were working with a large, high-dimensional dataset, which regularization method (L1, L2, or Elastic Net) would you try first, and why?

Type your reflection here...

3. How might increasing the regularization strength improve or harm model performance in your specific use case?

Type your reflection here...