Skip to main content

Optimisation algorithms in practice

Instruction and application
In Progress

Training a model isn’t just about finding the right direction—it’s about getting there efficiently. Imagine driving toward your destination: gradient descent sets the route, but the optimiser decides how fast and smoothly you travel.

In this section, you’ll move beyond the basics of gradient descent to see how modern optimisers make training faster and more reliable. It’s time to see how real-world optimisers improve their performance. Each algorithm has its own way of balancing speed, stability, and computational cost.

Adaptive optimisers and their mechanics​

Standard gradient descent methods use a fixed learning rate throughout the training process. However, choosing an appropriate learning rate can be challenging, and a fixed rate might not be optimal for all parameters or all stages of training.

Adaptive learning rate methods address this by adjusting the learning rate for each parameter individually and/or adapting the learning rate during training based on the gradients observed so far.

The benefits of adaptive learning rate methods​

  • Faster convergence: they often converge faster than standard gradient descent methods, especially in complex loss landscapes.

  • Less sensitive to the initial learning rate: they can often work well with a wider range of initial learning rates, reducing the need for extensive tuning.

  • Handle sparse gradients well: methods like Adagrad and RMSprop can effectively adapt the learning rates for features that appear infrequently.

  • Individualized learning rates: they allow each parameter to have its own learning rate, which can be beneficial when different parameters have different sensitivities.Examples of adaptive rate learning methods

  • **Momentum and Nesterov Accelerated Gradient (NAG):**Momentum helps the model maintain direction by adding a fraction of the previous gradient to the current update—like pushing a ball downhill to build consistent speed. NAG improves on this idea by looking ahead before applying the update, helping the optimiser slow down before overshooting the minimum. Together, these methods make training faster and more stable.

  • Adam (Adaptive Moment Estimation): Adam is a popular and widely used optimisation algorithm that combines the benefits of both RMSprop and momentum. It maintains estimates of both the first moment (mean) and the second moment (uncentered variance) of the gradients, using these to adapt the learning rate for each parameter. Adam is generally robust and performs well on a wide range of problems, often requiring less hyperparameter tuning for the learning rate.

  • RMSprop (Root Mean Square Propagation): RMSprop adapts the learning rate for each parameter by dividing the learning rate by the exponentially decaying average of the squared gradients for that parameter. This helps to dampen oscillations in directions with large gradients and increase the learning rate in directions with small gradients, leading to faster convergence. Other adaptive learning rate methods include:

**Adagrad (Adaptive Gradient Algorithm)**Adagrad adapts the learning rate for each parameter based on the cumulative sum of squared gradients up to the current iteration. Parameters that have received large gradients in the past will have their learning rate decreased, while parameters with small historical gradients will have their learning rate increased.

However, Adagrad's learning rate can become very small over time, potentially halting learning too early.

AdadeltaAdadelta is an extension of Adagrad that addresses the diminishing learning rate issue by using a moving average of squared gradients instead of the cumulative sum. It also eliminates the need to manually set a global learning rate.**Nadam (Nesterov-accelerated Adaptive Moment Estimation)**Nadam combines the Nesterov Accelerated Gradient (NAG) with Adam. NAG is a momentum-based optimiser that looks ahead in the gradient direction before making an update, often leading to faster convergence.

Nadam incorporates this lookahead mechanism into Adam's adaptive learning rate framework. A popular variation, AdamW, decouples weight decay from gradient updates. This modification improves generalisation and helps prevent overfitting, making it a preferred choice for large deep learning models.

The risks of adaptive learning rate methods include​

  • Can over-generalize on small datasets: in some cases, the adaptive nature might lead to overfitting on small datasets.
  • Can get stuck in sharp minima: some adaptive methods might converge to a sharp local minimum that generalizes poorly.
  • Introduce more hyperparameters: while they often reduce the need to tune the initial learning rate, they introduce their own hyperparameters (e.g., the decay rates in Adam and RMSprop) that might need tuning.
  • Can be computationally more expensive per update: maintaining the moving averages or cumulative sums of gradients adds some computational overhead.

Choosing the right optimizer:​

In practice, Adam and RMSprop are often the go-to choices for many deep learning tasks due to their robustness and efficiency. However, the best optimizer ultimately depends on your dataset and computational setup, so experimentation is often necessary.

  • For small or clean datasets, use SGD or SGD with momentum for steady, interpretable convergence.
  • For large or noisy datasets, adaptive methods like RMSprop or Adam handle unstable gradients more effectively.
  • For sparse data, Adagrad works well by adjusting learning rates for infrequent features.
  • If computational cost is high, SGD remains the most lightweight option.

Action item: Quick check — Choosing the right optimizer​

Before moving on, see if you can apply what you've learned by answering the knowledge check questions below.