Mathematical foundations for ML
Instruction and application
CompleteNow that you've explored the connection between the ML lifecycle and mathematics, it's time to dive deeper into the core mathematical concepts that form the bedrock of ML. These foundations will empower you to make informed decisions throughout the ML lifecycle.
Mathematical foundations underpinning the ML lifecycle
The development and optimisation of ML models are rooted in various mathematical disciplines. Here are some key mathematical concepts that are critical to the ML lifecycle.
Statistical learning theory in ML
Understanding statistical learning theory (SLT) is fundamental to ML because it provides a theoretical foundation for why and how learning algorithms work, what their limitations are and how to improve them.
- Understanding noise and error decomposition: SLT helps decompose the total error in a model into three key components: Bias, Variance, and Irreducible error.
- Generalisation and overfitting: A key concept in SLT is generalisation—how well a model performs on unseen data. SLT provides bounds to measure this.
- Data bias and algorithmic bias: SLT helps in understanding how different sampling strategies and algorithms contribute to systematic biases.
- Learning efficiency and sample complexity: PAC learning and uniform convergence help determine the number of samples required for reliable learning.
- Trade-offs in model complexity: SLT formalises the balance between model complexity and error through tools like Occam’s Razor and Structural Risk Minimisation (SRM).
- Regularisation and capacity control: Techniques like L1/L2 regularisation and dropout prevent overfitting by limiting a model’s capacity.
- Algorithm selection and model evaluation: SLT provides theoretical guarantees and helps design better evaluation metrics.
- Robustness and stability: Helps understand why some models are robust to small changes in data while others are not.
Tip
Mathematics underpins almost all ML principles, so it's important to make sure you have the best foundation possible. To help you with this, make a note of any mathematical concepts that you feel a little rusty with and may want a quick refresher.
Linear algebra in ML
Linear algebra concepts like vectors, matrices and tensors form the backbone of many ML algorithms.
- Data representation (vectors, matrices and tensors):
- Vector: A 1D array representing a data point (e.g. a feature vector).
- Matrix: A 2D array representing a dataset.
- Tensor: A multi-dimensional generalisation commonly used in deep learning.
- Matrix operations and their importance:
- Matrix multiplication: Used in linear transformations and neural networks.
- Dot product: Measures similarity between two vectors.
- Transpose: Used in optimising computations.
- Inverse: Used in solving linear systems like linear regression.
- Neural networks and deep learning: Weights and activations are represented as matrices; forward passes involve matrix multiplications.
- Convolutions in image processing: CNNs use special matrix operations (kernels) to extract patterns from images.
Calculus and optimisation in ML
Understanding calculus and optimisation is essential because they form the core principles behind model training and error minimisation.
- Derivatives and gradients in learning: The fundamental goal is to minimise a loss function. Derivatives measure how a function changes, while theGradient points in the direction of steepest increase.
- Gradient descent: The heart of optimisation: An iterative algorithm used to minimise loss functions by updating parameters in the opposite direction of the gradient.
- Variants of gradient descent:
- Batch: Uses the entire dataset (slow but stable).
- Stochastic (SGD): Updates per sample (faster but noisier).
- Mini-batch: A balance between efficiency and stability.
- Adam: Combines momentum and adaptive learning rates.
- Backpropagation in neural networks: Computing gradients efficiently using the chain rule of calculus to update weights.
- Convex vs non-convex optimisation: Convex functions have a single global minimum, while non-convex functions (common in deep learning) may have multiple local minima.
Tip
When approaching mathematical concepts in ML, focus on understanding the ‘why’ behind each principle, not just the formulas. For example, understanding why regularisation prevents overfitting or why gradient descent minimises error will help you apply these concepts more effectively.
Action item: Pause and think
Consider the mathematical principles and concepts described in this section. Reflect on your current understanding of them, then answer the questions below.