Essential maths in ML models and algorithms
Instruction and application
CompleteNow that we've explored the fundamental mathematical concepts underpinning ML, it's time to see how these principles come to life in real-world ML models and algorithms. In this section, we'll dive into the essential mathematical operations that power the most widely used ML techniques.
Regression models
In ML, regression models predict continuous outcomes. Key mathematical concepts in regression include:
- Least squares estimation: A method used to estimate the coefficients in linear regression.
- Maximum likelihood estimation: A statistical method for estimating parameters that maximise the likelihood of the observed data.
- Optimisation: Using gradient descent to minimise error in models like linear regression.Mean squared error (MSE): MSE is a key metric in regression models used to measure the quality of a model’s predictions. It calculates the average of the squared differences between actual values and predicted values.
L1 and L2 norms in regression: These techniques add penalty terms to the loss function based on theL1 andL2 norms:
| L1 norm (lasso regression) | L2 norm (ridge regression) |
|---|---|
| Adds a penalty proportional to the absolute values of coefficients. Encourages sparsity and results in automatic feature selection. | Adds a penalty proportional to the square of coefficients. Prevents large coefficients to avoid overfitting, especially with correlated features. |
Key point
Both L1 andL2 norms help improve model generalisation and prevent overfitting by constraining the size of the model coefficients.
Classification models
Classification problems involve predicting discrete categories or labels.
Probability theory
Bayes' Theorem provides a mathematical framework for updating probabilities:
P(H|D) = (P(D|H) * P(H)) / P(D)
Likelihood functions quantify how probable observed data is given a particular model. MLE finds the parameter thatmaximises the likelihood function.
Information theory
Entropy quantifies the uncertainty or randomness in a system:
H(X) = -Σ P(xi) log P(xi)
| Feature selection | Data compression | Uncertainty quantification | Deep learning |
|---|---|---|---|
| Measures information gain in decision trees. | Determines minimum bits needed to encode data. | Helps in probabilistic modelling. | Used in regularisation (loss functions). |
Information gain (IG) measures the reduction in entropy when splitting a dataset.Gini impurity measures how impure or uncertain a dataset is with respect to class labels.
Linear algebra & Ensemble methods
Decision boundaries are mathematical representations that separate classes.Ensemble Learning combines multiple weak learners to form a strong learner.
| Bagging | Boosting | Stacking |
|---|---|---|
| Reduces variance by averaging multiple models. (e.g. Random Forest). | Reduces bias by training models sequentially, correcting previous errors. | Combines predictions of multiple models using a meta-model. |
Clustering models
Clustering is a fundamental unsupervised learning technique used to group similar data points without predefined labels.
- Distance metrics: Quantify similarity (Euclidean distance, Manhattan distance, Cosine similarity).
- Centroid calculation: The centre of mass or average position of a set of points.
Tip
As you begin applying mathematical concepts to real-world ML models, focus on understanding how each concept influences model behaviour. This practical awareness will help you select and fine-tune models more effectively.
Action item: Pause and think
Consider the mathematical principles and concepts described in this section. Reflect on your current understanding of them, then answer the questions below.