Understanding variables and their impact
Instruction and application
In ProgressUnderstanding variables and their impact
Not every feature contributes to a model in the same way. To engineer useful features, you need to understand what each variable represents, how it behaves mathematically and how it interacts with other inputs.
This lesson explores variable types, feature behaviour and one of the most common modelling risks in tabular data: collinearity.
Example: Predicting purchase likelihood in an online shop
Imagine you are building a machine learning model for an e-commerce platform to predict whether a customer will make a purchase during their next visit.
Your dataset might include the following variables.
| Variable | Type | What to consider |
|---|---|---|
avg_spend_per_visit | Continuous | Often needs scaling or transformation to reduce skew and improve numerical stability. |
items_purchased_last_30_days | Discrete numeric | Count data may benefit from log or root transforms rather than categorical encoding. |
preferred_category | Categorical | Needs encoding such as one-hot or target encoding because the labels have no natural order. |
customer_satisfaction_rating | Ordinal | Carries rank order, so encoding should preserve ordering without inventing false distances unless appropriate. |
Why variable type matters
Choosing the wrong treatment can distort your signal. One-hot encoding a ranked variable can throw away order, while label encoding an unordered variable can create a relationship that does not really exist.
Collinearity and multicollinearity
Once you understand the types of variables in your dataset, the next question is how those variables relate to each other.
Collinearity refers to a strong linear relationship between two input variables. Multicollinearity extends that idea to three or more correlated variables.
For example, in a house-price model, square_footage, number_of_bedrooms and total_rooms often carry overlapping information. That overlap can make it hard for a model to isolate the unique contribution of each feature.
Typical warning signs
- Large swings in model coefficients between training runs
- Features that seem important in one fold but irrelevant in another
- Redundant variables that all describe the same underlying concept
Why is it problematic?
- Interpretability becomes unstable: Linear models can assign erratic or even contradictory coefficients to correlated features.
- Variance is inflated: It becomes harder to estimate the significance of any single predictor.
- Training can become less stable: Optimisation routines struggle when multiple features point in almost the same direction.
- Feature selection gets harder: Redundant inputs can hide the true value of a genuinely useful feature.
How can you detect it?
- Correlation matrices: Look for pairs with very high absolute correlation.
- Variance Inflation Factor (VIF): Measures how much a coefficient's variance is increased by collinearity.
- PCA or eigenvalue screening: Reveals whether a small number of components explain most of the variance across many features.
Tip: Model instability is a clue
If feature importances or coefficients swing drastically across cross-validation folds or after minor data changes, multicollinearity may be the underlying cause.
What solutions exist?
- Drop one feature: If two features carry nearly the same signal, remove the less useful one.
- Combine overlapping features: Merge them into a single metric, ratio or normalised measure.
- Use regularisation: Lasso, Ridge and ElasticNet shrink unstable coefficients.
- Apply dimensionality reduction: PCA, ICA or autoencoders can compress redundant inputs.
- Choose robust models: Tree-based models often tolerate multicollinearity better than linear ones.
Key point
Understanding whether your variables are continuous, discrete, categorical or ordinal is only the first step. You also need to understand how they interact, overlap and reinforce one another if you want your engineered features to be stable and meaningful.