Feature selection and dimensionality reduction
Instruction and application
In ProgressFeature selection and dimensionality reduction
When datasets grow in width, more features do not always mean better performance. High-dimensional data can slow models down, hide useful signal and make interpretation much harder.
What is high-dimensional data?
High-dimensional data refers to datasets with a large number of features. This is common in image processing, NLP, IoT, customer analytics and scientific data.
Why too many dimensions can hurt
- Computational complexity: More memory and longer training times.
- Sparsity: Data points become more isolated and harder to compare.
- Overfitting: Models can memorise noise instead of learning robust patterns.
- Distance degeneracy: Similarity-based methods become less meaningful in very large feature spaces.
Why reduce dimensionality?
- Improve training efficiency
- Remove noise and redundancy
- Make models easier to interpret
- Improve generalisation on unseen data
Feature selection
Feature selection removes irrelevant or redundant variables while keeping the original feature space intact.
Filter methods
Filter methods score features statistically before modelling. Examples include correlation checks, chi-square tests and mutual information.
Wrapper methods
Wrapper methods search across feature subsets using model performance itself. Recursive Feature Elimination and forward selection are common examples.
Embedded methods
Embedded methods select features during training. LASSO, tree-based feature importance and some boosting methods fit into this category.
Feature selection vs dimensionality reduction
Feature selection keeps original variables. Dimensionality reduction creates a smaller set of transformed components or representations.
Dimensionality reduction
Principal Component Analysis (PCA)
PCA transforms the original features into orthogonal components ordered by how much variance they explain. It is widely used when you want to compress structured data while preserving as much information as possible.
t-SNE and UMAP
These are primarily visualisation-focused methods. They are useful when you want to inspect clustering and structure in high-dimensional data, especially in text and image workflows.
Autoencoders
Autoencoders learn compressed latent representations with neural networks. Unlike PCA, they can capture non-linear relationships and are especially useful for unstructured data.