Measuring feature impact
Instruction and application
In ProgressMeasuring feature impact
Engineering a new feature is only worthwhile if it improves the model or helps you make better decisions. This lesson focuses on how to evaluate whether a feature is truly adding value.
Scenario: Streamly churn prediction
Imagine you are working with Streamly, a subscription-based media company that wants to identify customers at risk of cancelling their service.
The data science team has engineered a new feature called binge_watch_sessions, which counts how many times a user watches three or more episodes of the same show in a single day.
The business question is simple: does this feature actually help predict churn?
Feature importance metrics
Feature importance metrics estimate how useful a variable is for prediction. The definition of “importance” depends on the model and the evaluation method you choose.
Model-based importance
- Tree-based importance: Measures how much a feature reduces impurity or loss across model splits.
- Linear model coefficients: Show direction and approximate strength after features are scaled and multicollinearity is addressed.
Permutation importance
Permutation importance is model-agnostic. You shuffle one feature, re-score the model and see how much performance drops. If the drop is large, the model was relying on that feature.
Why permutation importance is useful
It tests the feature independently of the model's internal mechanics, which makes it especially helpful when you need to explain feature relevance in production settings.
Correlation analysis
Correlation analysis measures the statistical relationship between a feature and the target, or between features themselves.
Pearson correlation
Use Pearson's r when you are testing linear relationships between continuous variables. Values close to -1 or +1 indicate stronger linear relationships.
Chi-square tests
For categorical predictors in classification problems, a chi-square test can tell you whether the distribution of the target differs meaningfully across categories.
Limitations to remember
- Pearson correlation only captures linear relationships and is sensitive to outliers.
- Chi-square only works with categorical predictors and expected cell counts must be large enough.
- Neither method tells the full story when features interact strongly with one another.
Information gain and mutual information
These metrics measure statistical dependency between a feature and the target, including non-linear relationships.
- Information Gain: Measures how much uncertainty is reduced when a dataset is split on a feature. It is common in decision trees.
- Mutual Information: Measures how much information one variable tells you about another, regardless of the exact model being used.
High scores suggest the feature is helping the model make better decisions, although these metrics still need to be interpreted in context.
Visualisation techniques for feature evaluation
Visualisations help you explain feature choices to stakeholders and spot behaviour that metrics can hide.
- Box plots: Compare feature distributions across target classes.
- Histograms: Reveal skew, outliers and unusual distributions.
- Bar plots: Compare group means or category frequencies.
- Scatter plots: Surface correlation, clusters or non-linear behaviour.
Stakeholder-friendly framing
Combine quantitative evidence with a visual explanation. A metric might prove a feature is useful, but a well-chosen plot usually makes that usefulness easier to trust and communicate.
binge_watch_sessions into the Streamly model, which metric would you check first and why?