Skip to main content
Instruction and application
In Progress

Measuring feature impact

Engineering a new feature is only worthwhile if it improves the model or helps you make better decisions. This lesson focuses on how to evaluate whether a feature is truly adding value.

Target illustration

Scenario: Streamly churn prediction​

Imagine you are working with Streamly, a subscription-based media company that wants to identify customers at risk of cancelling their service.

The data science team has engineered a new feature called binge_watch_sessions, which counts how many times a user watches three or more episodes of the same show in a single day.

The business question is simple: does this feature actually help predict churn?

Feature importance metrics​

Feature importance metrics estimate how useful a variable is for prediction. The definition of “importance” depends on the model and the evaluation method you choose.

Model-based importance​

  • Tree-based importance: Measures how much a feature reduces impurity or loss across model splits.
  • Linear model coefficients: Show direction and approximate strength after features are scaled and multicollinearity is addressed.

Permutation importance​

Permutation importance is model-agnostic. You shuffle one feature, re-score the model and see how much performance drops. If the drop is large, the model was relying on that feature.

Lightbulb icon

Why permutation importance is useful​

It tests the feature independently of the model's internal mechanics, which makes it especially helpful when you need to explain feature relevance in production settings.

Correlation analysis​

Correlation analysis measures the statistical relationship between a feature and the target, or between features themselves.

Pearson correlation​

Use Pearson's r when you are testing linear relationships between continuous variables. Values close to -1 or +1 indicate stronger linear relationships.

Chi-square tests​

For categorical predictors in classification problems, a chi-square test can tell you whether the distribution of the target differs meaningfully across categories.

Pause icon

Limitations to remember​

  • Pearson correlation only captures linear relationships and is sensitive to outliers.
  • Chi-square only works with categorical predictors and expected cell counts must be large enough.
  • Neither method tells the full story when features interact strongly with one another.

Information gain and mutual information​

These metrics measure statistical dependency between a feature and the target, including non-linear relationships.

  • Information Gain: Measures how much uncertainty is reduced when a dataset is split on a feature. It is common in decision trees.
  • Mutual Information: Measures how much information one variable tells you about another, regardless of the exact model being used.

High scores suggest the feature is helping the model make better decisions, although these metrics still need to be interpreted in context.

Visualisation techniques for feature evaluation​

Visualisations help you explain feature choices to stakeholders and spot behaviour that metrics can hide.

  • Box plots: Compare feature distributions across target classes.
  • Histograms: Reveal skew, outliers and unusual distributions.
  • Bar plots: Compare group means or category frequencies.
  • Scatter plots: Surface correlation, clusters or non-linear behaviour.
Lightbulb icon

Stakeholder-friendly framing​

Combine quantitative evidence with a visual explanation. A metric might prove a feature is useful, but a well-chosen plot usually makes that usefulness easier to trust and communicate.

A data scientist has engineered a new feature called 'average_daily_activity_level' for a model predicting user engagement. She wants to assess its usefulness. The model is a Random Forest classifier, and the target variable 'user_engaged' is binary (0 or 1). She also suspects that 'average_daily_activity_level' might have a non-linear relationship with engagement. Which combination of evaluation metrics would be most effective for determining the new feature's impact on the Random Forest model?

A team has added a new feature, 'days_since_last_login', to their model predicting customer churn. They are concerned about potential multicollinearity with an existing feature, 'total_login_sessions_last_month', as they both reflect aspects of user activity. The model being used is a Logistic Regression. After training the model, they observe that the coefficients for both 'days_since_last_login' and 'total_login_sessions_last_month' are unexpectedly large and have switched signs in different training runs. Which of the following is the most appropriate next step to diagnose and address this issue?

You are evaluating a new feature, 'customer_lifetime_value_segment' (categorical: 'Low', 'Medium', 'High'), that you've engineered for a classification model predicting product upgrade probability. You want to visually demonstrate to stakeholders how this new feature might be useful. Which visualisation technique would best illustrate the relationship between this categorical feature and the binary target variable 'upgrade_probability'?

In progress