Numerical feature engineering
Instruction and application
In ProgressNumerical feature engineering
Not all numbers speak the same language. Even when data looks clean and numeric, raw values often need to be reshaped, stabilised or enriched before a model can learn from them effectively.
In this lesson, you will explore some of the most common techniques for improving numerical features.
Polynomial features and feature crossing
Many models assume linear relationships, but real-world behaviour often curves or depends on interactions between variables.
- Polynomial features add higher-order terms such as
x²orx³. - Feature crossing multiplies variables together to model interactions, such as
income * education_level.
When to use this
Use polynomial or interaction terms when scatter plots show curves, threshold effects or relationships that become meaningful only when two variables are considered together.
Worked example
Suppose you want to predict house prices from square_footage and num_rooms, but the relationship is clearly non-linear.
import numpy as np
import pandas as pd
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
np.random.seed(0)
square_footage = np.random.uniform(500, 4000, 100)
num_rooms = np.random.randint(2, 8, size=100)
price = (
50 * square_footage
- 0.01 * square_footage **2
+ 20000 * num_rooms
+ 3 * square_footage * num_rooms
+ np.random.normal(0, 10000, size=100)
)
df = pd.DataFrame(`{`
"square_footage": square_footage,
"num_rooms": num_rooms,
"house_price": price,
`}`)
X = df[["square_footage", "num_rooms"]]
y = df["house_price"]
model = make_pipeline(
PolynomialFeatures(degree=2, include_bias=False),
LinearRegression(),
)
model.fit(X, y)
This gives a linear model access to squared terms and interaction terms without changing to a fully non-linear algorithm.
Mathematical transformations
Real-world numerical data is often skewed, noisy or dominated by extreme values. Mathematical transforms can make it easier for the model to detect useful patterns.
log(x + 1): Compresses wide ranges and reduces the impact of outliers.- Square root: Softens right skew without flattening the distribution too aggressively.
- Reciprocal: Useful when the effect of a variable diminishes as values increase.
- Sine and cosine: Helpful for cyclical quantities such as time of day or day of week.
Important constraint
Log and square-root transforms require non-negative inputs. If the feature contains zero or negative values, use a small offset or choose a more flexible method such as Yeo-Johnson.
Binning and discretisation
Sometimes numbers are more useful when grouped into meaningful buckets.
- Equal-width binning: Splits a variable into intervals of the same size.
- Equal-frequency binning: Ensures each bucket contains roughly the same number of observations.
- Custom bins: Uses domain logic such as risk bands or business thresholds.
Binning can improve interpretability and help models capture threshold effects, but too many bins can simply reintroduce noise.
Box-Cox and Yeo-Johnson transformations
When simple transforms are not enough, these methods estimate the most suitable power transformation for the data.
- Box-Cox works on strictly positive values.
- Yeo-Johnson also works when the feature includes zero or negative values.
These are useful when you want a more statistically grounded way to reduce skew or stabilise variance before modelling.