Feature Engineering for Machine Learning

What Is Feature Engineering?

Feature engineering is the process of using domain knowledge to create, transform, or select input variables (features) that make machine learning models more accurate. Raw data rarely arrives in the form that best exposes the underlying patterns — a transaction timestamp is more useful as day-of-week and hour-of-day; a free-text address becomes more useful as a country code and population density. The quality of features almost always matters more than the choice of algorithm.

This article covers the most important feature engineering techniques with Python code examples using pandas and scikit-learn.

Types of Feature Engineering

Encoding Categorical Variables

Most ML algorithms require numeric inputs, so categorical columns must be encoded. The encoding strategy depends on cardinality (number of unique values) and whether the categories have a natural order.

import pandas as pd
from sklearn.preprocessing import OrdinalEncoder, LabelEncoder

df = pd.read_csv('orders.csv')

# One-hot encoding — best for low-cardinality nominal categories
df = pd.get_dummies(df, columns=['country', 'product_category'], drop_first=True)

# Ordinal encoding — use when categories have a meaningful order
size_order = [['XS', 'S', 'M', 'L', 'XL', 'XXL']]
enc = OrdinalEncoder(categories=size_order)
df['size_encoded'] = enc.fit_transform(df[['size']])

# Target encoding — replace category with mean of target variable
# Useful for high-cardinality columns (e.g., zip code, product_id)
target_means = df.groupby('city')['revenue'].mean()
df['city_encoded'] = df['city'].map(target_means)

# Frequency encoding — replace with how often the value appears
freq = df['city'].value_counts() / len(df)
df['city_freq'] = df['city'].map(freq)

Encoding Strategies by Cardinality

Numeric Transformations and Scaling

Many algorithms (linear regression, SVM, KNN, neural networks) are sensitive to feature scale. Tree-based models (random forests, XGBoost) are not. Always scale after splitting into train and test sets — fit the scaler on the training set only to avoid data leakage.

from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler
import numpy as np

# StandardScaler: mean=0, std=1 — works well for normally distributed features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)   # fit + transform on train
X_test_scaled = scaler.transform(X_test)          # transform only on test

# MinMaxScaler: scales to [0,1] — good for neural networks and image data
mm_scaler = MinMaxScaler()
X_train_mm = mm_scaler.fit_transform(X_train)

# RobustScaler: uses median and IQR — best when data has outliers
rob_scaler = RobustScaler()
X_train_rob = rob_scaler.fit_transform(X_train)

# Log transformation for right-skewed numeric features
df['revenue_log'] = np.log1p(df['revenue'])  # log1p = log(1+x), handles 0

# Box-Cox transformation (requires positive values)
from scipy.stats import boxcox
df['revenue_bc'], lambda_val = boxcox(df['revenue'] + 1)

Datetime Feature Extraction

Timestamps contain rich information that models cannot use directly. Extracting components like hour, day-of-week, and month unlocks cyclical patterns that drive many business metrics.

import pandas as pd

df['order_date'] = pd.to_datetime(df['order_date'])

# Extract calendar components
df['hour'] = df['order_date'].dt.hour
df['day_of_week'] = df['order_date'].dt.dayofweek   # 0=Monday, 6=Sunday
df['month'] = df['order_date'].dt.month
df['quarter'] = df['order_date'].dt.quarter
df['year'] = df['order_date'].dt.year
df['is_weekend'] = df['day_of_week'].isin([5, 6]).astype(int)
df['is_month_end'] = df['order_date'].dt.is_month_end.astype(int)

# Cyclical encoding: convert periodic features to sin/cos pairs
# so the model knows that hour 23 is close to hour 0
import numpy as np
df['hour_sin'] = np.sin(2 * np.pi * df['hour'] / 24)
df['hour_cos'] = np.cos(2 * np.pi * df['hour'] / 24)
df['month_sin'] = np.sin(2 * np.pi * df['month'] / 12)
df['month_cos'] = np.cos(2 * np.pi * df['month'] / 12)

# Days since a reference event (e.g., customer signup)
df['days_since_signup'] = (df['order_date'] - df['signup_date']).dt.days

Creating Interaction and Aggregation Features

# Ratio and interaction features
df['revenue_per_session'] = df['revenue'] / (df['sessions'] + 1)   # +1 avoids div/0
df['discount_impact'] = df['original_price'] * df['discount_rate']
df['cart_value_x_sessions'] = df['cart_value'] * df['sessions']

# Aggregation features: group-level statistics joined back
customer_stats = df.groupby('customer_id').agg(
    total_orders=('order_id', 'count'),
    avg_order_value=('revenue', 'mean'),
    max_order_value=('revenue', 'max'),
    days_since_last_order=('order_date', lambda x: (pd.Timestamp.today() - x.max()).days)
).reset_index()

df = df.merge(customer_stats, on='customer_id', how='left')

# Lag features for time series / sequential data
df = df.sort_values(['customer_id', 'order_date'])
df['prev_order_value'] = df.groupby('customer_id')['revenue'].shift(1)
df['revenue_change'] = df['revenue'] - df['prev_order_value']

Feature Selection

More features are not always better — irrelevant or redundant features add noise, slow training, and hurt interpretability. Feature selection finds the subset that gives the best model performance.