Feature Engineering for Machine Learning and Analytics

What Is Feature Engineering?

Feature engineering is the process of transforming raw data into features — input variables — that better represent the underlying patterns in a dataset, improving the performance of machine learning models and analytical queries. It sits at the intersection of domain knowledge and statistical thinking: good features require an understanding of what the data means in the real world and how predictive algorithms interpret numerical inputs.

The importance of feature engineering is often underestimated. A simple logistic regression model with thoughtfully engineered features will frequently outperform a deep neural network trained on poorly structured raw data. The quality of features is the primary determinant of model quality, and no amount of algorithmic sophistication can compensate for irrelevant or poorly encoded inputs.

Types of Features

Numerical Feature Transformations

Raw numerical features often need transformation before they can be used effectively. Many real-world variables like income, transaction amounts, and website traffic follow power-law or log-normal distributions with long right tails. Linear models and distance-based algorithms assume roughly Gaussian inputs; skewed features can dominate distance calculations and distort model coefficients.

Log transformation is the most common remedy for right-skewed data:

import numpy as np
import pandas as pd

df['log_price'] = np.log1p(df['price'])
df['log_revenue'] = np.log(df['revenue'] + 1)

Standardization (z-score normalization) centers features at zero with unit variance, important for algorithms sensitive to feature scale like SVM, PCA, and regularized regression:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
df['price_scaled'] = scaler.fit_transform(df[['price']])

Binning converts continuous variables into categorical buckets, useful when the relationship with the target is non-linear or when domain knowledge suggests meaningful thresholds:

df['age_group'] = pd.cut(df['age'],
    bins=[0, 18, 35, 55, 100],
    labels=['under_18', '18_35', '35_55', '55_plus'])

Categorical Encoding

Machine learning models require numeric inputs, so categorical variables must be encoded. The choice of encoding method significantly affects model performance.

One-hot encoding creates a binary column for each category level — appropriate for nominal variables with low cardinality (fewer than about 20 unique values):

df_encoded = pd.get_dummies(df, columns=['country', 'device_type'], drop_first=True)

Target encoding replaces each category with the mean of the target variable for that category, handling high-cardinality categoricals efficiently. It must be applied with cross-validation to prevent data leakage:

from category_encoders import TargetEncoder

encoder = TargetEncoder(cols=['product_category'])
df['cat_encoded'] = encoder.fit_transform(df['product_category'], df['target'])

Frequency encoding replaces categories with their frequency in the training set — a simple, leakage-free alternative for shigh-cardinality features:

freq_map = df['city'].value_counts() / len(df)
df['city_freq'] = df['city'].map(freq_map)

Datetime Feature Engineering

Datetime columns contain rich temporal signals that are invisible to models unless explicitly extracted. Standard decomposition pulls out year, month, day of week, hour, and whether a date falls on a weekend or holiday:

df['timestamp'] = pd.to_datetime(df['timestamp'])
df['hour'] = df['timestamp'].dt.hour
df['day_of_week'] = df['timestamp'].dt.dayofweek
df['month'] = df['timestamp'].dt.month
df['is_weekend'] = df['day_of_week'].isin([5, 6]).astype(int)
df['days_since_signup'] = (df['timestamp'] - df['signup_date']).dt.days

Cyclical features like hour of day and month have a circular structure — hour 23 is closer to hour 0 than hour 12 is. Sine and cosine transformation preserves this cyclical relationship:

df['hour_sin'] = np.sin(2 * np.pi * df['hour'] / 24)
df['hour_cos'] = np.cos(2 * np.pi * df['hour'] / 24)
df['month_sin'] = np.sin(2 * np.pi * df['month'] / 12)
df['month_cos'] = np.cos(2 * np.pi * df['month'] / 12)

Interaction and Derived Features

Some of the most predictive features are combinations or transformations of existing ones. Interaction features capture joint effects that individual features cannot express alone:

df['revenue_per_session'] = df['revenue'] / (df['sessions'] + 1)
df['price_x_quantity'] = df['price'] * df['quantity']

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, include_bias=False)
df_poly = poly.fit_transform(df[['age', 'tenure']])

Aggregation features compute statistics about a group — for example, the average purchase amount for a customer's region, or the number of times a product has been returned in the last 30 days. These features encode historical context and are particularly valuable in recommendation systems, fraud detection, and churn prediction.