Exploratory Data Analysis: A Practical Guide

What Is Exploratory Data Analysis?

Exploratory Data Analysis (EDA) is the practice of systematically examining a dataset before building models or drawing conclusions. Popularised by statistician John Tukey in the 1970s, EDA emphasises visualisation, summary statistics, and pattern detection as a first step in any analytical workflow. The goal is not to confirm a hypothesis but to understand the data's structure, quality, and behaviour well enough to ask better questions.

A thorough EDA catches data quality issues early, reveals distributions that violate model assumptions, uncovers unexpected relationships between variables, and often surfaces the most important findings before any formal analysis begins. Skipping EDA is one of the most common reasons analytical projects produce misleading results.

The EDA Workflow

Data Quality Checks in Python

The first thing to run on any new dataset is a quality audit. Pandas makes this concise:

import pandas as pd
import numpy as np

df = pd.read_csv("orders.csv", parse_dates=["order_date"])

# Shape and dtypes
print(df.shape)
print(df.dtypes)

# Null counts and percentages
null_summary = pd.DataFrame({
    "null_count": df.isnull().sum(),
    "null_pct":   df.isnull().mean().mul(100).round(1)
}).query("null_count > 0").sort_values("null_pct", ascending=False)
print(null_summary)

# Duplicate rows
print("Duplicate rows:", df.duplicated().sum())

# Value ranges for numeric columns
print(df.describe())

# Cardinality for categorical columns
for col in df.select_dtypes(include="object").columns:
    print(f"{col}: {df[col].nunique()} unique values")

Univariate Analysis: Understanding Distributions

For numeric columns, histograms and box plots reveal shape, spread, and outliers. For categorical columns, bar charts show frequency imbalances that can bias models.

import matplotlib.pyplot as plt
import seaborn as sns

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# Histogram with KDE overlay
sns.histplot(df["order_value"], kde=True, ax=axes[0], bins=40)
axes[0].set_title("Distribution of Order Value")

# Box plot to highlight outliers
sns.boxplot(y=df["order_value"], ax=axes[1])
axes[1].set_title("Order Value Box Plot")

plt.tight_layout()
plt.show()

# Skewness and kurtosis
print(f"Skewness:  {df['order_value'].skew():.2f}")
print(f"Kurtosis:  {df['order_value'].kurtosis():.2f}")

# IQR-based outlier detection
Q1  = df["order_value"].quantile(0.25)
Q3  = df["order_value"].quantile(0.75)
IQR = Q3 - Q1
outliers = df[(df["order_value"] < Q1 - 1.5 * IQR) | (df["order_value"] > Q3 + 1.5 * IQR)]
print(f"Outliers: {len(outliers)} rows ({len(outliers)/len(df):.1%})")

Bivariate Analysis: Finding Relationships

Scatter plots and correlation matrices expose how pairs of variables move together. Always visualise before trusting a correlation coefficient — Anscombe's Quartet famously showed four datasets with identical statistics but completely different patterns.

# Correlation matrix heat map
import matplotlib.pyplot as plt
import seaborn as sns

numeric_df = df.select_dtypes(include="number")
corr = numeric_df.corr()

plt.figure(figsize=(8, 6))
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm",
            vmin=-1, vmax=1, square=True)
plt.title("Correlation Matrix")
plt.tight_layout()
plt.show()

# Scatter plot with regression line
sns.regplot(data=df, x="days_since_signup", y="order_value",
            scatter_kws={"alpha": 0.3}, line_kws={"color": "red"})
plt.title("Order Value vs Days Since Signup")
plt.show()

A high correlation between two features does not imply causation. When correlations are high, check whether a third variable (a confounder) could be driving both.