A/B Testing and Experimentation for Data Analysts

What Is A/B Testing?

A/B testing (also called split testing) is a controlled experiment in which two or more variants of a feature, design, or message are shown to randomly assigned user groups so that their effect on a target metric can be measured with statistical rigor. The core idea is to isolate the impact of a single change by holding everything else constant. Unlike before/after comparisons, a properly randomized A/B test controls for external confounds such as seasonality, marketing campaigns, or natural user behaviour shifts.

Data analysts are responsible for designing experiments correctly, monitoring them during execution, and drawing valid conclusions from the results — all three stages are where mistakes commonly occur.

Key Concepts and Terminology

Sample Size Calculation

Running a test without computing the required sample size is one of the most common mistakes in experimentation. Too few users means the test lacks power to detect real effects; too many wastes time and exposes users to an inferior experience longer than necessary.

from scipy import stats
import math

def required_sample_size(baseline_rate, min_detectable_effect, alpha=0.05, power=0.80):
    """
    Calculate required sample size per variant for a two-proportion z-test.
    baseline_rate: current conversion rate (e.g., 0.10 for 10%)
    min_detectable_effect: smallest relative lift worth detecting (e.g., 0.05 for 5%)
    """
    p1 = baseline_rate
    p2 = baseline_rate * (1 + min_detectable_effect)
    # Z-scores for alpha (two-tailed) and power
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta  = stats.norm.ppf(power)
    pooled  = (p1 + p2) / 2
    n = (z_alpha * math.sqrt(2 * pooled * (1 - pooled)) +
         z_beta  * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2 / (p2 - p1) ** 2
    return math.ceil(n)

n = required_sample_size(baseline_rate=0.10, min_detectable_effect=0.10)
print(f"Users needed per variant: {n}")  # ~15,745

In this example, detecting a 10% relative lift (from 10% to 11% conversion) at 80% power requires roughly 15,745 users per variant — a number that surprises many teams running tests on low-traffic pages.

Running the Statistical Test

Once the experiment has collected enough data, use a two-proportion z-test (for binary outcomes like conversion) or a t-test (for continuous outcomes like revenue per user). The mechanics in Python:

from scipy import stats
import numpy as np

# Conversion data
control_visitors   = 12500
control_converts   = 1175   # 9.4% conversion
treatment_visitors = 12500
treatment_converts = 1312   # 10.5% conversion

# Two-proportion z-test
count = np.array([treatment_converts, control_converts])
nobs  = np.array([treatment_visitors, control_visitors])
z_stat, p_value = stats.proportions_ztest(count, nobs)

control_rate   = control_converts / control_visitors
treatment_rate = treatment_converts / treatment_visitors
relative_lift  = (treatment_rate - control_rate) / control_rate * 100

print(f"Control rate:   {control_rate:.2%}")
print(f"Treatment rate: {treatment_rate:.2%}")
print(f"Relative lift:  {relative_lift:.1f}%")
print(f"z-statistic:    {z_stat:.3f}")
print(f"p-value:        {p_value:.4f}")
print("Significant!" if p_value < 0.05 else "Not significant.")

Querying Experiment Results in SQL

Most experimentation platforms store assignment and event data in a data warehouse. Analysts query it directly to compute per-variant metrics before passing them to statistical tests.

-- Step 1: Get users assigned to each variant
WITH assignments AS (
  SELECT user_id, variant   -- 'control' or 'treatment'
  FROM experiment_assignments
  WHERE experiment_id = 'checkout_button_color'
    AND assigned_at BETWEEN '2024-03-01' AND '2024-03-14'
),
-- Step 2: Join to conversion events
conversions AS (
  SELECT
    a.variant,
    COUNT(DISTINCT a.user_id)                        AS visitors,
    COUNT(DISTINCT e.user_id)                        AS converters,
    COUNT(DISTINCT e.user_id) * 1.0
      / COUNT(DISTINCT a.user_id)                    AS conv_rate,
    AVG(e.order_value)                               AS avg_order_value
  FROM assignments a
  LEFT JOIN events e
    ON a.user_id = e.user_id
   AND e.event_type = 'purchase'
   AND e.event_time BETWEEN '2024-03-01' AND '2024-03-14'
  GROUP BY a.variant
)
SELECT * FROM conversions;

Common Pitfalls and How to Avoid Them

Beyond A/B: Multivariate and Bandit Testing

Standard A/B tests compare two variants but become impractical when testing many combinations simultaneously. Two alternatives are common in production systems.