A/B Testing for Data Analysts

What Is A/B Testing?

A/B testing (also called split testing) is a controlled experiment used to compare two versions of something — a webpage, email, feature, or product — to determine which performs better. One group of users sees version A (the control), another sees version B (the variant), and the difference in outcomes is measured to determine whether the change had a real effect.

A/B testing is one of the most powerful tools available to data analysts because it allows teams to make decisions based on causal evidence rather than correlation or intuition. Instead of guessing whether a new button color will increase conversions, you can measure it.

The A/B Testing Process

A well-run A/B test follows a structured process: define the goal, form a hypothesis, calculate the required sample size, run the experiment, analyze the results, and make a decision.

Step 1: Define the Goal and Metric

Every A/B test needs a clearly defined primary metric — the single number that determines success or failure. Common metrics include conversion rate (percentage of visitors who complete an action), click-through rate, revenue per user, session duration, and churn rate.

Avoid testing too many metrics at once. Choose one primary metric that maps directly to your business goal, and track a small number of secondary metrics to watch for unintended effects.

Step 2: Form a Hypothesis

A good hypothesis has three parts: the change you are making, the expected effect, and the reason you expect it. For example: "Changing the CTA button from grey to blue will increase the click-through rate because blue is more visually prominent and creates a stronger affordance."

The hypothesis forces you to think through the logic before running the test, which helps you interpret the results later and prevents post-hoc rationalization.

Step 3: Calculate Sample Size

One of the most common A/B testing mistakes is stopping the test too early. Before running an experiment, calculate the minimum sample size needed to detect a meaningful effect with sufficient statistical power.

This calculation requires three inputs: the baseline conversion rate (current performance), the minimum detectable effect (the smallest improvement that would be worth implementing), and the desired statistical power and significance level (typically 80% power and 95% significance, i.e., alpha = 0.05).

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

baseline = 0.05       # 5% current conversion rate
mde = 0.01            # Want to detect at least 1pp improvement (to 6%)
alpha = 0.05          # 5% false positive rate
power = 0.80          # 80% chance of detecting a true effect

effect_size = proportion_effectsize(baseline + mde, baseline)
analysis = NormalIndPower()
n = analysis.solve_power(effect_size=effect_size, alpha=alpha, power=power)
print(f"Required sample size per group: {int(n)}")

Running the test until you see a significant result (known as "peeking") inflates the false positive rate dramatically. Commit to your sample size in advance.

Step 4: Run the Experiment

Randomly assign users to control (A) or variant (B) groups. Randomization is critical — it ensures the two groups are comparable and that any difference in outcomes can be attributed to the change rather than pre-existing differences between users.

Key principles for a clean experiment: run both variants simultaneously (not sequentially, as time-based confounders like seasonality will bias results), ensure each user is consistently assigned to the same group throughout, avoid making other changes to the product during the test, and let the experiment run for at least one full business cycle (usually one or two weeks) to smooth out day-of-week effects.

Step 5: Analyze the Results

Once you have reached your target sample size, analyze the results using a statistical test. For conversion rate experiments, use a two-proportion z-test or chi-squared test. For continuous metrics like revenue per user, use a t-test.

import numpy as np
from scipy import stats

# Conversion counts
control_conversions = 430
control_visitors = 9800
variant_conversions = 512
variant_visitors = 9750

control_rate = control_conversions / control_visitors
variant_rate = variant_conversions / variant_visitors

# Two-proportion z-test
from statsmodels.stats.proportion import proportions_ztest

counts = np.array([variant_conversions, control_conversions])
nobs = np.array([variant_visitors, control_visitors])
z_stat, p_value = proportions_ztest(counts, nobs)

print(f"Control rate: {control_rate:.4f} ({control_rate*100:.2f}%)")
print(f"Variant rate: {variant_rate:.4f} ({variant_rate*100:.2f}%)")
print(f"Relative lift: {(variant_rate - control_rate)/control_rate*100:.1f}%")
print(f"p-value: {p_value:.4f}")
print(f"Result: {'Significant' if p_value < 0.05 else 'Not significant'}")

Interpreting the p-value and Statistical Significance

The p-value is the probability of observing a difference at least as large as the one you measured, assuming the null hypothesis (no difference) is true. A p-value below 0.05 means: if there were truly no difference, you would see a result this extreme less than 5% of the time. By convention, this is considered statistically significant.

Common misinterpretations to avoid: the p-value is NOT the probability that the null hypothesis is true, and it is NOT the probability that your result was due to chance. A statistically significant result is not necessarily practically significant — a 0.1% lift in conversion may be detectable but not worth implementing.

Always report the confidence interval alongside the p-value. A 95% confidence interval gives the range of plausible values for the true effect. If the interval is [+0.2%, +3.8%], the result is statistically significant but the effect could be anywhere in that range.