Experiment Design Guide

How to design a rigorous A/B test that produces trustworthy results.

Pre-Experiment Checklist

1. Define the hypothesis clearly

• Bad: "The new design is better"
• Good: "Reducing signup form fields from 8 to 4 increases completion rate by at least 5pp"

2. Choose the primary metric

• Must be directly affected by the change
• Must be measurable within the experiment timeframe
• Must have low enough variance to detect your MDE

3. Calculate sample size

• Use required_sample_size_proportions() for conversion metrics
• Use required_sample_size_means() for continuous metrics
• Rule of thumb: if you can't run for at least 7 days, your MDE is too small

4. Define guardrail metrics

• Revenue, error rates, engagement — things that must not degrade
• Set specific thresholds for each

5. Plan for multiple testing

• One primary metric (p<0.05 threshold)
• Apply Bonferroni correction for secondary metrics

Common Pitfalls

Sample Size Rules of Thumb

These assume alpha=0.05, power=0.80, two-sided test.