9 min read
How long to run an A/B test: from sample size to whole weeks
How long to run an A/B test: divide the required sample by eligible daily traffic, round up to whole weeks, and plan for novelty effects and sales.
Articles about A/B test design
RSS feedArticles for product managers and marketers about A/B test design: plain explanations, examples and common mistakes.
9 min read
Minimum detectable effect (MDE): how to choose it for an A/B test
What minimum detectable effect (MDE) means, relative vs absolute, and how to set it from business value and your traffic. With a worked example.
9 min read
Peeking in A/B testing: why "already significant" is not a result
Peeking in A/B testing inflates false positives. In 100,000 simulated A/A tests, daily checks turned a 5% error rate into 27.6%. What to do instead.
9 min read
A/B test sample size: how many users you need
How to calculate A/B test sample size from baseline, MDE, power and significance, with a worked example and the mistakes that waste weeks.
8 min read
Correlation vs causation in product analytics
Correlation vs causation for product teams: confounders, reverse causality, selection bias, and how to test causality with experiments.