A/B test sample size and duration for product tests

Maya

The test plan for the trial-first paywall is signed. Let's run it for a week and read it next Monday.

Alex Tutor

A week might be enough, or it might be a coin toss. The length isn't picked; it's computed from four inputs and one fact about traffic.

Baseline: how often the primary metric happens today. Last quarter, 4.0% of first-time paywall viewers paid within 14 days. MDE, the minimum detectable effect: the smallest change worth finding. Our plan says +1 pp, from 4% to 5%.

Significance level α = 5%, two-sided: the risk of a false win when the paywall does nothing. Power = 80%: the chance of catching a real +1 pp. The theory is in Power and sample size.

You

And the fact about traffic?

Alex Tutor

How many new users enter the test each week. We'll get there.

Alex Tutor

Question 1

Match each input of the size calculation to what it means for the paywall test.

Pick an answer for each row. Choosing an answer that is already used moves it to this row.

Baseline
Choose an answer
MDE
Choose an answer
Significance level α
Choose an answer
Power
Choose an answer

Alex Tutor

For a conversion metric, a standard two-proportion formula gives the users needed per variant. 1.96 is the z-value for α = 5% two-sided and 0.84 for 80% power, so the first factor is (1.96 + 0.84)² = 7.84.

Alex Tutor

n = \frac{(z_{\alpha/2}+z_{\beta})^2\,\left[p_1(1-p_1)+p_2(1-p_2)\right]}{(p_2-p_1)^2}

Users per variant for comparing two conversion rates: p₁ is the baseline, p₂ = p₁ + MDE. Round up.

You

So p₁ = 0.04 and p₂ = 0.05. Everything else is in the formula.

Alex Tutor

Right. Online calculators, including ours, use slightly different versions of this formula, so expect answers a few users apart. Anything close is the same answer.

Alex Tutor

Question 2

How many users per variant does the paywall test need? Baseline 4.0%, MDE +1 pp, α = 5% two-sided, power 80%. Round up to a whole user.

Your answer

Units: users per variant

Alex Tutor

Now the traffic. Only users who see the paywall for the first time enter the test, about 3,400 a week across both variants.

You

13,470 ÷ 3,400 = 3.96 weeks. So 28 days, almost exactly.

Alex Tutor

Round up to whole weeks anyway: 4. Halves has a weekly rhythm. Rent is split at weekends, trips start on Fridays. A test that stops on a Wednesday over-represents some days.

And one more thing: the metric is “paid within 14 days”. The last user who joins on day 28 needs 14 more days before we can count them.

Maya

So when do I actually get the answer?

Alex Tutor

Question 3

How many calendar weeks pass from the test's launch to the readout? Enrollment runs in whole weeks at 3,400 new users a week, and every user needs a full 14-day window.

Your answer

Units: weeks

Theo

Six weeks to see +1 pp. Why not aim for +0.5 pp? Every bit of revenue counts.

You

Half the effect, so twice the users?

Alex Tutor

More. The effect is squared in the denominator: halve the MDE and you need roughly four times the users. Work it out for 4.0% → 4.5%.

Alex Tutor

Question 4

Same test, but the MDE is +0.5 pp (4.0% → 4.5%). How many users per variant? Round up.

Your answer

Units: users per variant

Maya

Sixteen weeks is a whole season. No.

Alex Tutor

That's the trade. The MDE comes from what's worth shipping and what the traffic allows, not from wishes. If the effect you care about is too small to find, test a bolder change.

You

And the baseline? Does it matter which users it comes from?

Alex Tutor

A lot. Take it from the same people who'll enter the test: first-time paywall viewers, not all users. At an 8% baseline, the same +1 pp would need 12,192 per variant, nearly twice as many.

Alex Tutor

Question 5

Which change reduces the number of users the paywall test needs?

Choose one answerRaising power from 80% to 90%Lowering α from 5% to 1%Splitting traffic 90/10 instead of 50/50Setting the MDE at +1.5 pp instead of +1 pp

Maya

Fine, but what's actually wrong with one week? We'd have 1,700 users per variant. That's a lot of people.

Alex Tutor

It's a lot of people and not much evidence. With 1,700 per variant, a real +1 pp would show up only about 29% of the time. Most likely we'd read “not significant” and wrongly call the paywall useless.

You

And if the first week happens to look great, stopping then is the peeking trap.

Alex Tutor

Question 6Short answer · AI-checked task

Maya asks: “Why can't we just run the paywall test for one week?” Answer her in 3–5 sentences, with the numbers from this lesson and at least one honest way to get an answer sooner.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Six weeks it is. Write the date of the readout into the plan now, so nobody is tempted to read it sooner.

Alex Tutor

That’s the lesson. You answered every task — nicely done.

Next lesson: Who is in the test