Why sample size matters
Conversion rates bounce around by chance. With too few visitors, a variant can look 20% better on Monday and 10% worse by Friday without anything real having changed. Working out the sample size before you start tells you how long the test needs and whether it is worth running at all.
The formula
p̄ = (p₁ + p₂) ÷ 2
n = (z₁₋α/₂ × √(2p̄(1 − p̄)) + z₁₋β × √(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ (p₂ − p₁)²
p₁ is the baseline rate, p₂ the rate you want to be able to detect, and n the visitors each variant needs. z₁₋α/₂ is 1.645, 1.960 or 2.576 for 90%, 95% or 99% confidence; z₁₋β is 0.842 for 80% power and 1.282 for 90%. Total visitors is n times the number of variants, and test length is the total divided by daily visitors.
Getting a test you can finish
- Low baseline rates need far more traffic. If you can, test on a step with a higher rate, such as add-to-cart instead of purchase.
- Testing for a large effect needs fewer visitors, but you will miss smaller real wins.
- Each extra variant adds a full group of visitors and raises the chance of a false positive.
- If the test would take months, test a bolder change or pick a busier page.