A/B Test Sample Size Calculator

Know how many visitors each variant needs before you call a test.

The control's current rate.

Relative: 20% on a 5% baseline means 5% → 6%.

Effect is

Two-sided. 95% is the usual choice.

Chance of detecting a real effect of this size.

Including the control. An A/B test is 2.

Optional. Visitors entering the test per day, all variants.

Results

Enter a baseline conversion rate and the smallest effect you want to detect. The variant's rate must stay below 100%, and you need at least 2 variants.

Why sample size matters

Conversion rates bounce around by chance. With too few visitors, a variant can look 20% better on Monday and 10% worse by Friday without anything real having changed. Working out the sample size before you start tells you how long the test needs and whether it is worth running at all.

The formula

p̄ = (p₁ + p₂) ÷ 2

n = (z₁₋α/₂ × √(2p̄(1 − p̄)) + z₁₋β × √(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ (p₂ − p₁)²

p₁ is the baseline rate, p₂ the rate you want to be able to detect, and n the visitors each variant needs. z₁₋α/₂ is 1.645, 1.960 or 2.576 for 90%, 95% or 99% confidence; z₁₋β is 0.842 for 80% power and 1.282 for 90%. Total visitors is n times the number of variants, and test length is the total divided by daily visitors.

Getting a test you can finish

  • Low baseline rates need far more traffic. If you can, test on a step with a higher rate, such as add-to-cart instead of purchase.
  • Testing for a large effect needs fewer visitors, but you will miss smaller real wins.
  • Each extra variant adds a full group of visitors and raises the chance of a false positive.
  • If the test would take months, test a bolder change or pick a busier page.

Frequently asked questions

How many visitors do I need for an A/B test?

It depends mostly on your baseline conversion rate and the smallest change you care about. Detecting a move from 5% to 6% at 95% confidence and 80% power takes about 8,160 visitors per variant. Halving the change you want to detect roughly quadruples the sample.

What is the minimum detectable effect?

The smallest difference between control and variant that the test is designed to find reliably. Pick the smallest change that would be worth shipping. A relative effect of 10% on a 4% baseline means 4% → 4.4%; an absolute effect of 1 point means 4% → 5%.

What do confidence level and power mean?

Confidence controls false positives: at 95%, a test with no real difference will look significant about 5% of the time. Power controls false negatives: at 80%, a real effect of the size you entered is detected 4 times out of 5.

Can I stop the test as soon as it is significant?

Not with this kind of test. Checking repeatedly and stopping at the first significant result makes false positives far more likely than the confidence level suggests. Fix the sample size first and read the result once it is reached.

How long should an A/B test run?

Until each variant reaches the sample size, and for at least one full week, so weekday and weekend behaviour are both included. Rounding up to whole weeks is a good habit.

Which formula does this calculator use?

The normal approximation for a two-sided test of two proportions, the same method as R's power.prop.test. Each variant is compared with the control at the confidence level you choose, with no correction for multiple comparisons.

Related tools

All tools