How many visitors, and how many weeks, does your A/B test need?
A conversion is a visitor who does what you want, such as buying or signing up. Enter the share of visitors who convert today, the smallest improvement worth acting on, and your weekly traffic. You get the number of visitors you need, how long to run the test, and what that improvement would be worth. Nothing you type leaves your browser.
Visitors needed per version
–
Enter your numbers to see the answer.
- Total visitors in the test
- –
- How long to run it at your traffic
- –
- Smallest improvement you could catch in 2 weeks
- –
- Worth per week, if the improvement is real
- –
The calculator compares two rates and allows for a change in either direction. With more than two versions, it compares each one with the current version on its own. It does not adjust for the extra chance of a fluke that comes from making several comparisons.
How to read the result
The smallest improvement worth catching is a business choice, not a math one. Ask what improvement would change your decision, then size the test for that. A smaller improvement needs a much bigger sample. Halving it roughly quadruples the visitors you need.
Run the test for whole weeks, even if you reach the sample sooner, so every weekday counts equally. Decide the sample size before you start, and do not stop early because the result looks good. If you keep checking and stop the first time the result looks like a clear win, you will often pick false winners.
When the test would take too long
- Test a bigger change. Small tweaks have small effects. A bolder version is easier to catch.
- Measure closer to the change. A click on the part you changed happens more often than a purchase, so it needs fewer visitors.
- Accept the trade-off openly. If you cannot reach the sample, say so, and treat the result as a hint, not proof.
The formula, for anyone checking
n per version = (z1-α/2 √(2 p̄ (1 − p̄)) + z1-β √(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ (p₂ − p₁)²
Here p₁ is the share who convert today, p₂ is that share after the improvement, and p̄ is the average of the two. The two z values come from the bell curve and match your "how sure" and "chance of catching" settings. Run time is the total sample divided by weekly traffic, rounded up to whole weeks.
Statisticians call this a two-sided test of two proportions with a normal approximation. "Two-sided" means it allows for a version that does worse as well as better.
Related: ROI and payback calculator, Difference-in-differences, explained for changes you cannot test by splitting visitors at random, and our analytics and reporting service.
Need a test you can defend?
We design tests and evaluations whose answers hold up when leaders and reviewers question them.
Contact us