Enter visitors and conversions for version A and version B to see whether the difference is statistically significant, how big it could really be, and the chance B is better. Plan your next test's sample size below.
How to use the A/B test calculator
- Enter the visitors (or sessions, or users) and conversions for the control (A) and the variant (B).
- Choose the confidence level, usually 95%, and a two-sided test unless you decided in advance to look only for B being better.
- Read the verdict, the uplift with its interval, and the chance to beat. Open the mismatch check if your split wasn't 50/50.
Worked example
A converts 500 of 10,000 visitors (5.00%) and B 580 of 10,000 (5.80%): a 16% relative uplift. The pooled rate is 1,080 ÷ 20,000 = 5.4%, so the standard error is √(0.054 × 0.946 × (1/10,000 + 1/10,000)) = 0.0032, and z = 0.008 ÷ 0.0032 = 2.50. The two-sided p-value is 0.0123, below 0.05, so the result is significant at 95%. The 95% interval for the difference is +0.17 to +1.43 points, and the chance B beats A is 99.4%.
Planning the sample size
Visitors per variant = (z₁₋α/₂ √(2p̄q̄) + z₁₋β √(p₁q₁ + p₂q₂))² ÷ (p₂ − p₁)², where p₁ is today's rate, p₂ the rate you hope to detect and p̄ their average. To spot a 10% relative lift on a 5% conversion rate (5% → 5.5%) at 95% confidence and 80% power you need 31,234 visitors per variant, 62,468 in all. Small lifts on small rates need a lot of traffic, which is why it pays to plan first.
Methods
- Significance: two-proportion z-test with pooled variance (the 2×2 χ² test).
- Interval for the difference: Newcombe's hybrid score method (1998); for the relative uplift: the log of the ratio.
- Chance to beat: exact Beta posteriors with flat priors (Evan Miller's formula).
- Mismatch check: χ² test of the visitor split; flagged below p = 0.001.
Runs in your browser; your numbers aren't uploaded.
Frequently asked questions
What does statistically significant mean in an A/B test?
That a difference as big as the one you saw would be unlikely if the two versions really converted equally. At 95% confidence the p-value must be below 0.05: if there were no real difference, a result this extreme would turn up less than 5% of the time. It doesn't tell you how big the real effect is; the confidence interval does.
How do I read the confidence interval?
It's the range of differences in conversion rate the data supports. With 500 of 10,000 converting on A (5%) and 580 of 10,000 on B (5.8%), the 95% interval for B − A runs from about +0.17 to +1.43 percentage points: B is very likely better, but the true lift could be small. If the interval includes 0, the test hasn't shown a difference.
What is 'chance to beat'?
A Bayesian figure: with a flat starting assumption, the probability that B's true conversion rate is higher than A's, given your data. In the example above it's 99.4%. It's easy to explain to a team, but it says nothing about how much better B is, so read it together with the uplift and the interval.
Why might another calculator give a slightly different p-value?
This one uses the standard two-proportion z-test with pooled variance, the same as a 2×2 chi-squared test without continuity correction. Calculators that use the unpooled (Wald) variance, a continuity correction, Fisher's exact test or a t-test give slightly different p-values, especially with small samples. For the interval this tool uses Newcombe's score method, which holds up when a variant converts at 0% or 100%.
Can I stop the test as soon as it's significant?
Not safely. Checking every day and stopping at the first p below 0.05 makes false winners much more likely than 5%. Decide the sample size before you start (use the planner), run whole weeks, and check significance at the end. The SRM check also warns if the traffic split is off, which usually means the test is broken.
