A/B Test Significance Calculator
B's 6.20% beats A's 5.00%, significant at 95% (p = 0.009).
What is A/B test significance?
A/B test significance tells you whether the difference between two variants is a genuine effect or just random noise, using a pooled two-proportion z-test that returns a two-tailed p-value. A p-value below 0.05 means the result is significant at 95% confidence.
The lift figure is the relative gap between the variants: B converting at 6.20% against A at 5.00% is a +24.0% lift. The p-value then says how much to trust that gap.
Both matter. A large lift on tiny samples can be noise, and a small lift on huge samples can be rock-solid. The calculator reports the rates, the lift and the p-value together so you read all three.
How to calculate A/B test significance
Two-proportion z-test → two-tailed p-value
- Enter both variants. Put in visitors and conversions for A and for B over the same test window.
- Read the conversion rates. Each variant gets a rate: 250 of 5,000 is 5.00% for A, 310 of 5,000 is 6.20% for B.
- Check the lift. The lift is B relative to A. 6.20% over 5.00% is a +24.0% relative improvement.
- Check the p-value. A p-value under 0.05 clears the 95% confidence line. Here p = 0.009, so the lift is significant.
Worked example
Variant A converts 250 of 5,000 visitors and variant B converts 310 of 5,000.
| Variant A visitors | 5,000 |
|---|---|
| Variant A conversions | 250 |
| Variant B visitors | 5,000 |
| Variant B conversions | 310 |
| Result | +24.0% lift · 95% significant |
B converts at 6.20% against A at 5.00%, a +24.0% relative lift. The p-value of 0.009 is well below 0.05, so there is under a 1% chance the difference is noise: B is a real winner.
What is a good A/B test significance?
The benchmark for an A/B test is not a "good number" but a confidence threshold. The convention is 95% confidence, meaning a p-value below 0.05, before you call a winner.
| A/B test significance | Read |
|---|---|
| Strong p < 0.01 | Under 1% chance the result is noise |
| Significant p < 0.05 | The 95% confidence convention, a winner you can act on |
| Borderline p ≈ 0.05 to 0.10 | Suggestive, not conclusive: keep the test running |
| Not significant p > 0.10 | The difference is plausibly chance |
95% is a convention, not a law: a higher-stakes change might warrant 99% (p < 0.01). And significance is not the whole story: a significant result on a tiny lift may not be worth shipping, so weigh the lift alongside the p-value.
How to improve A/B test significance
- Gather enough conversions before judging, since small samples swing across the 0.05 line by chance.
- Run over at least one full week so day-of-week behaviour does not bias one variant.
- Pick one change to test at a time, or you cannot attribute the lift to anything specific.
- Weigh the size of the lift, not just the p-value: a significant but tiny gain may not justify the rollout.
Frequently asked questions
- What is A/B test significance and how is it calculated?
- Statistical significance tells you whether the gap between two variants is a real effect or just random noise. This calculator runs a pooled two-proportion z-test on the conversion rates of A and B and returns a two-tailed p-value. A p-value below 0.05 means the result is significant at 95% confidence. With A at 250 of 5,000 and B at 310 of 5,000, the p-value is 0.009, comfortably significant.
- What does a p-value of 0.05 mean?
- A p-value is the probability of seeing a difference this large by chance if the two variants were actually identical. At 0.05 there is a 5% chance the result is a fluke, which is the conventional line for calling a test significant at 95% confidence. A smaller p-value means stronger evidence: 0.009 says there is under a 1% chance the lift is noise.
- My variant looks like the winner but it is not significant. Why?
- A higher conversion rate alone is not proof, because small samples produce big swings by chance. If the p-value is above 0.05, the observed lift could plausibly be noise, so you keep the test running to gather more data. The smaller the true difference between variants, the more traffic you need before it shows up as significant.
- How long should I run an A/B test?
- Run it until you have enough conversions for the result to reach significance, and over at least one full business cycle (usually a week or more) so day-of-week effects average out. Stopping the moment a test crosses 95% is a known trap: results wobble across that line early on, so let the sample build rather than calling it at the first green light.
- What is the difference between lift and significance?
- Lift is how much better B is than A in relative terms (B at 6.20% over A at 5.00% is a +24.0% lift). Significance is how confident you can be that the lift is real and not chance. A test can show a large lift that is not significant on small numbers, or a small lift that is significant on large ones. You want both: a meaningful lift and a low p-value.