A/B Test Significance Calculator

How to use

1. 1Enter the control group's visitor count and conversion count. A conversion is any binary event you are testing: signup, purchase, click, form submit, anything counted per visitor.
2. 2Add one or more variants and enter the visitor and conversion counts for each. Use Add variant for multi-arm tests with up to eight variants.
3. 3Pick a confidence level. 95 percent is the standard for product and marketing experiments. Use 99 percent for higher-stakes decisions or 90 percent for early-stage exploration.
4. 4Choose the test type. Two-tailed is the safe default. Only use one-tailed if the experiment was designed in advance to only act on variants outperforming control.
5. 5Read the results card for each variant. The colored badge tells you whether the variant is a significant winner, a significant loser, or not significant at your confidence level.
6. 6Check the p-value and confidence interval. A p-value below 1 minus your confidence level means significant. The confidence interval shows the plausible range for the true lift.
7. 7Glance at observed power. If a result is not significant and power is well under 80 percent, the test is inconclusive and may need more traffic, not a verdict of no effect.
8. 8Click Copy summary to grab a clean text block of the verdict, lift, p-value, and z-score for every variant, ready to paste into a launch doc or a Slack message.

About this tool

A/B Test Significance Calculator turns the visitor and conversion counts from a running or finished split test into the standard statistical readout that product managers, growth marketers, and experimentation teams need before declaring a winner. Enter the control group, add one or more variants, and the tool computes each variant's conversion rate with a Wilson 95% confidence interval, the absolute lift in percentage points, the relative lift in percent versus control, the z-score from a two-proportion z-test using a pooled standard error, the one-tailed and two-tailed p-values from the standard normal CDF, the confidence interval for the lift built from the unpooled Wald standard error, and the observed post-hoc power of the test at your chosen alpha. Pick the standard 90, 95, or 99 percent confidence level or type a custom value, and switch between a two-tailed test (the default and the right choice when you care about any direction of change) and a one-tailed test (only when the experiment was set up in advance to detect variants outperforming control). Every result is tagged with a clear verdict: a green significant winner, a red significantly worse than control, or a neutral not significant at the chosen confidence level, so non-statisticians on the team can read the table without having to interpret a raw p-value. Up to eight variants can be compared against a single control, which is the typical setup for a multi-arm split test or a sequential A/B/n redesign. The Wilson interval on each rate keeps the confidence bound calibrated even when conversion rates are very low or very high, where the textbook normal approximation breaks down. The lift interval uses the unpooled standard error because the pooled SE is only appropriate under the null hypothesis (which is what the z-test itself is checking) and underestimates uncertainty in the actual difference once you reject the null. Observed power gives a quick check on whether a non-significant result reflects no effect or simply too small a sample: if power is well below 80 percent at the observed effect, the test is inconclusive rather than negative. The Copy summary button produces a multi-line text block ready to drop into a Slack message, a launch doc, a pull request description, or a quarterly experimentation review. Useful for marketing landing page tests, email subject line tests, signup funnel optimization, checkout flow experiments, pricing page tests, push notification copy tests, mobile app onboarding splits, ad creative tests, and any binary-outcome experiment where one group sees a control experience and other groups see one or more variants. Everything runs locally in your browser; the visitor and conversion counts you enter are never uploaded.

Free to use. Works in your browser. No signup, no login.