Bayesian A/B Test Calculator

About the Bayesian A/B Test Calculator

Bayesian A/B testing provides a more intuitive answer than frequentist methods: "What is the probability that B is better than A?" Instead of p-values and significance thresholds, you get a direct probability (e.g., "there is a 96% chance that the variant outperforms the control").

This calculator uses Beta posterior distributions to compute P(B > A) from conversion data. Each group's conversion rate is modeled as a Beta distribution updated with observed successes and failures. The probability of B being better is computed analytically or via simulation.

Bayesian analysis also naturally handles the peeking problem — you can check results at any time without inflating false positive rates, because the probability statement is always valid. This makes Bayesian methods particularly appealing for teams that want to monitor tests continuously.

When This Page Helps

Bayesian analysis answers the question stakeholders actually ask: "How confident are we that B is better?" in a direct probability. No more explaining p-values. The results are actionable and intuitive for non-technical decision-makers.

How to Use the Inputs

1. Enter visitors and conversions for the control group (A).
2. Enter visitors and conversions for the variant group (B).
3. The calculator computes the posterior Beta distribution for each group.
4. Review the probability that B beats A.
5. A probability above 95% is typically used as the decision threshold.

Example Calculation

Tips & Best Practices

• A probability of 95%+ is a common threshold, but the exact threshold depends on the cost of being wrong.
• The non-informative prior (Beta(1,1)) lets the data speak; use informative priors only if you have strong prior knowledge.
• Bayesian analysis handles sequential peeking naturally — the probability is valid at any point during the test.
• For high-stakes decisions, wait until P(B > A) exceeds 99%.
• Expected loss (see companion calculator) adds a risk dimension beyond just probability.
• Bayesian and frequentist methods usually agree for well-powered tests.

The Bayesian Advantage

Bayesian A/B testing eliminates the most common frustrations with frequentist methods: unintuitive p-values, the prohibition on peeking, and binary significant/not-significant conclusions. Instead, you get a continuous probability that naturally accommodates monitoring and evolves as data arrives.

Understanding the Beta Distribution

The Beta distribution is the natural model for proportions (conversion rates). With parameters α (successes + 1) and β (failures + 1), it represents our uncertainty about the true conversion rate. More data = narrower distribution = more certainty.

Decision Framework

Instead of a binary ship/no-ship decision, Bayesian analysis enables nuanced decision-making. You can set different thresholds based on risk: 90% probability for low-cost reversible changes, 95% for standard features, 99% for irreversible or high-cost decisions. Combine with expected loss for even more robust decisions.

Frequently Asked Questions

• What does P(B > A) = 95% mean?

  There is a 95% probability that the variant's true conversion rate is higher than the control's. Unlike a p-value, this is a direct probability statement about the hypothesis, which is more intuitive and easier to act on.

• What prior should I use?

  The default prior Beta(1,1) is non-informative (uniform). This means you have no prior belief about the conversion rate. For mature products, you could use an informative prior based on historical data (e.g., Beta(30, 970) if you know the CR is around 3%).

• Can I check Bayesian tests early?

  Yes, with caveats. The probability P(B > A) is always valid, so peeking does not inflate false positives like in frequentist testing. However, early probabilities fluctuate more due to small samples. Wait until the probability stabilizes before deciding.

• How does Bayesian differ from frequentist testing?

  Frequentist: "If there is truly no difference, would we see data this extreme?" (p-value). Bayesian: "Given the data, what is the probability B is better?" (posterior). Bayesian answers the more natural question and allows valid peeking.

• What is the expected loss?

  Expected loss measures the average cost of choosing B when A might actually be better. It combines probability AND magnitude of being wrong. A test might have 90% probability B wins but low expected loss, making it safe to ship B.

• When do Bayesian and frequentist disagree?

  They rarely disagree meaningfully for well-powered tests with non-informative priors. Disagreements arise mainly with small samples, informative priors, or borderline results. In practice, most A/B tests produce similar conclusions with either method.