Expected Loss Calculator

About the Expected Loss Calculator

Expected loss measures the average conversion rate you'd sacrifice by choosing the wrong variant. While P(B > A) tells you the probability B is better, expected loss tells you the cost of being wrong. A test with 90% P(B > A) and an expected loss of 0.01% is safe to ship because even if B is worse, the damage is negligible.

This calculator computes expected loss using Monte Carlo simulation on Beta posterior distributions. Enter visitors and conversions for control and variant, and see the expected loss for each decision (choosing A vs. choosing B).

Expected loss is arguably the most practical Bayesian metric because it directly answers: "How much would we lose if we make the wrong call?" This risk-based framework enables earlier, more confident decisions than probability alone.

When This Page Helps

Probability alone is insufficient for decision-making. A 92% probability B wins with 0.01% expected loss is safer than a 98% probability with 0.5% expected loss. This calculator adds the risk dimension that makes Bayesian testing actionable.

How to Use the Inputs

1. Enter visitors and conversions for the control (A).
2. Enter visitors and conversions for the variant (B).
3. Review the expected loss for choosing A and for choosing B.
4. Choose the option with the lower expected loss.
5. A common threshold: ship the variant when its expected loss is below 0.1%.

Example Calculation

Tips & Best Practices

• Use expected loss thresholds instead of (or in addition to) probability thresholds for decisions.
• A common threshold is 0.1% (one-tenth of a percentage point of CR) for shipping variants.
• Expected loss near zero means even if you're wrong, the cost is negligible — safe to ship.
• Higher-stakes decisions warrant lower expected loss thresholds.
• Expected loss decreases as you collect more data, even if the probability stays the same.
• Combine expected loss with the revenue impact calculator to convert percentage-point risk to dollar risk.

Expected Loss: The Practical Bayesian Metric

Expected loss completes the Bayesian decision framework. P(B > A) tells you direction, expected loss tells you magnitude of risk. Together, they provide everything needed for a rational shipping decision without arbitrary significance thresholds.

Loss Thresholds by Business Context

For a high-traffic e-commerce site: 0.05% expected loss is very safe, 0.1% is acceptable, 0.5% warrants caution. For low-traffic or high-AOV sites, convert to dollar terms for a more intuitive threshold. The key insight is that expected loss should be compared to the cost of waiting (opportunity cost of not shipping).

Expected Loss Converges Faster Than Probability

An interesting property: expected loss often reaches actionable levels before P(B > A) reaches 95%. A test might show P(B > A) = 88% but expected loss of only 0.01%. This means you can make confident shipping decisions earlier than frequentist or even probability-based Bayesian methods suggest.

Frequently Asked Questions

• What is a good expected loss threshold?

  It depends on your traffic and AOV. A common rule: if the expected loss in CR is below 0.1%, ship the variant. For a 3% baseline CR, 0.1% expected loss means you'd lose at most 0.003 percentage points of CR on average — effectively nothing.

• How is expected loss different from P(B > A)?

  P(B > A) measures how likely B is to be better. Expected loss measures how much you'd lose if B is actually worse. You can have high P(B > A) but also high expected loss (rare but large potential downside), or moderate P(B > A) with negligible loss.

• Why is expected loss better for decisions?

  Expected loss directly quantifies risk in the same units as your metric (conversion rate). A stakeholder can understand "we'd lose 0.02% CR at most" better than "p = 0.03" or "P(B>A) = 93%." It enables practical, risk-based decision-making.

• Can I convert expected loss to revenue?

  Yes. Multiply expected loss (as a proportion) by traffic and AOV. If expected loss is 0.0002 (0.02%) and you have 100,000 monthly visitors at $80 AOV, the expected revenue loss is 100,000 × 0.0002 × $80 = $1,600/month. Very manageable.

• Does expected loss work for revenue metrics?

  Yes. The same framework applies to any metric. For revenue per visitor, the expected loss is in dollars per visitor. The concept generalizes to any posterior distribution and any loss function.

• When should I not use expected loss?

  Expected loss assumes your posterior accurately represents uncertainty. With very small samples (<100 per group) or extreme conversion rates (<0.1%), the posterior may not be well-calibrated. In such cases, gather more data before relying on expected loss.