Holdout Test Calculator

About the Holdout Test Calculator

A holdout test withholds marketing from a randomly selected portion of your audience or geographic area, then compares outcomes between the exposed and holdout groups. The revenue difference, adjusted for group sizes, reveals the true incremental impact of your marketing spend.

This calculator takes the total revenue generated, the holdout group's projected revenue (extrapolated to the full audience), and the campaign spend to compute incrementality, incremental revenue, and incremental ROI. It's the most reliable method for answering: "What would have happened if we hadn't run this campaign?"

Holdout tests are essential for validating attribution models, optimizing media mix, and preventing wasteful spend on campaigns that don't drive true incremental value. Every sophisticated marketing organization runs regular holdout tests.

When This Page Helps

Holdout tests answer the fundamental question: does this marketing actually work? By measuring what happens when you stop marketing to a subset, you get the clearest picture of incremental value and can calculate true ROI on your marketing investment.

How to Use the Inputs

1. Enter total revenue during the test period.
2. Enter holdout group revenue and the holdout percentage.
3. The calculator projects what revenue would have been without marketing.
4. Enter campaign spend.
5. View incremental revenue, incrementality rate, and incremental ROI.
6. Use results to validate or recalibrate your attribution model.

Example Calculation

Tips & Best Practices

• Hold out 10–20% of your audience for sufficient statistical power.
• Run holdout tests for at least 4 weeks to capture full conversion cycles.
• Use geographic holdouts when user-level randomization isn't possible.
• Randomize at the user level if your platform supports it for cleaner results.
• Run holdout tests quarterly to track changing incrementality over time.
• Start with your largest spend channels — they offer the most optimization potential.

Why Holdout Tests Are the Gold Standard

Holdout tests provide the most rigorous answer to the fundamental marketing question: what is the true incremental value of our spend? Unlike attribution models that redistribute credit for conversions that may have happened anyway, holdout tests reveal the counterfactual by actually observing what happens without marketing.

Designing Effective Holdout Tests

The key to a reliable holdout test is proper randomization. Whether at the user level or geographic level, the holdout group must be statistically equivalent to the exposed group in all relevant dimensions (demographics, purchase history, activity level). Any systematic differences between groups will bias results.

Common Pitfalls

Avoid ending tests early when results look good (or bad) — this creates statistical bias. Don't contaminate the holdout group with marketing from other channels. Account for the full conversion lag before concluding. And be transparent about the revenue sacrifice: holdout testing has a short-term cost for long-term optimization gains.

Frequently Asked Questions

• What is a holdout test?

  A holdout test withholds marketing activity from a randomly selected portion of your audience (the holdout/control group). By comparing outcomes between the exposed group and the holdout, you can measure the true incremental impact of your marketing investments.

• How large should my holdout group be?

  Typically 10–20% of your audience. Larger holdouts provide more statistical power but reduce campaign reach. The optimal size depends on your conversion volume, expected lift, and desired confidence level.

• What is the difference between a holdout test and an A/B test?

  An A/B test compares two variations of a marketing element (e.g., two ad creatives). A holdout test compares marketing exposure vs. no marketing. Holdout tests measure whether marketing works at all; A/B tests optimize which version works best.

• What is incrementality rate?

  Incrementality rate is the percentage of total conversions or revenue that would not have occurred without the marketing campaign. A 30% incrementality rate means 30% of conversions were truly driven by marketing and 70% would have happened anyway.

• How do geographic holdout tests work?

  In geographic holdouts, you select matched pairs of similar markets (cities, DMAs). One in each pair receives the campaign, the other doesn't. Comparing performance between exposed and holdout markets reveals the campaign's incremental impact.

• Can holdout tests hurt my revenue?

  Yes, holding out audience means some potential customers won't see your campaign. However, the revenue impact is typically small (10–20% holdout at most), and the insights gained help optimize 80–90% of your spend for much greater long-term returns.