Managing holdouts

Overview

This topic explains how to manage holdouts and interpret their results.

Manage holdouts

You can view all of a project's holdouts on the Holdouts list:

Holdouts can have four possible states:

• Created: you have not yet added any experiments to the holdout.
• Ready: you have added at least one experiment added to the holdout, but none of the experiments have started.
• Analyzing: at least one experiment is in the holdout, and it is currently running or it has one or more completed iterations.
• Ended: the holdout is finished analyzing the holdout audience against the experiments' audiences, and everyone is now receiving the same experience.

Click on the name of a holdout to view its details page.

The details page lists the holdout's name, description, holdout amount, randomization unit, and any experiments included in the holdout:

Read results

On the holdout details page, the Probability report tab displays the results of the metric for two variations:

• "Not in holdout" includes any contexts included in an experiment within the holdout.
• "In holdout" includes the contexts that were excluded from experiments within the holdout.

You can see which variation is performing better if least one of the experiments in the holdout is running. However, we don't recommend making any decisions about your holdout experiment until all of the experiments within the holdout are finished and you have shipped winning variations for all of them. To learn how, read Winning variations.

When you have recorded enough data, you can stop the holdout and analyze its results. To do this, navigate to the holdout details page and click End. The contexts that were in the holdout will no longer be excluded from future experiments.

You can then make a decision about the results of your holdout based on which variation is performing better:

• If the "Not in holdout" variation is performing better, then this means your experiments are overall having a positive impact on the metric.
• If the "In holdout" variation is performing better, then this means your experiments are overall having a negative impact on the metric, and you may want to examine the experiments you're running and the variations you're testing to figure out how you can build better or more effective experiments.