Statistics details

Overview

This topic explains how to read and use the statistics details tab of an experiment.

Experiments display more information about the experiment's results on the statistics details tab.

The statistics details tab includes:

• the relative difference from each variation
• the credible interval
• the conversion rate or posterior mean, depending on the metric type
• the number of conversions or total value, depending on the metric type
• the number of exposures

Hover over each column's header to view more information about how LaunchDarkly calculated the results for that column. To learn more, read Analytic formulas for experiment variation means.

Relative difference

To view the relative difference between variations, choose a variation from the Relative difference from menu. The relative difference displays in the table for each variation.

Relative difference is the difference between the mean of the variation in the dropdown menu, and the upper and lower bounds of the credible interval of the variation in the table. This range contains 90% of the variation's probable values. For example, imagine you have a chosen variation with a mean of 1%, and the variation in the table has a lower credible interval of 1.1% and an upper credible interval of 1.5%. The difference between 1 and 1.1 is 10%, and the difference between 1 and 1.5 is 50%, so the treatment's relative difference from control is 10% to 50%.

The longer you run an experiment, the more the width of this interval decreases. This is because the more data you gather, the more confidence you can have because the range of plausible values is smaller.

Credible interval

The credible interval is the range that contains 90% of the metric's probable values for the variation.

This means the effect of the variation on the metric you're measuring has a 90% probability of falling between these two numbers. The longer you run an experiment, the more the width of this interval decreases. This is because the more data you gather, the more confidence you can have because the range of plausible values is smaller.

If the metric's aggregation method is by sum, the credible interval will be expressed in values rather than percentages. To learn more, read Unit aggregation method.

Conversion rate

The conversion rate displays for all conversion metrics. Examples of conversions include clicking on a button or entering information into a form.

Conversion metrics can be one of two types: count or binary.

Count conversion metrics

The value for each unit in a count conversion metric can be any positive value. The value equals the number of times the conversion occurred. For example, a value of 3 means the user clicked on a button three times.

The aggregated statistic for count conversion metrics is the average number of conversions across all units in the metric. For example, the average number of times users clicked on a button.

Count conversion metrics include:

• Clicked or tapped metrics using the Count option
• Custom conversion count metrics
• Page viewed metrics using the Count option

Binary conversion metrics

The value for each unit in a binary conversion metric can be either 1 or 0. A value of 1 means the conversion occurred, such as a user viewing a web page or submitting a form. A value of 0 means no conversion occurred.

The aggregated statistic for binary conversion metrics is the percentage of units with at least one conversion. For example, the percentage of users who clicked at least once.

Binary conversion metrics include:

• Clicked or tapped metrics using the Occurrence option
• Custom conversion binary metrics
• Page viewed metrics using the Occurrence option

For funnel optimization experiments, the conversion rate includes all end users who completed the step, even if they didn't complete a previous step in the funnel. LaunchDarkly calculates the conversion rate for each step in the funnel by dividing the number of end users who completed that step by the total number of end users who started the funnel. LaunchDarkly considers all end users in the experiment for whom the SDK has sent a flag evaluation event as having started the funnel.

Posterior mean

The posterior mean displays only for numeric metrics. To learn more, read Custom numeric metrics.

The posterior mean is the variation's average numeric value that you should expect in this experiment, based on the data collected so far.

All of the data in the results table are based on a posterior distribution, which is the combination of the collected data and our prior beliefs about that data. To learn more about posterior distributions, read Frequentist and Bayesian modeling.

LaunchDarkly automatically performs checks on the results data, to make sure that actual context traffic matches the allocation you set. To learn more, read Understanding sample ratios.

Conversions, Total value, and Exposures

Depending on the metric type, the statistics details tab displays one of the following two columns containing the sum of unit values for the numerator of the metric:

• Conversions: the total number of conversions for a conversion metric
• Total value: the total value for a numeric metric

The statistics details tab also displays the exposures column, which is the total number of exposures, or experiment units, for the denominator of the metric.

The raw conversion rate is the number of conversions divided by the number of exposures. The raw mean is the total value divided by the number of exposures.

The raw conversion rate and raw mean may not equal the estimated conversion rate and estimated posterior mean shown in the "Conversion rate" and "Posterior mean" columns.

This can be due to:

• regularization through empirical Bayes priors, or
• covariate adjustment through CUPED (Controlled experiments Using Pre-Experiment Data).