Guarded rollouts
Read time: 4 minutes
Last edited: Nov 05, 2024
Guarded rollouts is available as an add-on to an Enterprise plan. To learn more, read about our pricing. If you want access to this feature, contact your LaunchDarkly account representative.
Overview
This topic explains how to monitor metrics on flag releases and configure LaunchDarkly to take action on the results. You can use guarded rollouts to notify you or automatically roll back changes that have a negative effect on your app's performance.
About guarded rollouts
When you make a flag change, you can attach metrics to the flag to monitor changes in your app performance and guard against any regressions that result from the release. You can use metrics to track a variety of system health indicators and end-user behaviors, from engineering metrics like errors and latencies, to product metrics like clicks and conversions. To learn more, read Metrics.
You can connect metrics to LaunchDarkly using any of the following methods:
- Use one of our metrics integrations
- Call the metric import API
- Use a LaunchDarkly SDK to send custom events, and connect them to metrics
- Enable OpenTelemetry in a LaunchDarkly SDK, send the OpenTelemetry traces to LaunchDarkly, and connect them to metrics
You can monitor metrics on any flag rule, except rules already serving a percentage rollout or progressive rollout, and those in use in a running experiment.
Guarded rollouts are one of several options that LaunchDarkly provides to help you release features to production safely and gradually. To learn about other options for releasing features, read Releasing features with LaunchDarkly.
Regressions
When you attach metrics to a flag and then toggle the flag on or change the variation served, LaunchDarkly detects if the change is having a negative impact on your app or audience. This negative effect is called a "regression." You can configure LaunchDarkly to either notify you of the regression, or notify you of the regression and automatically roll back the release. To learn how LaunchDarkly detects regressions, read Time series charts and tables.
Regression statistical methodology
This section explains the statistical methodology LaunchDarkly uses to determine if a metric is detecting a regression. We provide this for informational purposes, but you do not need to understand the statistical methodology to use the feature.
Expand Statistical methodology
LaunchDarkly uses two different criteria for regression and methods, depending on whether the metrics are custom numeric metrics or conversion metrics. Conversion metrics include clicked or tapped, custom conversion binary, custom conversion count, and page viewed metrics.
This table explains the different regression criteria for the two metric types:
| Metric type | Unit level aggregation | Contexts without events included or excluded | Analysis type | Regression criteria | Methodology |
|---|---|---|---|---|---|
| Conversion | Number of events | Included | Mean | Probability (mean of treatment > mean of control) > 95% | Bayesian difference in means. To learn more, read Numeric metrics. |
| Numeric | Average of event values | Excluded | p95 | Lower bound of 90% confidence interval of the 95th percentile > 0 | Frequentist always-valid confidence intervals for quantiles. To learn more, read Rapid Regression Detection in Software Deployments through Sequential Testing. |
Conversion metrics
Conversion metrics measure the number of events received per unit during the release. To check for regressions, LaunchDarkly compares the mean of the test variation to the mean of the control variation. LaunchDarkly defines a regression as the mean of the test variation performing worse than the mean of the control variation.
This means that:
- If you set up a metric with the success criteria of "higher is better," LaunchDarkly considers it a regression when the mean of the test variation is lower than the mean of the control variation.
- If you set up a metric with the success criteria of "lower is better," LaunchDarkly considers it a regression when the mean of the test variation is higher than the mean of the control variation.
LaunchDarkly uses the same Bayesian model used for numeric metrics in its Experimentation platform, and considers it a regression when the posterior odds of the model indicate that the probability of a regression is greater than 95%.
The implication of this conversion method is that a release starts with a 50-50% probability of being a regression. As data is ingested, the probability of a regression can move to more extreme values of either 0% or 100%. This method lets LaunchDarkly continuously monitor without alerting on false positives.
Numeric metrics
Numeric metrics measure the average value per unit during the release. To check for regressions, LaunchDarkly compares the 95th percentiles of the treatment and control groups. LaunchDarkly uses the sequential quantile model described in Rapid Regression Detection in Software Deployments through Sequential Testing.
For metrics where a higher value is worse, LaunchDarkly considers it a regression when the lower bound of the 90% confidence interval of the difference in 95th percentiles between the treatment and control groups is greater than zero. For metrics where a lower value is worse, LaunchDarkly uses the criteria that the upper bound is less than zero.
To learn more, read Success criteria.