Using funnel optimization experiments

Overview

This guide provides an example of how to create and read the results of a funnel optimization experiment.

A "funnel" is a marketing model that describes a customer's journey through your purchasing cycle, typically from the awareness stage to the purchasing stage. LaunchDarkly's funnel optimization experiments use multiple metrics to track the performance of each of the steps in your funnel over time, and provide results specific to those product funnels. You can use funnel metric groups with funnel optimization experiments to measure the results of multi-step user journeys.

Funnel optimization experiments let your product and marketing teams safely and rapidly increase the performance of your funnels by:

• measuring the performance of your main acquisition and engagement channels
• identifying strong points and weak points in your funnel flow
• testing changes to ensure your funnel's effectiveness improves over time

Prerequisites

To complete this guide, you must have the following prerequisites:

• an understanding of your user funnel and the metrics you want to track at each step in the funnel
• an active LaunchDarkly account with Experimentation enabled, and with permissions to create flags, metrics, and experiments

Concepts

Before you build an experiment, you should read about and understand the following:

• randomization units
• primary and secondary metrics
• mutually exclusive experiments
• attribute filters

Example: new user sign-up flow

In this example, we'll demonstrate how to create an experiment that measures the effectiveness of two different versions of a sign-up form on increasing sign-ups for a product. The experiment will measure the effect on each step in the funnel and determine the most effective variation overall.

Creating this experiment requires six steps:

1. Creating a feature flag to experiment on: You can create a new flag, or you may already have an existing flag, to run an experiment on. This flag should control the feature on your site you want to experiment on, and include the variations you want to test as well as a control variation. You can use a boolean flag if you have just one variation to test in addition to the control, or a multivariate flag if you have more than one.
2. Creating the metrics for your metric group: Create a metric for each end-user action you want to track as part of your funnel. This might include actions like clicking a sign-up button, entering contact information, opting in to receive emails, and completing payment.
3. Creating a funnel metric group: Funnel metric groups organize your metrics together in a set order. This ensures you're tracking end-user behavior in the correct sequence, and lets you reuse the same group in multiple experiments so you don't have re-set up your metrics each time.
4. Building the experiment: Building the experiment connects the metric group to your flag and determines the audience you want to run the experiment on.
5. Turning flag targeting on and starting an iteration: Your experiment will begin recording data after you turn on flag targeting and start an iteration. You can stop the iteration after you have received enough results to make a decision about the winning variation.
6. Reading experiment results: The experiment's Results tab includes information about how each variation performed in each step of the funnel, as well as the winning variation overall.

These steps are explained in detail below.

Create the feature flag

Imagine you have noticed that in your sign up flow, a lot of users drop off at your payment step. You think it might be related to the user interface (UI), so you’re going to test two new versions of the payment screen.

To test these versions, first create a feature flag for your experiment with three variations: a payment option with a drop-down menu, with a radio button, and with the current UI which will serve as the control.

To create the flag:

1. Click Create and choose Flag. The "Create new flag" page appears.
2. Enter "Enable new sign-up menu options" for the Name.
3. (Optional) Enter a Description of the flag.
4. Choose the Experiment flag template.

5. Select the String flag variation type.
6. For the first variation, update the Name to "Control."
7. For the second variation, update the Name to "Dropdown select."
8. For the third variation, update the Name to "Radio select."
9. Click Create flag.

Here is what the Variations section of a flag's right sidebar looks like:

To learn more about creating flags, read Creating new flags.

Create metrics

Next, create the metrics in your funnel.

In this example, your sign-up flow has four steps:

1. The customer clicks your site's "Sign up" button
2. They enter their personal information
3. They add a payment method
4. They click the "Pay now" button

This table explains the metrics you will create for this example:

Create the "Sign-up button clicked" metric

To create the first clicked or tapped metric:

1. Navigate to the Metrics list.
2. Click Create metric. The "Create metric" dialog appears.
3. Select an event kind of Clicked or tapped.
4. Enter the CSS selector for your sign-up button in the Click targets field. In this example, the selector is .button.
5. Select the "exact match" Target type.
6. Enter the Target URL that corresponds to your sign-up button:

7. Choose Occurrence as what you want to measure.
8. In the "Metric definition" section, select "Percentage of User units that clicked"
9. Enter "Sign-up button clicked" as the Name.
10. Click Create metric.

Create the "Sign-up info entered" metric

To create the second custom conversion binary metric:

1. Navigate to the Metrics list.
2. Click Create metric. The "Create metric" dialog appears.
3. Select an event kind of Custom.
4. Enter an Event key such as info-entered. You will reference this event name in your code snippet when you insert the metric information into your app.
5. Choose Occurrence as what you want to measure.
6. In the "Metric definition" section, select the following:

• Percentage of User units that sent the event,
• where higher is better

7. Enter "Sign-up info entered" as the Name.
8. Click Create metric.

Create the "Sign-up payment added" metric

To create the third metric, repeat the process for creating the "Sign-up info entered" custom conversion binary metric, but name the metric "Sign-up payment added" and choose an event name like payment-entered.

Create the "Sign-up completed" metric

To create the fourth metric, repeat the process for creating the "Sign-up button clicked" clicked or tapped metric, but name the metric "Sign-up completed" with a target type of "Exact match."

To learn more, read Metrics.

Create a funnel metric group

Now that you have created your four metrics, you can combine them together into a funnel metric group.

To create your funnel metric group:

1. Navigate to the Funnel groups tab from the Metrics list.
2. Click Create metric group. The "Create metric group" dialog appears.
3. Enter "New plan sign-up flow" for the Name.
4. Enter "click sign up" as the first Step name.
5. Choose the "Sign-up button clicked" metric from the Metric menu.
6. Enter "Info entered" as the second Step name.
7. Choose the "Sign-up info entered" metric from the Metric menu:

8. Click Add metric and repeat steps 4-5 for the remaining two steps in your funnel.
9. Click Create metric group.

To learn more, read Metric groups.

Build the experiment

Next, it's time to put it all together to create a funnel optimization experiment.

As you decide on the percentage of traffic you want to include in the experiment, you may want to consider the number of Experimentation keys you have available in the LaunchDarkly plan you subscribe to. For example, if you have 50,000 Experimentation keys per month included in your plan, and you run ten experiments per month, you may want to limit your experiment audiences to no more than 5,000 keys each. To learn more, read Experimentation keys.

To build the experiment:

1. Click Create and choose Experiment. The "Create experiment" page appears.
2. Enter "New user sign-up flow" for the Name.
3. Enter a Hypothesis such as "Using a better sign-up UI will result in a higher number of overall sign ups for the product."
4. Select Funnel optimization.
5. Click Next. The "Choose randomization unit and attributes" step opens.
6. Choose the "user" context kind from the Randomization unit menu.
7. Click Next. The "Select metrics" step opens.
8. Choose your "New plan sign-up flow" Funnel group:

9. Click Next. The "Choose flag variations" step opens.
10. Choose "Enable new sign-up menu options" from the Flag menu.
11. Click Next. The "Set audience" step opens.
12. (Optional) Check the Add experiment to layer checkbox if you want to exclude contexts in this experiment from other experiments.
13. Choose the default targeting rule to run the experiment on.

14. Select the percentage of traffic you want to include in the experiment. In this example, 10% will be in the control variation, 10% will be in the "Dropdown select" variation, and 10% will be in the "Radio select" variation.
15. Select the control variation to serve to the remaining population.
16. Choose to split the experiment audience Equally.
17. Click Finish. You are returned to the experiment's Design tab.

To learn more, read Creating experiments.

Turn flag targeting on and start an iteration

When you're ready, turn your flag's targeting on and begin recording results by clicking Start on the experiment's Design tab:

To learn more, read Turning flags on and off and Start experiment iterations.

Experiment results

After enough users have encountered your experiment, you can evaluate its performance on the experiment's Results tab.

The winning variation displays at the top of the probability report:

The "Radio select" variation is performing best for this funnel.

You can examine the results more closely in the probability report. Here, you can see that the "Radio select" variation has a conversion rate of about 18%:

However, you can also see that in the third "sign up payment added" step, the "Dropdown select" variation actually performed slightly better at 32% than the "Radio select" variation at 31%, but for some reason users aren’t completing the purchase as often. Still, LaunchDarkly declared the "Radio select" variation the winner because it had the highest conversion rate of all the variations in the funnel overall.

To learn more, read Winning variations.

Conclusion

In this guide you learned how to create a funnel optimization experiment to measure the success of a user funnel. You learned how to create the flag, how to set up metrics and a metric group, how to build the experiment, and how to interpret funnel optimization experiment results. To learn more about how you can use Experimentation to meet your business needs, contact contact Sales or your LaunchDarkly representative.