Metric analysis

Overview

This topic explains the different ways you can analyze a metric, and how to select the metric definition options appropriate for your needs.

Unit aggregation method

When you create a custom numeric metric, you must select whether to aggregate event values per unit by sum or by average, depending on your use case.

For example, if your metric is tracking visitor purchases, you can use sum and average to understand the following information as part of an experiment:

• sum: the sum total of purchases customers made, divided by the number of customers, to find the average number of purchases per customer
• average: the overall percentage of customers that made at least one purchase.

Imagine you have three visitors in your online store who make the following number of purchases:

• Visitor A: 3 purchases
• Visitor B: 0 purchases
• Visitor C: 2 purchases

In the example above, three customers make a total of five purchases. Because you have configured the number of purchases as your metric, this number is what LaunchDarkly uses to calculate average and sum.

This table shows an example of how LaunchDarkly would calculate the sum versus the average of your metric's results:

When you create a metric, if you select the "count" measurement method, LaunchDarkly automatically sets the unit aggregation method to "sum." If you select the "occurrence" measurement method, LaunchDarkly automatically sets the unit aggregation method to "average."

Analysis method

You can analyze a metric by the following methods:

• Average: calculated by adding together the metric events received and dividing by the number of units in the group. "Average" is the default analysis method.
• P50: the 50th percentile, which is also the median, is when half of the data is above and half of the data is below this number.
• P75: the 75th percentile is when 75% of the data is below this number and 25% is above this number.
• P90: the 90th percentile is when 90% of the data is below this number and 10% is above this number.
• P95: the 95th percentile is when 95% of the data is below this number and 5% is above this number.
• P99: the 99th percentile is when 99% of the data is below this number and 1% is above this number.

Funnel experiments can only use metrics that use the average analysis method.

The closer the percentile is to 50, the closer it is to the median. Percentiles closer to the median, such as P75, are useful for analyzing general trends. For example, you might want to use the P50 or P75 methods for analyzing things like API latency.

Higher percentiles are better for detecting outliers. For example, if you have an endpoint that generally works well, but handles a single customer with large amounts of data poorly, you might want to use the P95 or P99 methods for the related metric.

Randomization unit

When you create a metric, you must select a randomization unit. The randomization unit is the context kind, such as user, device, or request, that the metric can measure events from.

Some examples of common metrics and their randomization units include "sign-ups by user," "clicks by user," "purchase amount by user," and "latency by request":

To learn more, read Randomization units.

Success criteria

For custom conversion count, custom conversion binary, and custom numeric metrics, you must select whether you want the events the metric is tracking to increase or decrease:

• Higher is better: choose this option for metrics measuring positive things like cart checkouts or sign-ups.
• Lower is better: choose this option for metrics measuring negative things like errors.

LaunchDarkly automatically sets the success criteria to "higher is better" for clicked or tapped metrics and page viewed metrics.

Units without events

Sometimes a context instance will encounter a metric without completing the action the metric is measuring. When this happens, the SDK does not send a metric event to LaunchDarkly. For example, if a conversion metric is measuring completed sign-ups through an email submission form, the SDK will not send an event if the user context opens the form, but never clicks the "submit" button.

For some custom numeric metrics, if you prefer you can configure LaunchDarkly to exclude these units from the analysis entirely:

• For custom numeric metrics that use a percentile analysis method, LaunchDarkly automatically excludes units that did not send any events from the analysis.
• For custom numeric metrics that use the "Average" analysis method, you can:
  • Exclude units that did not send any events from the analysis: this option is best for latency metrics. If LaunchDarkly never receives an event for a context instance, you do not want to default to 0 because LaunchDarkly would interpret this as an extremely fast latency time, which would skew or invalidate the results.
  • Include units that did not send any events and set their value to 0: this option is best for metrics where an incomplete process can be treated the same as 0, such as tracking cart totals for an online store. In this example, customers who put items in their cart but never completed the checkout process are treated as if they purchased $0.

LaunchDarkly automatically sets the value for these units without events to zero for clicked or tapped conversion, custom conversion binary, custom conversion count, and page viewed conversion metrics.