Tracking AI metrics

Use the TrackRequest function to record metrics from your AI model generation.

Here's how:

var response = tracker.TrackRequest(Task.Run(() =>
  {
    // Make request to a provider, which automatically tracks metrics in LaunchDarkly.
    // When sending the request to a provider, use details from tracker.Config.
    // For instance, you can pass tracker.Config.Model and tracker.Config.Prompt.
    // Optionally, return response metadata, for example to do your own additional logging.
    return new Response
    {
      Usage = new Usage { Total = 1, Input = 1, Output = 1 }, /* Token usage data */
      Metrics = new Metrics { LatencyMs = 100 } /* Metrics data */
    };
  }
));
COPY

If you would like to do any additional tracking, besides what LaunchDarkly provides, it is your responsibility to fill in the Response object with the data you want to track.

You can also use the SDK's other Track* functions to record these metrics manually. The TrackRequest function is expecting a response, so you may need to do this if your application requires streaming.

Each of the Track* functions sends data back to LaunchDarkly. The Monitoring tab of the AI config in the LaunchDarkly UI aggregates data from the Track* functions from across all versions of the AI config.

Here's how to record metrics manually:

/// Track your own start and stop time.
/// Set duration to the time (in ms) that your AI model generation takes.
/// The duration may include network latency, depending on how you calculate it.
tracker.TrackDuration(response.Metrics.LatencyMs);
COPY

To learn more, read LDAIConfigTracker.

Use one of the track[Model] functions to record metrics from your AI model generation. The SDK provides separate track[Model] functions for several of the models that you can select when you set up your AI config version in the LaunchDarkly user interface.

Here's how:

const { tracker } = aiConfig;
const completion = await tracker.trackOpenAI(
  // Pass in the result of the OpenAI operation.
  // When you call the OpenAI operation, use details from aiConfigValue.
  // For instance, you can pass aiConfig.config.prompt
  // and aiConfig.config.model to your specific OpenAI operation.
  //
  // For a complete example, visit https://github.com/launchdarkly/js-core/tree/main/packages/sdk/server-ai/examples/openai.
);
COPY

You can also use the SDK's other track* functions to record these metrics manually. You may need to do this if you are using a model for which the SDK does not provide a convenience track[Model] function. The track[Model] functions are expecting a response, so you may also need to do this if your application requires streaming.

Each of the track* functions sends data back to LaunchDarkly. The Monitoring tab of the AI config in the LaunchDarkly UI aggregates data from the track* functions from across all versions of the AI config.

Here's how to record metrics manually:

// Track your own start and stop time.
// Set duration to the time (in ms) that your AI model generation takes.
// The duration may include network latency, depending on how you calculate it.
aiConfig.tracker.trackDuration(duration);
COPY

To learn more, read LDAIConfigTracker.

Use one of the track_[model] functions to record metrics from your AI model generation. The SDK provides separate track_[model] functions for several of the models that you can select when you set up your AI config version in the LaunchDarkly user interface.

Here's how:

tracker = config_value.tracker
completion = tracker.track_openai(
  # Pass in the result of the OpenAI operation.
  # When calling the OpenAI operation, use details from config_value.
  # For instance, you can pass config_value.config['model']['modelId']
  # and config_value.config['prompt'] to your specific OpenAI operation.
)
COPY

You can also use the SDK's other track* functions to record these metrics manually. You may need to do this if you are using a model for which the SDK does not provide a convenience track_[model] function. The track_[model] functions are expecting a response, so you may also need to do this if your application requires streaming.

Each of the track* functions sends data back to LaunchDarkly. The Monitoring tab of the AI config in the LaunchDarkly UI aggregates data from the track* functions from across all versions of the AI config.

Here's how to record metrics manually:

# Track your own start and stop time.
# Set duration to the time (in ms) that your AI model generation takes.
# The duration may include network latency, depending on how you calculate it.
config_value.tracker.track_duration(duration)
COPY

To learn more, read LDAIConfigTracker.