03_L_Metric Definition Overview- Part 2 | Summary and Q&A

3.7K views
March 24, 2015
by
Udacity Videos
YouTube video player
03_L_Metric Definition Overview- Part 2

TL;DR

This video discusses the steps involved in creating metric definitions, including high-level concept development, defining specific details, summarizing data measurements, and considering multiple metrics or composite metrics.

Install to Summarize YouTube Videos and Get Transcripts

Questions & Answers

Q: What are the three steps involved in making a metric definition?

The three steps involved in making a metric definition are coming up with a high-level concept for the metric, defining specific details such as measurement criteria, and summarizing individual data measurements into a single metric.

Q: Should companies stick to one metric for evaluation, or is it better to use multiple metrics?

The decision to stick to one metric or use multiple metrics for evaluation depends on company culture and objectives. Some companies prefer a whole suite of metrics to track progress and make informed decisions, while others may settle on a single objective for external reporting purposes.

Q: What are composite metrics, and what challenges do they pose?

Composite metrics are weighted functions that combine multiple individual metrics into a single evaluation criterion. While they can provide an overall understanding, challenges include agreeing on a definition, over-optimization, and the need to analyze individual metrics when variations occur.

Q: Why is it important to consider the general applicability of metrics in AB testing?

Considering the general applicability of metrics in AB testing allows for meaningful comparisons across multiple tests. It is often better to use slightly less optimal metrics that can be applied consistently across tests, minimizing risks associated with custom metrics.

Summary & Key Takeaways

  • The process of creating metric definitions involves coming up with a high-level concept for the metric, defining specific details such as measurement criteria, and summarizing individual data measurements into a single metric.

  • Depending on the company culture and objectives, one can choose to stick to one metric for evaluation or use multiple metrics to gain a comprehensive understanding.

  • Composite metrics, which combine multiple metrics into a single weighted function, can be effective but may face challenges in defining, optimizing, and interpreting the results.

  • The use of generally applicable metrics across AB tests is preferred over custom metrics to minimize risk and enable meaningful comparisons.

Share This Summary 📚

Summarize YouTube Videos and Get Video Transcripts with 1-Click

Download browser extensions on:

Explore More Summaries from Udacity Videos 📚

Summarize YouTube Videos and Get Video Transcripts with 1-Click

Download browser extensions on: