Sensitivity and Robustness

TL;DR

When choosing a summary metric, consider its sensitivity to changes you care about and its robustness against changes you don't care about.

Transcript

You mentioned that to choose a summary metric, we need to think about the sensitivity and the robustness of the metric. Can you explain that more? >> Yeah, so in lesson one, we talked about the sensitivity of a test to a change that you care about. The sensitivity of a metric is kind of the same thing. The idea is that you want to choose a metric t... Read More

Key Insights

• 💱 Sensitivity ensures that a metric detects meaningful changes, while robustness ensures it remains stable during periods of no change.
• 👤 The mean can be sensitive but not robust, as it is influenced by outliers. The median is robust but may not capture changes affecting a small fraction of users.
• ❓ Alternative statistics like percentiles can provide a better reflection of the measurement you're interested in.
• 🧑‍💻 Sensitivity and robustness can be measured through experiments, A vs A experiments, or retrospective analysis of logs.

Questions & Answers

Q: How do sensitivity and robustness impact the choice of summary metric?

Sensitivity is crucial to ensure that a metric detects changes that matter to you. Robustness, on the other hand, ensures that the metric remains stable when there are no significant changes, avoiding false alarms.

Q: Can you provide an example illustrating the differences between using the mean and the median?

In the example of measuring video load time, the mean can be influenced by outliers, such as long load times. This makes it less robust. The median, however, is less influenced by outliers but may not move if only a fraction of users experience changes.

Q: How can sensitivity and robustness be measured?

Running experiments is one way to measure sensitivity and robustness. Increase a factor that should affect the metric, like video quality, and observe if the metric responds as expected. A vs A experiments can also reveal if the metric is too sensitive. Retrospective analysis of past experiments or changes made to a website can provide insights into metric movements.

Q: What can be done if there is no experiment data available?

If experiment data is unavailable, a retrospective analysis of logs can be conducted. Reviewing the history of metric changes and correlating them with known changes made to the site can offer valuable insights into sensitivity and robustness.

Key Insights:

• Sensitivity ensures that a metric detects meaningful changes, while robustness ensures it remains stable during periods of no change.
• The mean can be sensitive but not robust, as it is influenced by outliers. The median is robust but may not capture changes affecting a small fraction of users.
• Alternative statistics like percentiles can provide a better reflection of the measurement you're interested in.
• Sensitivity and robustness can be measured through experiments, A vs A experiments, or retrospective analysis of logs.
• It is important to avoid false alarms by choosing a metric that strikes a balance between sensitivity and robustness.

Summary & Key Takeaways

• Sensitivity refers to how well a metric detects changes you care about, while robustness means the metric remains stable when nothing important is happening.

• Metrics like the mean can be sensitive to outliers, while the median is more robust but may not capture changes if they only affect a small fraction of users.

• Alternative statistics like the 90th or 99th percentile can provide a better understanding of the measurement you're trying to achieve.