Filtering and Segmenting

TL;DR

Learn about the importance of filtering traffic and de-biasing data in order to ensure accurate experimentation results.

Transcript

So, we've seen how important it is to precisely define what data you need to compute your metric. What other issues come up? >> Well, for one thing, you often see, sort of abuse on your site, such as spam or fraud, and you want to try to filter that out. For example, if you have a competitor, who's looking for your site clicking on absolutely every... Read More

Key Insights

• 💻 Spam and fraud on websites can distort data and should be filtered out to ensure accurate results. Competitors or malicious individuals may intentionally skew metrics.
• 📰 A big change in an experiment may attract a lot of traffic, but it's important to identify and potentially filter out this traffic to avoid biased results.
• 🌍 Both external and internal factors can lead to the need for traffic filtering. Changes may only impact specific subsets of traffic, such as language or platform, and filtering can increase the power and sensitivity of the experiment.
• 🎯 The goal of filtering is to de-bias data, but caution must be exercised to not introduce further bias. For example, excluding only logged-in users may skew the data by ignoring noncommittal or new users.
• 🔍 When considering applying filters, slicing the data by different criteria such as country, language, or platform can help determine if traffic is being disproportionately moved or biased in the results. ⏰ Analyzing week over week or day over day data and looking for patterns or anomalies can help identify potential spam or fraud activity.
• 🔒 The process of analyzing data and determining whether to filter traffic or not is about building intuition and understanding expected changes versus unexpected changes.
• 🔎 Building this intuition is important for accurately interpreting data and identifying potential problems or discrepancies in experiments.

Questions & Answers

Q: Why is filtering traffic important in experimentation?

Filtering traffic is crucial in experimentation as it helps to remove unwanted data that can bias the results and ensure accurate analysis and conclusions.

Q: What are some examples of external reasons for filtering traffic?

External reasons for filtering traffic include spam, fraud, and sudden traffic surges, such as when an experiment receives blog coverage or intentional clicks from competitors.

Q: How can filtering traffic affect the power and sensitivity of an experiment?

Filtering only the affected traffic can increase the power and sensitivity of an experiment by focusing the analysis on the specific impact of the change, thereby avoiding dilution of results.

Q: How can filtering traffic potentially introduce bias into the data?

Filtering traffic without careful consideration can introduce bias, especially when certain user groups are excluded, like non-committal or newer users who haven't created an account yet.

Q: What method can be used to assess the impact of filters on different data subsets?

Slicing the data and computing the metric on disjoint sets, such as by country, language, or platform, can help determine if filters disproportionately affect certain subsets, indicating potential bias.

Q: How can week over week or day over day data analysis help identify spam or fraud?

Analyzing traffic patterns over different time intervals can highlight any unusual behavior, such as sudden bursts of requests from a single IP address, which may indicate spam or fraud attempts.

Q: What is the role of intuition in determining whether to filter traffic or not?

Developing intuition by understanding expected and unexpected changes helps in making informed decisions when analyzing the data for an experiment and assessing if any issues or biases exist.

Summary & Key Takeaways

• Filtering traffic is necessary to remove unwanted data such as spam, fraud, and irrelevant traffic that may skew experiment results.

• Internal and external factors, such as specific user subsets or sudden traffic surges, may require filtering to maintain result validity.

• While filtering is important, it should be done cautiously to avoid introducing bias into the data.