Unit of Diversion Consistency

TL;DR

There are three main considerations when choosing between user ID, cookie, and IP-based diversion: user consistency, visibility of changes, and the desired measurement.

Transcript

Now that we've covered some different way you can divert traffic, how would you actually choose between them? >> So there are really three main considerations. The first was user consistency. If you're using a user ID, then the user gets a consistent experiences as they change devices as long as they're signed in. And so for a certain set of cha... Read More

Key Insights

• 🤘 User ID and cookie-based diversion provide user consistency for visible changes across devices or within sign-in and sign-out boundaries.
• ⚾ IP-based diversion is not very useful for consistency or clean randomization, but may be the only option for certain infrastructure-related changes.
• 💱 The choice of diversion method depends on whether the change is visible to users and what needs to be measured, such as learning effects or adaptations to change over time.
• ⚾ Analyzing IP-based diversion requires careful consideration of potential inconsistencies and finding comparable populations for accurate comparison.
• 🖐️ Traffic diversion methods play a crucial role in conducting experiments and measuring the impact of changes on user experiences and behavior.
• ❓ Understanding the limitations and considerations of different diversion methods is important for accurate analysis and decision-making.
• ⚾ User ID and cookie-based diversion are preferred for changes that are visible to users, while IP-based diversion may be necessary for specific infrastructure-related changes.

Questions & Answers

Q: When should you use a user ID or cookie for traffic diversion?

User IDs or cookies are recommended for user-visible changes, providing consistent experiences across devices or within sign-in and sign-out boundaries. They are also necessary for measuring learning effects or adaptations to change over time.

Q: How does IP-based diversion compare to user ID or cookie-based diversion?

IP-based diversion is generally not very useful due to lack of consistency and clean randomization. However, it may be the only choice for certain changes, such as testing infrastructure impact on latency. Analyzing IP-based diversion requires finding comparable populations between the experiment and control groups.

Q: What are the main considerations when choosing traffic diversion methods?

The three main considerations are user consistency, visibility of changes, and the desired measurement. User ID and cookie-based diversion provide consistency, while IP-based diversion may be necessary for specific changes. The choice also depends on whether the change is visible to users and what needs to be measured.

Q: What are the challenges of IP-based diversion analysis?

IP-based diversion analysis can be challenging due to inconsistencies between the experiment and control groups. For example, modem dialup users may be aggregated into a single IP address, making it difficult to find comparable users in the control group. Analytical post-analysis is often needed to address these challenges.

Key Insights:

• User ID and cookie-based diversion provide user consistency for visible changes across devices or within sign-in and sign-out boundaries.
• IP-based diversion is not very useful for consistency or clean randomization, but may be the only option for certain infrastructure-related changes.
• The choice of diversion method depends on whether the change is visible to users and what needs to be measured, such as learning effects or adaptations to change over time.
• Analyzing IP-based diversion requires careful consideration of potential inconsistencies and finding comparable populations for accurate comparison.
• Traffic diversion methods play a crucial role in conducting experiments and measuring the impact of changes on user experiences and behavior.
• Understanding the limitations and considerations of different diversion methods is important for accurate analysis and decision-making.
• User ID and cookie-based diversion are preferred for changes that are visible to users, while IP-based diversion may be necessary for specific infrastructure-related changes.
• The ultimate choice of diversion method depends on the specific goals, requirements, and constraints of the experiment or analysis.

Summary & Key Takeaways

• User consistency is a key factor when choosing between user ID and cookie-based diversion. User ID provides consistent experiences across different devices, while cookies provide consistency only within the sign-in and sign-out boundary.

• For visible changes, such as layout or sign-in location, cookies or user IDs are preferred. However, for changes that are not visible to users, IP-based diversion may be necessary.

• The choice of diversion method also depends on what needs to be measured. If measuring a learning effect or adaptation to change, a stateful unit of diversion like a cookie or user ID is required.