Single Metric: Example

TL;DR

Changing the color and placement of the Start Now button resulted in a statistically significant increase in click-through rates, indicating that the experiment should be launched.

Transcript

Okay. Let's suppose Audacity runs an experiment where they test changing the color and placement of the Start Now button. Their metric is click-through-rate and they divert by cookie. Their practical significance boundary is .01 and they use an alpha of .05 and a beta of 0.2. Now here are the results of the experiment which was run over seven days.... Read More

Key Insights

• 🅰️ Changing the color and placement of the Start Now button increased click-through rates significantly, providing strong evidence for its effectiveness.
• 🏴‍☠️ Analyzing click-through rates follows a Poisson distribution instead of a binomial distribution.
• ❓ Estimating the variance empirically is essential for calculating the confidence interval.
• 🎨 The sign test can provide additional evidence of the experiment's impact and help reinforce the decision to launch.
• ▶️ The experiment's sample size and duration play an important role in ensuring reliable and accurate results.
• 🆘 Confidence intervals and p-values help determine the statistical significance of the experiment's outcomes.

Questions & Answers

Q: What does the experiment measure?

The experiment measures the impact of changing the color and placement of the Start Now button on click-through rates.

Q: How was the variance estimated in the experiment?

The variance was estimated empirically, with a sample size of 10,000 page views per group, resulting in a standard error of 0.0035.

Q: How was the standard error calculated for the experiment?

The standard error for the experiment was calculated using the formula, dividing the empirical standard error by the scaling factor involving the number of page views in both the control and experiment groups.

Q: How was the decision to launch the experiment made?

Based on the confidence interval (0.0020 to 0.0380) not including the practical significance boundary of 0.01 and the statistically significant results of the sign test, it is recommended to launch the experiment.

Key Insights:

• Changing the color and placement of the Start Now button increased click-through rates significantly, providing strong evidence for its effectiveness.
• Analyzing click-through rates follows a Poisson distribution instead of a binomial distribution.
• Estimating the variance empirically is essential for calculating the confidence interval.
• The sign test can provide additional evidence of the experiment's impact and help reinforce the decision to launch.
• The experiment's sample size and duration play an important role in ensuring reliable and accurate results.
• Confidence intervals and p-values help determine the statistical significance of the experiment's outcomes.
• Business strategy and practical significance boundaries should be considered when making recommendations based on experiment results.

Summary & Key Takeaways

• The experiment tested changes to the Start Now button's color and placement, using click-through rate as the metric.

• The distribution of click-through rates follows a Poisson distribution, which requires empirical estimation of the variance.

• The experiment showed a click-through rate difference of 0.0300 with a confidence interval of 0.0020 to 0.0380, supporting the decision to launch the experiment.