24 MultipleComparisons

TL;DR

Multiple comparisons in medical studies can lead to misleading results, and researchers need to be cautious when interpreting statistical significance.

Transcript

it's time to address the concept known as multiple comparisons our goals right now are to show how multiple hypothesis testing is a major problem in medical studies we're going to talk about how you prevent it and how to account for it when it occurs but what is multiple hypothesis testing what exactly are we talking about here well one example com... Read More

Key Insights

• 😷 Multiple hypothesis testing is a problem in medical studies, as it can result in false positive results.
• 🥶 Cold reading in fortune telling demonstrates the concept of multiple comparisons, where numerous guesses are made to identify a hit.
• 🏆 Statistical significance based on p-values has limitations, especially when testing multiple hypotheses.
• ❓ The Bonferroni correction is one method to address multiple comparisons, but it can be overly conservative.
• ❓ Primary outcomes should be defined in advance and given priority over secondary outcomes in interpreting study results.
• 😋 Food frequency questionnaire-based studies are prone to multiple comparisons, raising skepticism about their findings.
• ❓ Researchers should be transparent about their primary outcomes and follow predetermined protocols to ensure reliable results.

Questions & Answers

Q: What is multiple hypothesis testing?

Multiple hypothesis testing refers to the practice of testing multiple hypotheses in a single study, which can increase the chance of obtaining statistically significant results by chance alone.

Q: How does cold reading relate to multiple comparisons?

Cold reading, a technique used by fortune tellers, involves throwing out numerous guesses and statements to see what resonates with the person. Similarly, in multiple comparisons, researchers may test multiple hypotheses and focus on the ones that show statistical significance.

Q: What is the significance of the p-value in medical studies?

The p-value is used to determine statistical significance in medical studies. A threshold of 0.05 is commonly used, meaning that results with a p-value below 0.05 are considered statistically significant.

Q: How does the Bonferroni correction address multiple comparisons?

The Bonferroni correction adjusts the significance threshold to account for multiple comparisons. By dividing the usual threshold (e.g., 0.05) by the number of comparisons, the new threshold becomes more stringent, reducing the chances of false positives.

Q: What are some potential issues with interpreting food frequency questionnaire-based studies?

Food frequency questionnaire-based studies are susceptible to multiple comparisons, as researchers can test numerous associations between different food items and outcomes. This increases the likelihood of finding statistically significant results by chance alone, potentially leading to misleading conclusions.

Summary & Key Takeaways

• Multiple hypothesis testing in medical studies is a problem that can lead to false positive results.

• Cold reading in fortune telling is used as an example to illustrate the concept of multiple comparisons.

• The use of p-values in determining statistical significance has limitations, especially when testing multiple hypotheses.