Statistical Learning: 13.2 Introduction to Multiple Testing and Family Wise Error Rate  Summary and Q&A
TL;DR
Testing multiple hypotheses can lead to an increased chance of making Type I errors, which can be problematic in situations with a large number of tests.
Questions & Answers
Q: Why does testing multiple hypotheses become more complicated with a large number of tests?
When there are thousands or tens of thousands of hypothesis tests, the chances of making Type I errors increase significantly. The methods used for adjusting in situations with a small number of tests may not work effectively in these cases.
Q: How do pvalues play a role in determining whether to reject a null hypothesis?
If the pvalue falls below a certain threshold, typically set at 1 percent, the null hypothesis is rejected. However, when there are multiple tests, this can lead to a higher chance of false positives.
Q: What is the significance of the thought experiment involving flipping coins?
The thought experiment demonstrates the concept of pvalues and Type I errors. Even with 1024 fair coins, flipping each coin 10 times would result in at least one coin with a pvalue below 0.002, leading to a false rejection of the null hypothesis.
Q: How does the concept of multiple testing relate to reproducibility issues?
Performing a large number of hypothesis tests increases the likelihood of finding false positives. This can lead to exaggerated claims and difficulties in reproducing the results, contributing to reproducibility issues.
Summary & Key Takeaways

Testing multiple hypothesis tests becomes more challenging when dealing with a large number of tests.

Rejecting all null hypotheses with pvalues below a certain threshold can result in a significant number of Type I errors.

Performing a large number of hypothesis tests increases the likelihood of false positives, even with a small pvalue cutoff.