P-Hacking: Crash Course Statistics #30

TL;DR

P-hacking means choosing or manipulating analyses to produce statistically significant p-values rather than following the best analysis plan. With 20 independent tests at a 0.05 cutoff, the chance that at least one appears significant by chance rises to about 64%, illustrating the Family Wise Error rate. Read on to understand why p-hacking happens, how multiple comparisons mislead, and how transparent reporting and the Bonferroni correction can reduce false positives.

Transcript

Hi, I’m Adriene Hill, and welcome back to Crash Course Statistics. Lies. Damn lies. And statistics Stats gets a bad rap. And sometimes it makes sense why. We’ve talked a lot about how p-values let us know something significant in our data--but those p-values and the data behind them can be manipulated. Hacked. P hacked. P-hacking is manipulating da... Read More

Key Insights

• P-hacking involves manipulating statistical analyses to produce significant p-values, potentially misleading scientific conclusions.
• The Null Hypothesis Significance Testing (NHST) framework can lead to misinterpretations if not properly understood or applied.
• Researchers are often pressured to produce significant results due to academic and professional incentives, leading to p-hacking.
• P-hacking can occur unintentionally due to gaps in statistical knowledge or honest mistakes by researchers.
• Multiple statistical tests increase the likelihood of false positives, known as the Family Wise Error rate.
• The Bonferroni correction is a method to adjust p-values to account for multiple comparisons, reducing the risk of false positives.
• P-hacking can have significant real-world implications, influencing public perception and policy decisions based on flawed research.
• Transparency in reporting all conducted tests, not just significant ones, is crucial to avoid misleading conclusions.

Questions & Answers

Q: What is p-hacking, and why is it a problem?

P-hacking is choosing or manipulating data analyses based on which approach produces a significant p-value rather than which analysis plan is best. It can create apparently significant findings when no real effect exists, misleading both researchers and the public.

Q: Why are researchers incentivized to find statistically significant results?

Academic journals and popular websites are less interested in results that lack convincing evidence of an effect. The transcript also says publishing can bring researchers job stability, a higher salary, and prestige, creating pressure to produce positive findings.

Q: Is p-hacking always intentional or malicious?

No. It can arise from limited statistical knowledge, a well-intentioned commitment to a scientific theory, or an honest mistake, although deliberate manipulation can also occur.

Q: How does Null Hypothesis Significance Testing relate to p-hacking?

In the NHST framework, researchers either reject or fail to reject the null hypothesis, which states that there is no effect. Because failing to reject provides a lack of evidence rather than evidence that nothing happened, researchers may be tempted to keep changing analyses until they obtain a significant result.

Q: How do multiple statistical tests increase the chance of a false positive?

Each test in the jelly-bean example has a 5% chance of appearing significant when the null hypothesis is true. Across 20 tests, the probability that none is significant is about 36%, so the chance that one or more appears significant purely by chance is about 64%.

Q: What is the Family Wise Error rate?

The Family Wise Error rate is the inflated Type I error rate that arises when multiple related tests or follow-up comparisons are performed. It means the chance of obtaining at least one false significant result can be much higher than the 5% rate associated with an individual test.

Q: What does the jelly-bean and acne example demonstrate?

An overall jelly-bean test produced a p-value greater than 0.05, but testing 20 colors separately produced a significant result for green jelly beans. Because one or more significant outcomes would occur by chance about 64% of the time in this setup, the green result is not reliable evidence by itself.

Q: How can researchers reduce the risk of p-hacking?

Researchers should choose their analyses before seeing the data and report all tests rather than only significant ones. The existing page summary also identifies the Bonferroni correction as a way to adjust for multiple comparisons and reduce false positives.

Summary & Key Takeaways

• P-hacking refers to the manipulation of data analysis to achieve significant p-values, often misleading scientific research. It can be intentional or due to lack of statistical knowledge. The practice is driven by the pressure on researchers to publish significant findings.

• The Null Hypothesis Significance Testing (NHST) framework is susceptible to p-hacking, especially when multiple tests are performed. This increases the Family Wise Error rate, leading to a higher chance of false positives. Adjustments like the Bonferroni correction can mitigate this risk.

• P-hacking has real-world implications, affecting public understanding and policy based on flawed research. Transparency in reporting all tests is essential to prevent misleading conclusions. The video emphasizes the importance of robust statistical practices in research.