Is Most Published Research Wrong?

TL;DR

Many published scientific studies have false positive results due to p-hacking and reliance on p-values, leading to a lack of reproducibility.

Transcript

In 2011 an article was published in the reputable "Journal of Personality and Social Psychology". It was called "Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect" or, in other words, proof that people can see into the future. The paper reported on nine experiments. In one, participants were show... Read More

Key Insights

• 🔮 People can potentially see into the future, according to a study published in the "Journal of Personality and Social Psychology" in 2011.
• 🎯 Participants in the study were able to accurately predict the presence of images behind curtains, with a higher hit rate for erotic images than neutral or negative ones.
• 🔬 Scientists use p-values to determine the significance of research findings, with a threshold of .05 generally considered significant.
• 📚 The .05 threshold for significance was arbitrarily selected in 1925, leading to concerns about the accuracy of published research.
• 📊 Published research has a high rate of false positives due to factors such as low statistical power, biased research, and p-hacking.
• 🔄 Replication studies have found that a significant number of previous research findings cannot be reproduced.
• 📉 Researchers are starting to address the issues of reproducibility and publication bias through measures like large-scale replication studies and open publication policies.
• 🧪 Despite its flaws, the scientific method remains the most reliable way to uncover the truth, highlighting the importance of rigorous research.

Questions & Answers

Q: How does statistical significance contribute to the reproducibility crisis in scientific research?

Statistical significance, determined by p-values, is often used as a threshold for publication, leading to the overrepresentation of false positive results and a lack of reproducibility. Researchers may engage in p-hacking to manipulate data analysis and increase the likelihood of obtaining significant results.

Q: Why are replication studies important in scientific research?

Replication studies are crucial in verifying the credibility of previous research findings. They help assess the reproducibility of results and identify potential flaws or biases in the original studies. However, these studies face challenges, such as the reluctance of journals to publish replication studies and the difficulty of obtaining statistically significant results in replication attempts.

Q: How does publication bias impact scientific research?

Publication bias refers to the tendency of journals to favor publishing studies with statistically significant results. This bias leads to an overrepresentation of positive findings and a lack of publication of negative or non-significant results. It can distort the overall body of scientific literature and contribute to the reproducibility crisis.

Q: What are some steps being taken to address the reproducibility crisis?

Some scientists are advocating for larger-scale replication studies, open sharing of research data, and pre-registration of hypotheses and methods to combat the reproducibility crisis. The creation of online repositories for negative results and the establishment of sites like Retraction Watch help bring attention to flawed or retracted studies.

Summary & Key Takeaways

• A study published in 2011 claimed to show evidence of people being able to see into the future, but further analysis revealed flaws in statistical significance.

• The prevalence of false positive results in published research is underestimated, with up to a third of published results being wrong.

• Replication studies have shown that many previous findings cannot be reproduced, raising concerns about the credibility of scientific research.

• Incentives for researchers to publish results and publication biases contribute to the problem and hinder the process of scientific self-correction.