Most Research in Deep Learning is a Total Waste of Time - Jeremy Howard | AI Podcast Clips | Summary and Q&A

TL;DR

Research in deep learning often focuses on minor advances and highly-studied topics, while practical and impactful areas like transfer learning and active learning are under-studied.

Key Insights

• 🧠 The difference between theory and practice in deep learning is that most research focuses on minor advances that have little practical impact, while practical solutions like transfer learning and active learning are under-studied.
• 🚀 Transfer learning and active learning have the potential to revolutionize the field by enabling more people to do world-class work with fewer resources and less data.
• 💡 Companies often innovate in active learning when they have to solve real problems, realizing the inefficiency of manual labeling and the potential for machine assistance in identifying the hardest cases for labeling.
• 📝 The academic world tends to prioritize publishing on familiar topics rather than focusing on practical results, leading to a lack of attention on experimental solutions like transfer learning and active learning.
• 📚 Despite personal disinterest in publishing, practical advancements like the introduction of transfer learning to NLP have gained attention and recognition in top computational linguistics conferences when researchers take the initiative to document their work.
• 🎯 Junior researchers may feel pressured to prioritize safe options for publishing, making incremental improvements on existing topics, due to the importance given to citations and papers in academia.
• ♀️ The speaker, however, prioritizes practical impact over publishing, focusing on real-world problem-solving rather than chasing citations and recognition.
• 🌍 Bridging the gap between theory and practice in deep learning requires a shift in the academic culture to value and encourage practical solutions that have significant real-world impact.

Transcript

so much fast they had students and researchers and the things you teach are pragmatically minded I practically minded freaking figuring out ways how to solve real problems and fast right so from your experience what's the difference between theory and practice of deep learning well most of the research in the deep mining world is a total waste of t... Read More

Questions & Answers

Q: Why is there a disconnect between deep learning theory and practice?

The disconnect can be attributed to the academic system, where researchers are incentivized to publish on topics familiar to their peers rather than focusing on practical and impactful areas like transfer learning and active learning. This results in a lack of emphasis on research that can make a real difference in solving problems.

Q: Are there any practical examples of active learning and transfer learning being used?

Absolutely, active learning and transfer learning are widely used in practical settings. In companies and organizations, when faced with the need to solve real-world problems, individuals often innovate in active learning by finding ways to optimize the labeling process and make it more efficient. Transfer learning has also shown great success in various fields, such as natural language processing, where it has outperformed existing algorithms and set new benchmarks.

Q: What motivates researchers to work on topics that are already highly studied and may not have practical impact?

Researchers are motivated to work on familiar topics by the need for recognition within their academic community. Publish or perish culture often prioritizes incremental advancements on well-studied topics, while practical, impactful research may go overlooked. This can be attributed to the pressure to gain citations and publish papers, which are seen as important for career progression within academia.

Q: How does the academic community perceive practical results in deep learning?

Practical results in deep learning are valued and recognized once they are achieved. However, there may be a lack of emphasis on practicality during the research process itself. Junior researchers or those focused on practical impact may find it challenging to prioritize practical results when the academic system predominantly values incremental advancements on well-established topics.

The disconnect between deep learning theory and practice highlights the need for a shift in the academic culture to encourage more research on practical and impactful areas. This can be accomplished by providing incentives and recognition for researchers who prioritize solving real-world problems and incorporating active learning and transfer learning techniques into their work. By bridging the gap between theory and practice, more significant advancements can be made in the field of deep learning.

Summary

In this video, the speaker discusses the difference between theory and practice in deep learning. They argue that much of the research in deep learning is a waste of time, as scientists are incentivized to work on topics that are already familiar to their peers and not practically useful. The speaker highlights the importance of topics like transfer learning and active learning, which have the potential to make a significant impact but are often overlooked in academic research. They also share their personal experience of successfully introducing transfer learning to natural language processing (NLP) and the challenges of publishing practical results in academia.

Questions & Answers

Q: What is the difference between theory and practice in deep learning?

The speaker argues that most of the research in deep learning is a waste of time. Scientists are driven to work on topics that their peers are already familiar with and can recognize in advance. This leads to minor advances and highly studied topics that have little practical impact.

Q: Why is there a lack of focus on practicability in deep learning research?

The speaker explains that the academic system incentivizes researchers to focus on publishing papers and citations. This discourages them from working on things that are practically useful but may not be trendy or well-known in the scientific community.

Q: Can you give an example of a practical approach that is overlooked in deep learning research?

Transfer learning and active learning are two examples the speaker mentions. Transfer learning has the potential to be a world-changing development, enabling more people to do world-class work with fewer resources and less data. However, very few researchers work on it. Similarly, active learning, which explores getting more out of human input in machine learning, is under-studied due to its lack of popularity.

Q: Is active learning being implemented by companies?

Yes, the speaker confirms that people inside companies, particularly those who need to solve problems practically, do innovate and implement active learning. When faced with the time-consuming and expensive task of manual labeling, professionals start to reinvent active learning by selectively labeling the difficult classes and questioning why the process can't be automated.

Q: What is the speaker's personal experience with publishing research on practical topics?

The speaker mentions that they have only ever written one paper and that it was actually written by their colleague. The paper introduced successful transfer learning to NLP and was published at a top computational linguistics conference. This experience highlighted that once practical results are achieved, people do care, but there might be challenges for junior researchers or those who are not driven by the need for citations and papers.

Q: Why has the speaker not written many papers themselves?

The speaker admits to hating writing papers and not considering them important in their life. They believe that nothing makes citations or papers significant to them personally. However, they recognize that many researchers feel the need to prioritize safe options and make slight improvements on existing topics, which everyone else is already working on, in order to secure their careers.

Takeaways

The video emphasizes that while theory and research are important, a focus on practical applications is necessary in the field of deep learning. Many groundbreaking ideas and concepts are often overlooked because they are not considered trendy or do not align with existing academic norms. Encouraging more practical experimentation and implementation, particularly in areas like transfer learning and active learning, can lead to significant advancements and real-world impact in the field of deep learning.

Summary & Key Takeaways

• Most research in deep learning is focused on topics that are already familiar to peers and have little practical impact.

• Transfer learning and active learning, which have the potential to be game-changers, are not given enough attention in academic research.

• Practical innovation in active learning often occurs within companies and organizations that need to solve real-world problems.