The Bitter Lesson: General Methods and the Power of Computation in AI Research

The Bitter Lesson: General Methods and the Power of Computation in AI Research

Introduction

Over the past 70 years of AI research, one lesson stands out as the most significant: general methods that leverage computation are ultimately the most effective. This lesson is often referred to as "The Bitter Lesson," and it provides valuable insights into the evolution of AI.

The Role of Moore's Law

A key factor behind the effectiveness of general methods is Moore's Law, which states that the cost per unit of computation falls exponentially over time. This generalization has allowed for the development of more powerful AI systems, as computation becomes increasingly affordable.

The Pitfall of Human-Knowledge Approach

Early AI research focused on leveraging human knowledge to build intelligent systems. Researchers believed that by incorporating their understanding of how the human mind works, they could create more advanced AI. However, these efforts often proved to be counterproductive.

Enormous initial efforts were made to avoid search and rely on human knowledge or special features of specific domains. However, once search algorithms were effectively applied at scale, these human-centric approaches became irrelevant or even inhibitive to progress.

The Rise of Statistical Methods

In various domains, statistical methods have emerged as superior to human-knowledge-based approaches. For example, in speech recognition, early competitions sponsored by DARPA in the 1970s showcased the advantage of newer statistical methods based on hidden Markov models (HMMs) that relied on more computation.

More recently, deep learning has revolutionized speech recognition by relying even less on human knowledge and utilizing more computation. Deep learning models, combined with extensive training on large datasets, have produced dramatically improved speech recognition systems.

The Evolution of Computer Vision

Similar to speech recognition, computer vision has followed a similar pattern. Early methods attempted to conceptualize vision as searching for edges, generalized cylinders, or using SIFT features. However, modern deep-learning neural networks, which rely on convolution and certain invariances, have significantly outperformed these earlier approaches.

Despite these advancements, the bitter lesson remains incompletely digested by the AI community. Researchers continue to make the same mistakes by trying to build AI systems that mimic their own thought processes. However, as Moore's law provides access to massive computation, it becomes clear that incorporating human knowledge into AI systems is ultimately counterproductive.

The Power of General Purpose Methods

The bitter lesson highlights the great power of general purpose methods that can scale with increased computation. Two methods that exemplify this scalability are search and learning. By leveraging computation and learning from large datasets, AI systems can achieve breakthrough progress.

Human-centric approaches, such as attempting to simplify the complexities of the mind or the external world, hinder progress. Instead, AI agents should focus on discovering knowledge, rather than containing pre-existing knowledge. This shift allows for the development of AI systems that can learn and discover like humans.

Actionable Advice:

• 1. Embrace general purpose methods: Instead of building AI systems based on human-centric approaches, prioritize general purpose methods that leverage computation and learning.

• 2. Invest in computation: Recognize the power of Moore's Law and ensure that AI systems have access to sufficient computation resources to scale effectively.

• 3. Foster a discovery-oriented mindset: Encourage AI researchers to focus on developing systems that can learn and discover like humans, rather than containing pre-existing knowledge.

Conclusion

"The Bitter Lesson" derived from 70 years of AI research emphasizes the superiority of general methods that leverage computation. The evolution of AI in speech recognition and computer vision demonstrates the limitations of human-knowledge-based approaches and the power of statistical methods and deep learning.

By understanding the bitter lesson, AI researchers can harness the scalability of search and learning to achieve breakthrough progress. Embracing general purpose methods, prioritizing computation, and fostering a discovery-oriented mindset are crucial for future advancements in the field of AI.