Vladimir Vapnik: Statistical Learning | Lex Fridman Podcast #5

TL;DR

Vladimir Vapnik, co-inventor of support vector machines and VC theory, discusses the nature of learning and intelligence, the limits of current approaches, and the importance of mathematical principles in understanding reality.

Transcript

The following is a conversation with Vladimir Vapnik. He is the co-inventor of support vector machines, support vector clustering, VC theory, and many foundational ideas in statistical learning. He was born in the Soviet Union and worked at the Institute of Control Sciences in Moscow. Then in the United States, he worked at AT&T, NEC Labs, Facebook... Read More

Key Insights

• 🧑‍🏭 The distinction between instrumentalism and realism is important in understanding the nature of reality and the role of unknown factors.
• ❓ Math can reveal the simple underlying principles of reality and provide insights that go beyond human intuition.
• 🥺 Invariants or predicates are crucial in machine learning, as they help decrease training data needs and lead to better understanding and classification.
• 🧑‍🏫 The role of intelligence and the teacher in learning is still not well understood, and more research is needed to uncover the mechanisms behind it.
• ⛔ The focus on imitating human activities in AI may limit progress in understanding intelligence and developing more effective learning methods.

Questions & Answers

Q: What is the difference between instrumentalism and realism?

Instrumentalism focuses on creating theories for prediction, while realism aims to understand the underlying principles of reality.

Q: How does Vapnik view the role of math in his life?

Vapnik believes that math can reveal the simple underlying principles of reality and that looking at mathematical equations can provide insights that go beyond human intuition.

Q: What is the importance of invariants or predicates in machine learning?

Invariants help decrease the amount of training data needed and can lead to better understanding and classification. They allow for the creation of admissible sets of functions with a small VC dimension.

Q: What is the difference between strong and big convergence mechanisms?

Strong convergence focuses on finding the best function within an admissible set, while big convergence uses predicates to narrow down the admissible set itself. Big convergence can lead to more efficient learning with less training data.

Key Insights:

• The distinction between instrumentalism and realism is important in understanding the nature of reality and the role of unknown factors.
• Math can reveal the simple underlying principles of reality and provide insights that go beyond human intuition.
• Invariants or predicates are crucial in machine learning, as they help decrease training data needs and lead to better understanding and classification.
• The role of intelligence and the teacher in learning is still not well understood, and more research is needed to uncover the mechanisms behind it.
• The focus on imitating human activities in AI may limit progress in understanding intelligence and developing more effective learning methods.
• The discovery of statistical learning theory and support vector machines had a profound impact on the field, and Vapnik believes that his recent work on invariants represents a complete understanding of learning.

Summary

In this conversation, Vladimir Vapnik, the co-inventor of support vector machines and support vector clustering, shares his insights on artificial intelligence, learning, and the nature of reality. He talks about the distinction between instrumentalism and realism in understanding the world, the role of mathematics in revealing the underlying principles of reality, and the limitations of deep learning. He emphasizes the importance of invariance and predicates in learning and intelligence and discusses the need to understand how teachers shape learning. Vapnik also touches on the role of intelligence in machine learning and the challenge of generating predicates. Overall, he highlights the value of mathematics and the search for ground truths in gaining a deeper understanding of the world.

Questions & Answers

Q: Does God play dice?

Vapnik suggests that because there are certain factors that we don't know, it may seem like God plays dice. However, he believes that the distinction between instrumentalism (creating theories of prediction) and realism (trying to understand what God did) is relevant in this context.

Q: Can you further explain instrumentalism and realism?

Instrumentalism involves creating theories for prediction, focusing on finding rules and classifications. Realism, on the other hand, aims to understand what God actually did in creating the world. These positions can be seen in the context of mechanical laws and the extent to which these laws are true and universally applicable.

Q: So, are you more aligned with the instrumentalist perspective?

Vapnik explains that from the perspective of a machine learning model, the instrumentalist view is more common. The goal of machine learning is to find rules for classification and prediction. However, he also acknowledges the importance of understanding conditional probability for a deeper comprehension of the world.

Q: How do you see the role of math in your life?

Vapnik views math as a tool for revealing the simple underlying principles of reality. He believes that mathematical structures contain knowledge about reality, and scientists use equations and mathematical models to understand different aspects of the world.

Q: Can math uncover the simple underlying principles of reality?

Vapnik acknowledges that uncovering these principles may be challenging, but once discovered, they often appear beautiful. He mentions the example of the Least Squares Method and how deriving it mathematically from equations revealed a simple but previously overlooked idea about the composition of point observations. He argues that math can uncover more about reality than our own fantasies and limited human intuition.

Q: Are there moments of brilliance in human intuition that can leap ahead of math?

Vapnik doesn't think so. He believes that human intuition can be limited and is best used to put forward axioms as the basis for mathematical reasoning. In his view, human brilliance often comes from following axioms and arriving at historically polished mathematical truths.

Q: Is imagination important in the discovery of scientific ideas like Einstein's theory of relativity?

Vapnik doesn't see imagination as crucial in the discovery of scientific ideas. He suggests that machine learning, for instance, doesn't require imagination but rather relies on clear mathematical equations and principles. In his opinion, mathematical reasoning can lead to profound insights without the need for fantastical or imaginative interpretations.

Q: Can we mathematically describe the learning process in the human brain?

Vapnik argues that describing the learning process in the human brain isn't purely a matter of description but of interpretation. He uses the analogy of Leeuwenhoek wrongly interpreting blood cells as an army fighting each other to highlight that our interpretations may not always correspond to reality. In his view, intelligence is an interpretation, and understanding how a teacher teaches and the role of predicates in learning intelligence are crucial aspects to study.

Q: What makes a good teacher from a mathematical point of view?

While Vapnik admits he doesn't know the specifics, he suggests a good teacher can introduce invariants and predicates for creating admissible sets of functions. These teachers have grounded knowledge of reality and can describe predicates and invariants based on their understanding. These invariants can reduce the number of required observations by a significant amount.

Q: What role does deep learning play in accomplishing learning tasks?

Vapnik criticizes deep learning, considering it more of a fantasy and interpretation than an effective approach. He argues that it often requires vast amounts of training data and fails to create admissible sets of functions efficiently. He believes that deep architectures are unnecessary and that optimal solutions can be found on shallow networks, pointing to the Representer theorem as evidence.

Q: What is intelligence and how can we describe it mathematically?

Vapnik explains that intelligence is not mere description but interpretation and requires understanding how predicates and invariants are generated. He suggests that intelligence may exist both inside and outside of us and emphasizes the need to study how teachers shape and improve intelligence in students. The challenge lies in formulating mathematical descriptions of intelligence, including the mechanisms of creating predicates.

Q: Can machines think?

Vapnik references Alan Turing's work on imitation and suggests that machines can imitate human behavior, but that is not equivalent to thinking. He raises the idea that intelligence may exist beyond human beings and could be connected to a broader network of intelligence but admits that we currently lack understanding in this area.

Q: How do you view the classification of algorithms by worst-case running time and the question of P vs. NP?

Vapnik considers worst-case running time and complexity analysis valuable tools in understanding functions and algorithms. He suggests that the edges and boundaries are particularly interesting because they reveal fundamental principles. Regarding the question of P vs. NP, he emphasizes the importance of understanding worst-case scenarios to gain a comprehensive understanding of functions and algorithms.

Q: How can we decrease the amount of training data needed for learning tasks?

Vapnik discusses the need for creating admissible sets of functions with small VC dimensions, which can lead to significant reductions in required training data. He recommends incorporating invariants and predicates into the learning process, allowing for more effective learning with smaller datasets. This challenge represents a key aspect of intelligence and learning.

Q: What are some open problems in statistical learning and machine learning?

Vapnik suggests that the problem of generating predicates is an open problem in the field. Understanding why certain predicates are more effective than others and how teachers can shape learning requires further research in the field of intelligence. He also presents the challenge of achieving the same level of accuracy with significantly less training data, highlighting the importance of invariance in this process.

Takeaways

Vladimir Vapnik emphasizes the significance of mathematics in revealing the underlying principles of reality and highlights the value of invariance and predicates in learning and intelligence. He challenges the interpretation-rich approach of deep learning and emphasizes the need for mathematics and understanding ground truths. Vapnik's work suggests that improving learning and intelligence requires studying how teachers shape and improve the learning process. The formulation of predicates and the reduction of required training data remain important open problems in statistical and machine learning.

Summary & Key Takeaways

• Vladimir Vapnik emphasizes the distinction between instrumentalism and realism in understanding the nature of reality, stating that the existence of unknown factors may make it seem like "God plays dice."

• He highlights the importance of math in revealing the simple underlying principles of reality and believes that mathematical structures can provide insights beyond human intuition.

• Vapnik discusses the need for predicates or invariants in the learning process, which can help decrease the amount of training data needed and lead to better understanding and classification.

• He calls for a deeper understanding of intelligence and the role of the teacher in learning, suggesting that the current focus on imitating human activities in AI is a limited approach.