19. Saddle Points Continued, Maxmin Principle

TL;DR

A lecture on saddle points, statistics, and covariance, with a focus on deep learning and the concepts of mean, variance, and covariance.

Transcript

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu. GILBERT STRANG: So I've got a list of things I'm hoping... Read More

Key Insights

• 😥 Saddle points are points in a function where the first derivatives are zero, but the second derivatives are not all zero.
• ❓ The Rayleigh quotient is an important function in deep learning, and its maximum and minimum values correspond to eigenvalues and eigenvectors of a matrix.
• 😫 The maximum value of the Rayleigh quotient in the example is 5, achieved by setting a specific vector.
• 😫 The minimum value of the Rayleigh quotient in the example is 1, achieved by setting a different vector.
• ❓ Covariance is a measure of the relationship between two variables or experiments.
• ❎ The variance is the expected value of the squared distance from the mean, and the covariance is the expected value of the product of the deviations of two variables or experiments.

Questions & Answers

Q: What are saddle points and why are they important in deep learning?

Saddle points are points in a function where the derivatives are zero, but the second derivatives are not all zero. They are important in deep learning because they affect the gradient descent algorithm used to find the minimum of a cost function.

Q: What is the maximum value of the Rayleigh quotient and how can it be achieved?

The maximum value of the Rayleigh quotient in the example given is 5. It can be achieved by setting the vector (u, v, w) to (1, 0, 0).

Q: What is the minimum value of the Rayleigh quotient and how can it be achieved?

The minimum value of the Rayleigh quotient in the example given is 1. It can be achieved by setting the vector (u, v, w) to (0, 0, 1).

Q: What is the relationship between eigenvalues and the Rayleigh quotient?

In the example, the eigenvalues of the matrix used in the Rayleigh quotient function correspond to the maximum, minimum, and saddle values. The eigenvectors of the matrix represent the locations where these values are reached.

Summary & Key Takeaways

• The lecture begins with a discussion on saddle points and their relevance in deep learning, specifically in finding the minimum of a total cost function using gradient descent.

• The topic then moves on to basic ideas of statistics, such as mean and variance, and how they are used in analyzing data.

• The concept of covariance is introduced, along with its matrix representation, and its importance in understanding the relationship between multiple experiments or variables.