Artificial Intelligence & Machine Learning 4 - Stochastic Gradient Descent | Stanford CS221 (2021)

TL;DR

Stochastic gradient descent is a faster alternative to gradient descent for optimizing training losses, as it performs updates on individual losses instead of looping over all examples in the training set.

Transcript

hi in this lecture i'm going to talk about sarcastic gradient descent so recall grading ascent which was the optimization algorithm that we decided on for optimizing all our training losses for classification and regression so recall that the training loss is an average over all the examples in the training set of the per example losses so graded d... Read More

Key Insights

• 🐢 Gradient descent is slow because it requires looping over all training examples for each update, while stochastic gradient descent updates after each individual example.
• 👻 Stochastic gradient descent allows for a larger number of updates in the same amount of time.
• 🏋️ The step size in stochastic gradient descent determines the magnitude of the weight vector update.
• 🗯️ Choosing the right step size is a trade-off between speed and stability in the optimization process.
• ❓ Stochastic gradient descent can converge to the optimum with fewer iterations compared to gradient descent.
• 💻 Stochastic gradient descent focuses on optimizing the objective function rather than computing the exact gradient values.
• 🏆 Synthetic data can be used to test and verify the performance of optimization algorithms.

Questions & Answers

Q: What is the difference between gradient descent and stochastic gradient descent?

The main difference is that gradient descent updates the weight vector based on the average gradient of all training examples, while stochastic gradient descent updates the weight vector after each individual example, resulting in faster updates.

Q: Why is gradient descent slow compared to stochastic gradient descent?

Gradient descent requires looping over all training examples to compute a single update, whereas stochastic gradient descent updates after each example, allowing for a larger number of updates in the same amount of time.

Q: What is the role of the step size in stochastic gradient descent?

The step size determines how far to move in the direction of the gradient. A smaller step size leads to more stable but slower updates, while a larger step size allows for faster updates but can be less stable.

Q: How can the step size be set in stochastic gradient descent?

The step size is typically a hyperparameter that needs to be tuned through trial and error. It can be set as a constant value or decrease over time, depending on the desired trade-off between speed and stability.

Summary & Key Takeaways

• Gradient descent is an optimization algorithm that updates the weight vector based on the gradient of the training loss, but it is slow as it requires looping over all training examples for each update.

• Stochastic gradient descent is a faster alternative that updates the weight vector after each individual example, allowing for a larger number of updates in the same amount of time.

• The step size in stochastic gradient descent determines how far to move in the direction of the gradient and can be set as a constant or a decreasing value.