Stanford ENGR108: Introduction to Applied Linear Algebra  2020  Lecture 54VMLS aug Lagragian mthd  Summary and Q&A
TL;DR
The augmented lagrangian method is a modification of the penalty method that helps solve optimization problems without the need for an infinitely large penalty parameter. It is particularly useful in controlling car motion.
Questions & Answers
Q: What is the difference between the augmented lagrangian method and the penalty method?
The augmented lagrangian method is a modification of the penalty method that addresses the rapid increase of the penalty parameter. It eliminates the need for an infinitely large parameter by adding an additional term to the lagrangian function, making it equivalent to the original problem for feasible solutions.
Q: How does the augmented lagrangian method handle the nonlinear least squares subproblem?
The augmented lagrangian method updates a separate vector, called z, in a different way to avoid making the nonlinear least squares subproblem harder to solve. By cleverly choosing this z term, the method ensures that the first optimality condition always holds, while also addressing the problem of infeasibility.
Q: How does the augmented lagrangian method update the penalty parameter (mu)?
Unlike the penalty method, the augmented lagrangian method does not increase mu every time. Mu is increased only if there has been no substantial progress in reducing the infeasibility. The method measures progress by dividing the infeasibility residual by a factor of four. This slower increase of mu makes the augmented lagrangian method more efficient.
Q: Is the augmented lagrangian method commonly used for solving optimization problems?
Yes, the augmented lagrangian method is a standard optimization approach, although most people use software packages that implement the method rather than implementing it from scratch. Understanding the inner workings of such algorithms can be beneficial for understanding optimization concepts.
Summary & Key Takeaways

The augmented lagrangian method is an improvement on the penalty method, allowing for good solutions without the need for a large penalty parameter.

The penalty method's drawback is that the penalty parameter increases rapidly, making the problem harder to solve.

The augmented lagrangian method adds an additional term to the lagrangian function, which helps make the problem equivalent to the original problem for feasible solutions.