24. Generalized Linear Models (cont.)

TL;DR

The Fisher scoring algorithm for optimizing log likelihood in a generalized linear model can be implemented using iteratively re-weighted least squares.

Transcript

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu. PHILIPPE RIGOLLET: So I apologize. My voice is not 100%... Read More

Key Insights

• 🏋️ The Fisher scoring algorithm can be implemented using iteratively re-weighted least squares, simplifying the computation process.
• 🏋️ Weighted least squares is a useful method for handling heteroscedasticity and incorporating different levels of variance in the data.
• 👋 Starting with a good initial guess for the parameter beta can improve the efficiency and accuracy of the algorithm.

Questions & Answers

Q: What is the purpose of the Fisher scoring algorithm in a generalized linear model?

The Fisher scoring algorithm is used to optimize the log likelihood in a generalized linear model by iteratively updating the parameter beta using gradient ascent. It helps find the value of beta that maximizes the log likelihood function.

Q: What are the steps involved in the Fisher scoring algorithm?

The Fisher scoring algorithm involves calculating the Fisher information matrix, updating the parameter beta using the gradient of the log likelihood, and iterating this process until convergence. It is an iterative optimization method that uses the derivatives of the log likelihood function to update the parameter.

Q: How does iteratively re-weighted least squares relate to the Fisher scoring algorithm?

Iteratively re-weighted least squares is an alternative method to implement the Fisher scoring algorithm. It involves solving a weighted least squares problem at each iteration, where the weights are calculated based on the gradient and the Fisher information matrix. This allows for a simpler computation of the iterations, as each step consists of solving a weighted least squares problem.

Q: What are the benefits of using iteratively re-weighted least squares in implementing the Fisher scoring algorithm?

Using iteratively re-weighted least squares simplifies the computational process of implementing the Fisher scoring algorithm. It avoids the need to explicitly compute and invert the Fisher information matrix at each iteration, making the algorithm more efficient. Additionally, it allows for solving a weighted least squares problem, which can handle heteroscedasticity and incorporate different levels of variance in the data.

Summary & Key Takeaways

• The Fisher scoring algorithm is used to optimize the log likelihood in a generalized linear model, specifically when the parameter of interest is beta.

• The algorithm iteratively calculates the Fisher information matrix and updates the parameter beta using gradient ascent.

• The Fisher-scoring algorithm can be re-implemented using iteratively re-weighted least squares, where each step involves solving a weighted least squares problem.