Statistical Learning: 6.3 Backward stepwise selection

TL;DR

Backward stepwise selection is an efficient alternative to best subset selection in regression, starting with a model containing all predictors and gradually removing the least useful predictor until a model with no predictors is reached.

Transcript

so that's forward stepwise selection um and now we're briefly going to talk about backward stepwise and backwards stepwise is once again just like forward it's an efficient alternative to best subset selection but it actually is exactly the opposite of forward stepwise so remember in forward stepwise we built the model m0 and then we added a featur... Read More

Key Insights

• ◀️ Backward stepwise selection removes predictors from a model starting with all predictors until reaching a model with no predictors.
• 🙅 It is an efficient computational alternative to best subset selection, considering around p squared over two models.
• 😘 Backward stepwise selection can provide better results on a test set but may not always have the lowest RSS or highest R-squared.
• #️⃣ It requires the number of observations to be greater than the number of predictors.
• 🏆 RSS and R-squared are not suitable for choosing among models with different numbers of predictors, as they are based on training error rather than test error.
• 🛩️ The model containing all predictors always has a smaller RSS and larger R-squared than any other model due to its relation to training error.
• 🏆 Choosing a model based on training error alone does not guarantee low test error, and training error is not a good estimate of test error.

Questions & Answers

Q: What is backward stepwise selection in regression?

Backward stepwise selection starts with a model containing all predictors and removes predictors one at a time until a model with no predictors is reached. It is an alternative to best subset selection and is computationally efficient.

Q: How is backward stepwise selection different from forward stepwise selection?

Backward stepwise selection starts with a model containing all predictors, while forward stepwise selection starts with a model with no predictors. In backward stepwise selection, predictors are removed one at a time, whereas in forward stepwise selection, predictors are added one at a time.

Q: How does backward stepwise selection choose which predictor to remove?

In backward stepwise selection, the predictor that has the least detrimental effect on either RSS or R-squared is removed at each step. The goal is to remove the least useful predictor while minimizing the impact on the model fit.

Q: Can backward stepwise selection guarantee the best model with a specific subset of predictors?

No, backward stepwise selection, like forward stepwise selection, cannot guarantee the best model containing a particular subset of predictors. While it may not always have the lowest RSS or highest R-squared, it can still provide better results on a test set and is more computationally efficient.

Summary & Key Takeaways

• Backward stepwise selection is the opposite of forward stepwise selection, starting with a model containing all predictors and removing one at a time until a model with no predictors is reached.

• It is a computational alternative to best subset selection, considering around p squared over two models instead of two to the power of p models.

• Backward stepwise selection may not give the best model in terms of RSS or R-squared, but it is computationally efficient and can provide better results on a test set.