Statistical Learning: 6.4 Estimating test error

TL;DR

Different approaches, such as adjusting training error and directly estimating test error, can be used to estimate test error and select models with different numbers of variables.

Transcript

so what i need is actually a way to estimate the test error for each of these models m0 m1 m2 all the way through mp so that i can choose among them and basically in order to estimate the test error i have two approaches and one approach is that i can indirectly estimate the test error by somehow computing the training error and then adjusting it a... Read More

Key Insights

• 😒 Indirect estimation of test error involves adjusting training error for bias due to overfitting, while direct estimation uses cross-validation or a validation set.
• 🏆 Cp, AIC, BIC, and adjusted R-squared are methods that adjust training error to estimate test error and assist in model selection.
• 🛩️ Cp, BIC, and AIC are similar, but BIC tends to choose smaller models when the number of observations is greater than seven.
• ❎ Adjusted R-squared is a useful measure as it adjusts the classical R-squared for models with different numbers of predictors.
• 😵 Cross-validation and validation set approaches are more versatile and can be applied to various model types.
• 😵 Cross-validation does not require knowledge of the number of predictors, making it suitable for models such as ridge regression and lasso.

Questions & Answers

Q: What are the two approaches to estimate test error?

The first approach is to indirectly estimate test error by adjusting the training error for bias due to overfitting. The second approach is to directly estimate test error using cross-validation or a validation set.

Q: What are Cp, AIC, BIC, and adjusted R-squared used for?

These methods adjust the training error to estimate test error and can be used to select models with different numbers of variables.

Q: How do Cp, BIC, and AIC differ?

Cp, BIC, and AIC are almost identical, but BIC tends to choose smaller models when the number of observations is greater than seven due to the choice of log(n) instead of 2.

Q: What is the advantage of adjusted R-squared?

Adjusted R-squared adjusts the classical R-squared to make it comparable for models with different numbers of predictors. It is easy to understand and does not require an estimate of sigma squared.

Summary & Key Takeaways

• Two approaches to estimate test error: indirect estimation by adjusting training error for overfitting bias, and direct estimation using cross-validation or validation set.

• Cp, AIC, BIC, and adjusted R-squared are methods that adjust training error to estimate test error and can be used to select models with different numbers of variables.

• Cp, BIC, and AIC are almost identical, with BIC tending to choose smaller models when the number of observations is greater than seven.

• Adjusted R-squared adjusts the classical R-squared to make it comparable for models with different numbers of predictors.