4.2.11 An Introduction to Trees - Video 6: Cross-Validation

TL;DR

Using cross validation, we can properly select the parameter values for CART models, avoiding overfitting or oversimplification.

Transcript

In CART, the value of minbucket can affect the model's out-of-sample accuracy. As we discussed earlier in the lecture, if minbucket is too small, over-fitting might occur. But if minbucket is too large, the model might be too simple. So how should we set this parameter value? We could select the value that gives the best testing set accuracy, but t... Read More

Key Insights

• 🗯️ Selecting the right parameter value in CART models is crucial for balancing model complexity and accuracy.
• ☠️ K-fold cross validation allows for proper parameter selection by evaluating models on unseen data.
• 😵 The complexity parameter (cp) in R is used instead of minbucket for cross validation in CART models.
• 😃 Lower cp values lead to bigger trees and potential overfitting, while larger cp values result in simpler models.

Questions & Answers

Q: How does the selection of the "minbucket" parameter affect CART model accuracy?

The "minbucket" parameter in CART models helps control model complexity. If it is too small, overfitting may occur, while if it is too large, the model might be too simple.

Q: Why is using the testing set to select the best parameter value not recommended?

The testing set should be used to measure model performance on unseen data. Using it to select the best parameter value would result in implicitly using the testing set to generate the model, which defeats its purpose.

Q: What is K-fold cross validation?

K-fold cross validation involves splitting the training set into k equally sized subsets or folds. Models are built using k-1 folds and predictions are made on the remaining fold (validation set). This process is repeated for each fold.

Q: How is the final parameter value determined in K-fold cross validation?

The accuracy of the model is computed for each candidate parameter value and each fold. The average accuracy over the folds is used to determine the final parameter value that should be selected.

Summary & Key Takeaways

• Setting the "minbucket" parameter in CART models can affect out-of-sample accuracy, with too small or too large values leading to overfitting or oversimplification.

• K-fold cross validation is a method used to select the parameter value. The training set is split into k subsets, and models are built and evaluated on each fold.

• The accuracy of the models for different parameter values is computed and averaged over the folds to determine the final parameter value.