Statistical Learning: 8.2 More details on Trees

TL;DR

Regression trees use sequential splitting to divide data into regions and make predictions based on the mean of training observations in each region. Cost complexity pruning helps find the optimal tree size by balancing fit and tree size.

Transcript

at any point once a tree is built you predict the the test observation by passing it down the tree obeying each of the splits it'll end up in a terminal node and then you'll you'll use the mean of the training observations in that region to to make the prediction let's look at a slightly bigger example a cartoon example in the in the next slide fir... Read More

Key Insights

• 🌲 Regression trees predict test observations by following splits and using the mean of training observations in terminal nodes.
• ✋ Tree size is crucial, and building as large as possible overfits the data, while stopping early can result in a suboptimal split.
• 🌲 Cost complexity pruning helps find the optimal tree size by penalizing the number of nodes.
• 😵 Cross-validation is used to estimate the best penalty parameter alpha, which balances fit and tree size.
• 🌲 The pruned tree with the smallest cost complexity criterion is selected as the final model.
• ✋ One approach to stopping tree growth is to have a minimum number of observations in each terminal node.
• 🌲 Training error is not a reliable metric to determine tree size as it always decreases with larger trees.

Questions & Answers

Q: How are test observations predicted in regression trees?

Test observations are predicted by passing them down the tree, following the splits based on the values of their variables. At each terminal node, the prediction is made using the mean of the training observations in that region.

Q: Why is it not advisable to build a tree with one observation in each terminal node?

Building a tree with one observation in each terminal node would result in overfitting the data. While it would have a training error of zero, it would not generalize well to new test data, leading to high prediction error.

Q: What is cost complexity pruning?

Cost complexity pruning is a strategy to find the optimal tree size that balances fit and tree size. It adds a penalty for the number of nodes in the tree, using a penalty parameter alpha. The best tree is selected by minimizing the cost complexity criterion.

Q: How is the penalty parameter alpha determined?

The penalty parameter alpha is determined through cross-validation. The data is divided into parts, and trees of various sizes are fit on the training data while evaluating prediction error on the left-out part. The value of alpha that minimizes the error is chosen.

Summary & Key Takeaways

• Regression trees predict test observations by following a series of splits down the tree, using the mean of training observations in each terminal node to make predictions.

• The size of the tree is crucial, as trees that are too large overfit the data, while trees that are too small have high bias. Cost complexity pruning finds the best tree size by penalizing the number of nodes in the tree.

• Cross-validation is used to estimate the best value of the penalty parameter, alpha, that balances fit and tree size. The sub-tree with the smallest cost complexity criterion is selected as the final pruned tree.