CatBoost Part 2: Building and Using Trees

TL;DR

In this StatQuest video, Josh Starmer explains how CatBoost creates trees and combines them to make predictions using techniques like Ordered Target Encoding and symmetric decision trees.

Transcript

Going to build a tree, one row at a time! StatQuest! Hello, I'm Josh Starmer and welcome to StatQuest. Today we're going to talk about CatBoost Part 2, Building and Using Trees. If you want to do it in the cloud, then do it with Lightning. It'll be easier and you'll thank me later. Bam! This StatQuest is also brought to you by the letters 'A', 'B',... Read More

Key Insights

• 🪈 CatBoost applies Ordered Target Encoding to prevent Leakage and ensure accurate predictions.
• 🤨 Randomizing rows and applying Ordered Target Encoding are crucial steps in CatBoost's tree-building process.
• 🦮 The cosine similarity is used to evaluate the performance of thresholds and guide the tree-building process.
• 🐎 Symmetric decision trees, although less accurate, offer speed advantages in making predictions.
• ❓ CatBoost is a powerful gradient boosting method that incorporates these unique techniques for improved performance.
• 😒 The use of learning rate and multiple trees helps CatBoost make predictions iteratively.
• 🍵 CatBoost's process can be scaled to handle larger datasets and more complex features.

Questions & Answers

Q: How does CatBoost avoid Leakage when using Ordered Target Encoding?

CatBoost treats the data as if it arrives sequentially, avoiding any effect of a row's target value on its own encoding.

Q: How does CatBoost determine the best threshold for each feature when building trees?

CatBoost compares the cosine similarity of different thresholds to decide which threshold to use for each split in the tree.

Q: Why does CatBoost use symmetric decision trees?

Symmetric decision trees, although weaker for predictions, offer faster computation as they can ask all questions in a single vector operation.

Summary & Key Takeaways

• CatBoost randomizes the rows of the training dataset and applies Ordered Target Encoding to the discrete columns with more than two options.

• The encoding replaces continuous values for the target variable into discrete bins, enabling the use of Ordered Target Encoding.

• CatBoost builds trees by finding the best thresholds for each feature and calculates the cosine similarity to evaluate the predictions.