Decision tree splitting criteria, random forests, bagging, boosting, gradient boosting, XGBoost and feature importance.
40 cards · basic cards · AI-written, checked twice. Edit anything.
- What is Gini impurity and what value indicates a pure node?
Show answer
A measure of node purity: Gini = 1 - sum(p_i squared) for each class probability p_i. Value 0 means pure (all samples of one class).- How does entropy-based splitting differ from Gini-based splitting?
Show answer
Entropy (H = -sum(p_i*log(p_i))) penalizes impurity logarithmically, while Gini is quadratic. Entropy often yields deeper trees; both are valid criteria.- What does information gain (IG) measure in a tree split?
Show answer
The reduction in impurity from a split: IG = impurity(parent) - weighted average impurity(children). Higher IG indicates a better split.- Why is gain ratio sometimes preferred over information gain?
Show answer
Information gain biases toward high-cardinality features (many unique values). Gain ratio normalizes by split entropy, favoring more balanced splits.- What is variance reduction splitting and when is it used?
Show answer
A splitting criterion for regression trees; chooses the split that minimizes the weighted sum of variance in child nodes.- Name two common stopping criteria for growing a decision tree.
Show answer
Maximum tree depth and minimum samples per leaf (or minimum samples to split). Both prevent overfitting and ensure node representativeness.- What does pruning accomplish in decision tree learning?
Show answer
Removes branches after tree growth to reduce overfitting. A pruned tree often generalizes better despite lower training accuracy.- How does max_depth control the bias-variance tradeoff?
Show answer
Lower max_depth increases bias, decreases variance. Higher max_depth decreases bias, increases variance and overfitting risk.- What does min_samples_split control?
Show answer
The minimum number of samples required in a node to attempt a split. Prevents tiny nodes from splitting and reduces overfitting.- Explain the min_samples_leaf hyperparameter.
Show answer
The minimum number of samples required in a leaf node. Ensures every terminal node represents sufficient data and prevents noise fitting.- What is bagging (bootstrap aggregating)?
Show answer
A meta-algorithm training multiple models on random bootstrap samples of data, then averaging predictions. Primarily reduces variance.- What is a bootstrap sample?
Show answer
A random sample of size n drawn with replacement from n original data points. Some points appear multiple times; others not at all.- Define out-of-bag (OOB) error in bagging.
Show answer
Error computed on samples not included in a given bootstrap sample. Used to estimate generalization error without a separate test set.- How do random forests differ from bagging?
Show answer
Random forests subsample features at each split in addition to subsampling rows. Feature subsampling decorrelates trees, reducing ensemble variance further.- Why does feature subsampling in random forests matter?
Show answer
It decorrelates individual trees. If one feature dominates, each tree would use it; feature subsampling forces diversity, making the ensemble stronger.
FlashKeepers