Bagging, Random Forests and Gradient Boosting

Choose the ensemble mechanism, not the brand name

You will compare a random forest with histogram gradient boosting and inspect validation behaviour, feature evidence and operational trade-offs.

• Explain bootstrap aggregation and random feature selection.
• Distinguish bagging’s parallel variance reduction from boosting’s sequential correction.
• Describe AdaBoost, gradient boosting, XGBoost and LightGBM at an appropriate conceptual level.
• Use early stopping, depth and learning rate to manage boosting complexity.

Diversity makes aggregation useful

Bagging trains base models on bootstrap samples and averages or votes. Deep decision trees have high variance, so their errors can cancel when the trees differ. Random forests add diversity by considering a random subset of features at each split.

AdaBoost increases attention to records that earlier learners misclassified and combines learners with weights. It can be sensitive to noisy labels and outliers because persistent mistakes attract attention. Gradient boosting instead fits each new learner to the current loss gradient, which generalizes to many differentiable objectives.

XGBoost and LightGBM are production-oriented gradient-boosting libraries with efficient tree construction, regularization and system features. Their exact options evolve. Use their official documentation and pin tested versions. Scikit-learn’s HistGradientBoosting offers a convenient built-in implementation for this course.

A smaller learning rate with more boosting iterations can improve generalization at added compute cost. Shallow trees limit interaction complexity. Early stopping uses held-out training data to stop when improvement ends. Keep the final test outside that decision.

Tree ensembles capture nonlinearities and interactions without manual expansion. They still inherit data leakage, biased labels and invalid splits. Feature importance is not a causal explanation. Compare held-out permutation importance and inspect errors.

Compare two tree ensembles

The example uses fixed cross-validation. Add tuning only inside the training process after this baseline.

Install the lesson dependencies

python -m pip install scikit-learn

Measure performance and variability

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
from sklearn.model_selection import StratifiedKFold, cross_validate

X, y = load_breast_cancer(return_X_y=True)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
models = {
    "forest": RandomForestClassifier(
        n_estimators=300, min_samples_leaf=3,
        class_weight="balanced", random_state=42, n_jobs=-1
    ),
    "hist_gradient_boosting": HistGradientBoostingClassifier(
        learning_rate=0.05, max_iter=200, max_leaf_nodes=15,
        l2_regularization=1.0, random_state=42
    ),
}

for name, model in models.items():
    result = cross_validate(model, X, y, cv=cv,
                            scoring="balanced_accuracy")
    print(name,
          "mean=", round(result["test_score"].mean(), 3),
          "min=", round(result["test_score"].min(), 3),
          "max=", round(result["test_score"].max(), 3))

Expected comparison format

forest mean= <score> min= <score> max= <score>
hist_gradient_boosting mean= <score> min= <score> max= <score>

Do not choose from the mean alone. Inspect the weakest fold, error costs, latency and the stability of the decision threshold.

Ensemble failure patterns

More trees can stabilize an estimate, but they cannot correct a broken target or evaluation.

• Out-of-bag estimates do not replace a time-aware or group-aware final evaluation.
• Unlimited leaves can fit tiny, unstable segments even inside a forest.
• Boosting noisy labels can spend capacity fitting annotation errors.
• Default probability output may be poorly calibrated for high-stakes thresholds.

Run a controlled ensemble experiment

Compare a single pruned tree, random forest and one boosting model under identical folds.

• Limit the tuning grid before viewing results.
• Plot validation score against tree depth or leaf count.
• Measure prediction latency on a realistic batch size.
• Use held-out permutation importance and inspect one feature’s failure cases.

Experiment evidence

• Declared search space.
• Fold and latency table.
• Model choice with one reason not to deploy the highest score.

Official references and further reading

• scikit-learn ensemble methods (Official bagging, forest and boosting guidance)
• XGBoost documentation (Primary project documentation for XGBoost)
• LightGBM documentation (Primary project documentation for LightGBM)

Review note for Bagging, Random Forests and Gradient Boosting: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.