Gradient boosting is a way of combining many simple models into one strong one, but unlike a random forest, which builds its trees independently and averages them, boosting builds trees one at a time in sequence. Each new tree focuses on the mistakes the previous trees made, gradually nudging the combined prediction closer to the truth. The “gradient” in the name refers to the mathematical signal, borrowed from gradient descent, that tells each new tree which direction to correct. Done carefully, this produces models that are often more accurate than any other method on structured, tabular data.
The version that turned this idea into an industry standard is XGBoost, short for Extreme Gradient Boosting, introduced by Tianqi Chen and Carlos Guestrin in their 2016 paper “XGBoost: A Scalable Tree Boosting System.” The paper’s contribution was as much engineering as theory: clever handling of missing values, smart use of memory and multiple processor cores, and built-in safeguards against overfitting, all of which let it train fast on large datasets. XGBoost and its later cousins, such as LightGBM and CatBoost, became famous on Kaggle, the data science competition platform, where boosted-tree models won a large share of contests on tabular data.
Why business readers should care: when the task is predicting something from rows-and-columns data, customer churn, credit default, click-through, equipment failure, gradient boosting is usually the highest-accuracy option that still trains cheaply and runs in milliseconds. It quietly powers ranking, pricing, and risk systems at many companies. Teams reach for it precisely because it tends to win on exactly the kind of data businesses already have.
The honest limits mirror those of decision trees generally. Boosting is not the right tool for raw images, audio, or free-form text, where neural networks lead. It has more knobs to tune than a random forest, and an over-eager configuration can overfit. The sequential training is harder to parallelize than independent trees. But for tabular prediction, gradient boosting remains the technique to beat.