Gradient Boosting

• A training algorithm where weak models are trained to iteratively improve the quality (reduce the loss) of a strong model. For example, a weak model could be a linear or small decision tree model. The strong model becomes the sum of all the previously trained weak models.
• In the simplest form of gradient boosting, at each iteration, a weak model is trained to predict the loss gradient of the strong model.