Model Understanding >

GBM

GBM (Gradient Boosting Machine) is an ensemble of weak learners like decision trees. By adding trees one by one, while correcting the errors of the previous, it achieves high performance.

Success of GBM

GBM is an exceptionally effective model for both tabular data prediction and time series forecasting. Despite the advent of numerous deep learning models, GBM remains a top performer. The Kaggle 2023 AI report highlights this, noting the "continuing dominance of gradient boosted trees" and that "tabular data ... remains largely unaffected by the deep learning revolution".

Predefined hyperparameter values

We use LightGBM, LGBMRegressor to be more specific. 'gbdt' (Gradient Boosting Decision Tree), early stopping, regression, and rmse were used.

GBM for Air Passengers

To demonstrate model performance, we show the model's prediction results for the air passengers dataset. The cross validation process identified the best transformation to make the time series stationary and the optimal hyperparameters. The Root Mean Squared Error on the next day's closing price was used to determine the best model.

In the chart, we display the model's predictions for last split of cross validation and test data.

  1. train: Training data of the last split.
  2. validation: Validation data of the last split.
  3. prediction (train, validation): Prdiction for train and validation data period. For each row (or a sliding window) of data, predictions are made for n days into the future (where n is set to 1, 2, 7). The predictions are then combined into a single series of dots. Since the accuracy of predictions decreases for large n, we see some hiccups in the predictions. The predictions from the tail of the train spills into the validation period as that's future from the 'train' data period viewpoint. These are somewhat peculiar settings, but it works well in testing if the model's predictions are good enough.
  4. test(input): Test input data.
  5. test(actual): Test actual data.
  6. prediction(test): The model's prediction given the test input. There's only one prediction from the last row (or the last sliding window) of the test input which corresponds to 1, 2, 7 days later after 'test(input)'.

GBM model predicts the increases percentage of air passengers as described in capturing trends. Hyperparameters such as max_depth, min_data_in_leaf, n_estimators, colsample_bytreem, etc. were tuned by grid search.