I am able to successfully improve the performance of my XGBoost model through Bayesian optimization, but the best I can achieve through Bayesian optimization when using Light GBM (my preferred choice) is worse than what I was able to achieve by using it’s default hyper-parameters and following the standard early stopping approach.
When tuning via Bayesian optimization, I have been sure to include the algorithm’s default hyper-parameters in the search surface, for reference purposes.
The code below shows the RMSE …