GBDTs Are Not Dead: Strong Tabular Baselines
Every few years, someone declares GBDTs dead — and every few years, they remain among the strongest baselines for structured tabular data. XGBoost, LightGBM, and CatBoost are not exciting because they are new. They are useful because they keep working.
TL;DR: GBDTs are not dead. They are still one of the first serious baselines worth beating. Every few years someone declares them outdated, and every few years XGBoost, LightGBM, and CatBoost remain among the strongest options for structured tabular data — not because they are new, but because they keep working. For many real tabular problems they combine strong performance with limited data, efficient training and inference, practical handling of nonlinearities and interactions, mature tooling, straightforward deployment, and useful compatibility with careful feature engineering. Deep learning still has a place: neural models can add useful diversity to an ensemble and shine when the problem includes text, images, sequences, graphs, or large-scale representation learning. But newer does not automatically mean better for a given dataset. The strongest workflow for structured business data stays familiar: define the prediction problem carefully, build features that reflect the process, validate realistically, establish a strong GBDT baseline, and add complexity only when held-out evidence justifies it. Do not retire a strong baseline because it is no longer new.
The Problem
Every few years, someone declares GBDTs dead. And every few years, they remain among the strongest baselines for structured tabular data. XGBoost, LightGBM, and CatBoost are not exciting because they are new — they are useful because they keep working.
That does not mean deep learning has no place in tabular ML. Neural models can add useful diversity to an ensemble, and they can be valuable when the problem includes text, images, sequences, graphs, or large-scale representation learning. But “newer” does not automatically mean “better for this dataset.”
The Approach
For many real tabular problems, gradient-boosted trees offer a difficult combination to beat. The most reliable way to use them is still a familiar workflow:
1. Define the prediction problem carefully
Fix the prediction moment, the target, and the decision the output will drive before choosing any model.
2. Build features that reflect the business process
Features that encode how the process actually works usually decide the outcome more than the algorithm does.
3. Validate realistically
Match the validation design to the real deployment decision: the right splits, the right horizon, point-in-time feature availability.
4. Establish a strong GBDT baseline
XGBoost, LightGBM, or CatBoost give strong performance with limited data, efficient training and inference, practical handling of nonlinearities and interactions, mature tooling, and straightforward deployment.
5. Add complexity only when held-out evidence justifies it
The burden of proof sits with the more complex candidate — not with the baseline it is trying to beat.
Outcome
The point is not to defend old tools. It is to avoid replacing a strong baseline because social media made it sound outdated. For structured business data, a carefully validated GBDT baseline remains one of the first serious results worth beating.
Keeping that baseline in place also gives every later experiment a fair reference point, so added complexity has to prove its value against something real.
Key Takeaway
Design insight: GBDTs are not dead. They are still one of the first serious baselines worth beating. Every few years someone declares them outdated, and every few years XGBoost, LightGBM, and CatBoost remain among the strongest options for structured tabular data — not because they are new, but because they keep working. For many real tabular problems they combine strong performance with limited data, efficient training and inference, practical handling of nonlinearities and interactions, mature tooling, straightforward deployment, and useful compatibility with careful feature engineering. Deep learning still has a place: neural models can add useful diversity to an ensemble and shine when the problem includes text, images, sequences, graphs, or large-scale representation learning. But newer does not automatically mean better for a given dataset. The strongest workflow for structured business data stays familiar: define the prediction problem carefully, build features that reflect the process, validate realistically, establish a strong GBDT baseline, and add complexity only when held-out evidence justifies it. Do not retire a strong baseline because it is no longer new.