Better Features Beat More Model Shopping in Tabular ML
Before trying your fifth model, make the table better. For structured data, better representation usually beats model shopping.
TL;DR: Don't make the model more complicated before making the data representation more informative. When performance stalls, ask: "What useful structure is missing from my table?" — not "Which algorithm should I try next?"
The Problem
Jumping between XGBoost/LightGBM/CatBoost and tuning stacks is tempting. But without improving representation, gains are often small and unstable. The question is: what's missing from the table?
The Approach
Start with fundamentals: what does one row represent? How is the target created? Build validation matching the real decision. Form feature hypotheses that express useful structure (ratios/rates, lags/rolling windows, group aggregates, frequency, missingness, interactions, domain signals). Establish a strong baseline, then tune/ensemble only after representation is solid.
Visual Summary
Outcome
A well-represented dataset + strong baseline often outperforms heavy tuning on weak inputs. Caveat: for images/text/foundation models representation is often learned; for time series validation/leakage often matter more. For structured data, improve representation first.
Key Takeaway
Design insight: Don't make the model more complicated before making the data representation more informative. When performance stalls, ask: "What useful structure is missing from my table?" — not "Which algorithm should I try next?"