Complexity Is Not a Feature in Tabular ML
A simple MLP can beat a tabular transformer. That should not be surprising — it should be a reminder. In tabular machine learning, newer architecture does not automatically mean better architecture.
TL;DR: Complexity is not a feature — it is a cost. In tabular machine learning, newer architecture does not automatically mean better architecture. Transformers earn their power where token order, context, or spatial structure are fundamental; tabular data has a different structure, with distinct column meanings, important interactions, high-cardinality categories, and temporal or grouped records. A carefully built MLP — sensible preprocessing, regularisation, residual connections, normalisation, valid evaluation, strong features — can be highly competitive, and gradient-boosted trees should stay in the comparison too. The practical question is not “MLP or transformer?” but “what inductive bias does this dataset need, and can I prove the extra complexity improves the real objective?” A fair comparison holds the splits, point-in-time feature availability, metric, and tuning budget equal, and weighs latency, memory, and maintenance alongside accuracy. Make the simpler baselines fail first, then choose the model that earns its complexity.
The Problem
In tabular machine learning, newer architecture does not automatically mean better architecture. Transformers are powerful because attention can model relationships between input tokens. For text, token order and context are fundamental. For images, spatial structure is fundamental. For tabular data, the structure is different.
Columns have distinct meanings. Some interactions matter a lot. Some categories are high-cardinality. Some records are temporal or grouped. But that does not automatically mean a more complex attention-based model will outperform a well-designed baseline.
The Approach
A carefully built MLP with sensible preprocessing, regularisation, residual connections, normalisation, valid evaluation, and strong features can be highly competitive. And for many tabular problems, gradient-boosted trees should remain in the comparison too.
The practical question is not “should I use an MLP or a transformer?” It is: what inductive bias does this dataset need — and can I prove this extra complexity improves the real objective?
A fair comparison means holding everything else equal
Same train/validation splits, same point-in-time feature availability, same metric, and a comparable tuning budget — with latency, memory, and maintenance considered alongside accuracy.
Without that discipline, a more complex model can look better for the wrong reasons: a lucky split, extra tuning its baseline never received, or metrics that ignore what the deployment actually needs.
Outcome
Complexity is not a feature. It is a cost. Pay for it only when it creates measurable value, and make the simpler alternatives fail first before reaching for the newest tabular architecture.
That framing keeps model selection honest: the burden of proof sits with the more complex candidate, not with the baseline.
Key Takeaway
Design insight: Complexity is not a feature — it is a cost. In tabular machine learning, newer architecture does not automatically mean better architecture. Transformers earn their power where token order, context, or spatial structure are fundamental; tabular data has a different structure, with distinct column meanings, important interactions, high-cardinality categories, and temporal or grouped records. A carefully built MLP — sensible preprocessing, regularisation, residual connections, normalisation, valid evaluation, strong features — can be highly competitive, and gradient-boosted trees should stay in the comparison too. The practical question is not “MLP or transformer?” but “what inductive bias does this dataset need, and can I prove the extra complexity improves the real objective?” A fair comparison holds the splits, point-in-time feature availability, metric, and tuning budget equal, and weighs latency, memory, and maintenance alongside accuracy. Make the simpler baselines fail first, then choose the model that earns its complexity.