Encoding is a modeling decision, not a preprocessing checkbox
Every encoding imposes an assumption. Better tabular modeling starts with representing variables in ways that reflect their actual structure.
TL;DR: Encoding is not just a preprocessing step. It is part of how the model is allowed to interpret the variable. Better tabular modeling often starts with representing variables in ways that reflect their actual structure.
The Problem
One thing I still see oversimplified in tabular ML is categorical encoding. Too often, the question becomes: "Which encoding is best?" But that is usually the wrong question.
A better question is: What structure does this variable actually have, and which encoding respects that?
The Approach
day_of_week → Helmert Sequential
Each day vs. average of all previous days
RUType → Polynomial Ordered
Hardware generations, spacing non-uniform
region_group → Backward difference
Adjacent categories more similar than distant ones
tech_grade → M-estimator
Noisy grouping, needs Bayesian smoothing toward global mean
site_type → WoE
Binary split indicator, preserves log-odds direction
city_band → Target encoding
High cardinality, direct mapping to conditional mean
The Assumptions
Every encoding imposes an assumption:
- One-hot assumes no ordering
- Ordinal imposes rank
- WoE imposes directional contrast
- Shrinkage encoders assume partial pooling toward a global signal
Key Takeaway
Design insight: Encoding is not just a preprocessing step. It is part of how the model is allowed to interpret the variable. Better tabular modeling often starts with representing variables in ways that reflect their actual structure.
Related
Polars Lazy Execution: More Than a Faster pandas → Delegate Optuna Boilerplate to an AI Coding Assistant →TabPFN: a Pre-Trained Prior for Tabular ML →Cyclical Encoding: Stop One-Hot Encoding Months →Categorical to Behavioral Signal →isnull() Is a Feature →Don't Ask Your Model to Learn What a Formula Already Knows →Categorical Encoding Cheat Sheet →SHAP Waterfall Local Explanation →Focal Loss for Imbalanced Classification →MLforecast Forecasting Pipeline →Multimodal Late Fusion →AI Can Review Feature Code. It Cannot Approve It →
Comments
FAQ
What is the key takeaway from "Encoding is a modeling decision, not a preprocessing checkbox"?
Encoding is not just a preprocessing step. It is part of how the model is allowed to interpret the variable. Better tabular modeling often starts with representing variables in ways that reflect their actual structure.
Who wrote this and what is it about?
This was written by Mahmoud Trigui, Senior Data Scientist. Categorical encoding should respect the variable's structure. Different variables need different encodings: Helmert, WoE, Target, Polynomial, M-estimator.