High-Cardinality Categoricals: One-Hot, Target Encoding, or Embeddings?
With thousands of categories, encoding is a modelling decision — one-hot, fold-safe target encoding, or entity embeddings — validated against a strong baseline.
TL;DR: Every encoding is a modelling assumption, not a default. One-hot is a sensible choice at manageable cardinality; fold-safe target encoding adds compact target-informed signal; entity embeddings can capture recurring category behaviour when a neural model is justified — but may simply memorise IDs. Fit encoders inside training folds, define an unseen-category fallback, and keep only the representation that generalises against a strong baseline.
The Problem
High-cardinality categoricals — product IDs, merchant IDs, ZIP codes, customer segments — look routine until there are thousands of unique values. Then the choice of representation becomes a real modelling decision: one-hot encoding explodes into a wide, sparse matrix; naive target encoding silently leaks target information; and entity embeddings only help when categories repeat enough to generalise.
The Approach
The three common approaches solve different problems, so I evaluate them as three distinct hypotheses rather than one default habit.
One-hot encoding
One category becomes one column: simple, transparent, and often still sensible at manageable cardinality. But with thousands of categories it produces a very wide, sparse feature matrix that is expensive in memory, training time, and operational complexity.
Target encoding
Replace a category with a smoothed statistic derived from the target. It is compact and often effective, but it must be fitted safely inside training folds — otherwise validation rows indirectly reveal their own target values. Rare categories need careful smoothing and an unseen-category policy.
Entity embeddings
A neural model learns a dense, lower-dimensional vector per category from the training objective. Categories with similar predictive behaviour can land closer together — but only when there are enough repeated observations per category and the embedding generalises instead of memorising IDs.
My rule: use one-hot when cardinality is manageable; use fold-safe target encoding when compact, target-informed features help; test embeddings when repeated category behaviour and neural modelling justify them.
Visual Summary
Outcome
Treating encoding as an explicit decision produced more reliable models than picking a default. The risk is not only compression — it is reliable generalisation: how the representation behaves on rare and unseen categories in production. Choosing per problem, and validating the choice on the same splits and metric as a strong baseline, kept only the encodings that actually generalised.
Key Takeaway
Design insight: Every encoding is a modelling assumption, not a default. One-hot is a sensible choice at manageable cardinality; fold-safe target encoding adds compact target-informed signal; entity embeddings can capture recurring category behaviour when a neural model is justified — but may simply memorise IDs. Fit encoders inside training folds, define an unseen-category fallback, and keep only the representation that generalises against a strong baseline.