10 years of DS thinking,
one post at a time

A SHAP summary plot shows which features tend to matter across a dataset. But the business question is often local: why did the model flag this customer?

Global SHAP (mean |SHAP|) helps answer what tends to matter overall. Local SHAP (waterfall) helps answer what drove this specific prediction.

Result: Interpretability becomes much more valuable when it shifts from feature ranking to decision explanation.

Every categorical encoding makes an assumption about the structure of a variable. A practical guide to choosing between one-hot, ordinal, target, WoE, and contrast coding based on what your data actually looks like.

Result: Ten minutes of encoding analysis per variable can save days of debugging a model that memorizes instead of generalizes.

Hierarchical clustering can reveal useful, nested structure in customer data — but applied directly to millions of customers, it becomes computationally expensive very quickly. K-Means scales much better, but using it as the final answer too early flattens meaningful structure.

A practical two-stage pattern: K-Means compresses the population into representative centroids, then hierarchical clustering explores structure across those centroids.

Result: An algorithm does not always have to be the final model. It can prepare a feasible representation for the next layer.

After years of using both in competitions and production, there is no universal winner. On a well-prepared tabular dataset, the gap between them is often smaller than the gap created by better features, a better validation split, or removing leakage.

Start with "what is expensive, risky, or awkward in this pipeline?" not "which library is best?" The data representation usually decides the project.

Result: LightGBM for large feature-heavy datasets, XGBoost for explicit regularization and custom objectives. Spend most of the time on features, labels, and leakage checks.

In 2018 I ranked 1st in an IBM Watson AI Lab competition building an NLP classification workflow. Today, a first prototype for a similar problem starts much faster: a system prompt, a structured output schema, one LLM call.

The interface changed — the engineering discipline did not. Clear instructions, useful examples, and well-defined contracts still decide quality, exactly as they did in 2018.

Result: Same engineering principles, expressed through better tools. A vague task definition still creates unreliable outputs.

Your model gets AUC 0.92 in cross-validation and drops to 0.71 in production. Sometimes the model is not the problem — train and production data simply do not look like the same world.

Adversarial validation trains a classifier to tell training data apart from production data. A high AUC means the two worlds have drifted apart — and that is detectable before you deploy.

Result: 10 lines of code that save you from deploying a model that is already outdated.

Missingness is often not a data quality problem — it is a behavioral signal. In telecom, missing recharge means no recharge happened. In credit, missing income may mean refusal to disclose. Those are not "unknown" — they carry information.

The order of operations matters: create missingness indicators first, then null_count meta-features, then impute. Imputing first destroys the signal permanently.

Result: In 3 of my last 5 projects, the null flag ranked above the imputed variable in feature importance. The model cannot learn a signal you erased before it saw it.

In a Nairobi ambulance deployment competition, the model predicted crash probability per road segment. But the real decision was: where should 6 ambulances be positioned right now?

The important part was the translation layer between prediction and action: predict risk → select highest-risk segments → cluster into geographic groups → place ambulances at centroids.

Result: In real-world ML, prediction quality and decision quality are related, but they are not the same thing.

Some categorical variables are informative but too coarse in raw form. Job_type mixes several underlying dimensions: how income is generated, how often it arrives, and how savings are managed.

I decomposed it into multiple behavioral features: income source, savings pattern, income frequency. Each derived feature captures a different financial mechanism.

Result: The variable is no longer a flat category — it becomes a behavioral representation layer separating latent mechanisms.

In a multi-entity time series problem, I spent more time on routing than on model tuning. The series were too heterogeneous: some decelerating, some accelerating, some near-zero, some ambiguous.

Before fitting anything, I built a classification layer: compute elasticity score, classify each entity into a trajectory regime, route to a different forecasting strategy per regime.

Result: When you have many entities to forecast, don't just build a better model. Build a better routing system.

I encoded six categorical variables differently based on what they represented: Helmert Sequential for day_of_week, Polynomial Ordered for hardware generations, WoE for binary splits, Target encoding for high cardinality.

Every encoding imposes an assumption. Better tabular modeling often starts with representing variables in ways that reflect their actual structure.

Result: Encoding is not preprocessing — it is part of how the model is allowed to interpret the variable.

Started from precomputed communication communities (SAS CLA results). A communication community is not necessarily a household — it can contain extended family, multiple households, and close friends.

Combined two signals: ranked communication intensity (top contacts weighted by position) and shared dominant evening cell-tower usage. Same community + top-ranked contact + shared evening tower = household tie.

Result: In applied graph analytics, the first cluster is often not the real unit you care about.

Most data science asks "what increases Y?" — leading to correlation, regression, SHAP. But some questions are different: what must already be true for high Y to even be possible?

Used Necessary Condition Analysis (NCA) to identify variables that act as minimum requirements, not average drivers. Below a threshold, the outcome simply does not appear.

Result: Some variables are not drivers — they are constraints. "Invest in X to boost" ≠ "guarantee X above threshold before anything else matters."

In a startup survival project (~50K funding rounds), the label "did not raise again" breaks if the company hasn't been observable for the full 36-month horizon. A startup founded in 2019, observed at 2021, looks like failure but simply hasn't had enough time.

Made two design decisions: eligibility window (only train on 36+ months observable history) and asymmetric holdout (mirror deployment prevalence, not balanced 50/50).

Result: Label design is often more important than algorithm choice. Once temporal contamination enters training, the model learns recency instead of risk.

In a telecom family-detection project, I needed to distinguish household-level ties from broader social communities. Added shared dominant cell-tower usage — "which cell tower shows up most in your evening CDR?" People spend ~8 hours a night in the same place.

Infrastructure data became behavioral evidence. The richest signals often hide in operational data, not in declared information.

Result: Cell tower dominance acts as a strong proxy for home location — operational traces as features.

Built an XGBoost model to predict hourly telecom data usage. Standard confidence intervals treated all predictions the same way — but peak hours had much larger errors. The business was making decisions based on false confidence.

Wrapped the model with conformal prediction: a distribution-free uncertainty quantification method. The interval width adapts to the data — wider in peak hours, narrower in off-peak.

Result: Point predictions without uncertainty are incomplete. The interval is the signal that tells you when to trust the forecast.

Focal loss reduces the learning effort spent on examples a classifier already predicts correctly, so rare, hard-to-predict cases actually drive model improvement. One term — (1 − p_t)^γ — controls how sharply the loss focuses on difficult examples.

Result: Focal loss is one lever among several for class imbalance — the right answer is the one that improves your actual validation metric, not the one that sounds most sophisticated.

In multi-series forecasting, the same mechanics repeat: lags, rolling transforms, calendar features, time ordering, multi-step forecasts. MLforecast lets you define that recipe once and apply it consistently across every series.

Result: Good tooling moves effort away from repeated feature plumbing and toward the forecasting decisions that actually matter.