isnull() is a feature. Sometimes the best one.
Missingness is often not a data quality problem — it is a behavioral signal. In 3 of my last 5 projects, the null flag ranked above the imputed variable in feature importance.
Visual Summary
The Core Idea
I treat null indicators as first-class features. Not cleanup — feature engineering.
df['feature_X_is_null'] = df['feature_X'].isnull().astype(int)
This one-liner has been more predictive than the imputed value itself in 3 of my last 5 projects.
Why Missing Values ARE Information
Telecom churn model
recharge_amount = NaN doesn't mean "unknown" — it means "this customer DIDN'T
recharge." That's a churn signal.
Credit scoring model
income = NaN doesn't mean "data entry error" — it might mean "applicant refused
to declare." That's a risk signal.
Medical dataset
blood_test_X = NaN doesn't mean "missing" — it means "doctor didn't order this
test." The absence IS clinical information.
The Correct Pattern
The order of operations matters:
- Step 1: Create missingness indicators
- Step 2: Create row-level null_count meta-feature
- Step 3: Create interactions around missingness (if needed)
- Step 4: THEN impute values
# Preserve the missingness signal first df["income_is_null"] = df["income"].isnull().astype(int) # Capture the overall missingness pattern df["null_count"] = df[cols_with_nulls].isnull().sum(axis=1) # Then impute df["income"] = df["income"].fillna(df["income"].median())
Common mistake: Impute first, then engineer features later. That destroys the missingness signal permanently.
Real Project Evidence
- The null flag ranked above the imputed variable in feature importance
- The absence of the action mattered more than its amount
- The missingness pattern itself segmented behavior
Key Takeaway
Design insight: In tabular ML, the absence of data is itself data. Before filling NaNs, ask: does this absence mean something operationally? If yes, encode it explicitly. The model cannot learn a signal you erased before it saw it.