STEG Fraud Detection in Electricity & Gas

The Problem

The first thing I noticed was the imbalance: only 5.6% of records were fraudulent. STEG, the Tunisian electricity and gas utility, was losing 200 million Dinars to meter manipulation — but in the data, fraudsters were needles in a haystack. Given 15 years of billing history for each client (invoices, counter readings, consumption patterns from 2005 to 2019), I needed to identify which clients were cheating.

Three challenges made this harder than a standard binary classification. First, the data was invoice-level — multiple rows per client — so I had to engineer meaningful aggregations before any model could run. Second, categorical variables had inconsistent factor levels between train and test, meaning naive encoders would silently break on unseen values. Third, with only 20 continuous hours for the hackathon, every design decision had to be high-impact and fast to implement.

The competition attracted 191 teams, but only 54 managed to submit a valid entry within the time constraint. This told me the data wrangling alone was a significant filter — most teams got stuck on the engineering before reaching the modeling.

My Approach

I knew that with a 20-hour constraint, I couldn't afford to spend time on approaches that might not pay off. So I focused on two high-leverage strategies: stochastic feature selection and ensemble diversity. Standard greedy feature selection (forward or backward) gets trapped in local optima — it evaluates features one at a time and misses combinations that are only useful together. Instead, I ran 100 iterations where each randomly sampled 12 to 35 features, trained a LightGBM, and recorded the AUC. The features appearing in the top-5 performing subsets formed my final feature set.

For the high-cardinality region variable, I applied five different encoding strategies and kept all of them as separate features: target mean, Weight of Evidence, James-Stein shrinkage, M-estimator, and leave-one-out. Rather than choosing one encoding before seeing the data, I let the model learn which encoding captures the most signal at each tree split. This is a small trick that adds almost no engineering time but consistently improves gradient-boosted models on high-cardinality categoricals.

The ensemble combined 14 base models across three diversity axes: algorithm (LightGBM, XGBoost, CatBoost, H2O AutoML, Random Forest), hyperparameters (deep vs. shallow trees, high vs. low learning rate, dart vs. gbdt boosters), and random seed. An elastic net meta-learner trained on the 14 probability outputs learned the optimal blend weight for each model — favoring those that were both accurate and contributed unique signal that other models missed.

For the class imbalance, I tested multiple strategies: stratified undersampling, SMOTE, 50/50 sampling for fast iteration, and full data with class weighting for final models. The answer depended on the algorithm — tree models preferred weighted full data while linear models benefited from rebalancing.

Key Decisions

Key Takeaway

In a 20-hour hackathon, the two highest-leverage decisions were feature selection strategy and ensemble diversity. Standard greedy selection would have missed interaction-dependent features. Standard single-model optimization would have capped the ceiling. The stochastic subset search and 14-model blend are what pushed the result to Rank 6. The lesson I carry forward: feature interaction spaces are too large for sequential search. Randomized exploration is not just a shortcut — it discovers combinations that greedy methods structurally cannot find.