Zindi New User Retention Challenge
This competition is hosted on Zindi, a machine learning platform for data science challenges.
TL;DR: For retention prediction, engineer features that capture engagement trajectories across multiple activity streams rather than static counts. Monthly distributions reveal whether engagement is rising or declining. Parallel regression (predicting activity intensity alongside the binary outcome) extracts additional signal from the same features and improves classification when used as an ensemble input.
The Problem
Predicting user engagement on a data science platform means understanding what signals in early behavior predict sustained participation. The task was binary classification: which new users on Zindi will remain active for two consecutive months? The subtlety is that "active" isn't just logging in — it means meaningful engagement with competitions, discussions, and content over a sustained period.
The data came from multiple activity streams: blog posts, comments, discussion threads, job listings, competition participation records, and hourly activity logs. Each stream captures a different facet of platform engagement. A user who reads blog posts behaves differently from one who enters competitions. A user who comments on discussions shows different commitment signals than one who just browses job listings. The challenge was synthesizing these disparate behavioral streams into features that predict long-term retention.
Class imbalance added a practical constraint. Most new users on any platform don't become long-term active members — that's the nature of user funnels. The model needed to identify the minority of users who will persist, without simply predicting "not retained" for everyone (which would be highly accurate but completely useless for the platform's retention team).
My Approach
I engineered features from every available activity stream, treating each as a different lens on user engagement. Blog engagement captured content creation behavior — not just whether a user posted, but the frequency and time gaps between posts, distributed across monthly buckets. Competition participation measured active learning behavior — how many competitions a user joined and their submission patterns. Discussion activity revealed community engagement — posting, commenting, and the timing of interactions.
The most revealing features turned out to be temporal patterns rather than simple counts. It's not how many blogs a user read, but when they read them relative to their signup date. Early, intensive engagement in the first weeks predicts retention far better than total activity counts that could accumulate slowly over months. Monthly activity distributions (12 features showing engagement per month) captured the trajectory of engagement — rising, flat, or declining — which is more predictive than any static snapshot.
The model architecture combined LightGBM and XGBoost in an ensemble, with both models trained using extensive hyperparameter search. LightGBM went through a two-stage optimization: first tuning structural parameters (num_leaves, max_bin, min_data_in_leaf), then refining regularization (L1/L2, feature fraction, bagging). XGBoost was separately optimized with 4-fold stratified cross-validation. A freeze set — separate from both training and validation — provided unbiased ensemble weight estimation.
I also built a parallel regression task predicting activity count, not just the binary retention outcome. The regression predictions provide additional signal for the classification task — users with higher predicted activity counts are more likely to be retained. This multi-task framing extracts more information from the same features by modeling engagement on both a binary and continuous scale.
Key Decisions
Multi-Stream Feature Engineering
Each activity stream (blogs, competitions, discussions, jobs, hourly logs) gets its own feature extraction logic. This reflects the reality that engagement is multidimensional — a user who enters competitions shows different commitment signals than one who writes blog posts. Combining diverse behavioral signals gives the model a complete picture of engagement patterns.
Temporal Trajectory Over Static Counts
Monthly activity distributions capture whether engagement is rising, flat, or declining — far more predictive than total counts. A user with 10 actions in month 1 and 0 in month 2 looks identical to one with 5 and 5 by total count, but their retention trajectories are completely different. The distribution is the signal.
Two-Stage Hyperparameter Optimization
First optimizing structural parameters (tree complexity, data requirements), then regularization (L1/L2, subsampling) avoids the combinatorial explosion of searching all parameters simultaneously. The structural search finds the right model capacity; the regularization search prevents that capacity from overfitting.
Parallel Regression Task for Additional Signal
Predicting both binary retention and continuous activity count extracts more information from the same features. The regression predictions become an additional input for the classification ensemble — high predicted activity is a strong signal for retention, bridging the gap between the binary outcome and the continuous engagement spectrum.
Key Takeaway
User retention prediction is fundamentally about engagement trajectories, not snapshots. The features that discriminate retained users from churned ones are temporal patterns — how quickly engagement ramps up, whether it's sustained across activity streams, and whether the trajectory is rising or declining. Static user attributes matter far less than behavioral dynamics in the first weeks after signup.
Design insight: For retention prediction, engineer features that capture engagement trajectories across multiple activity streams rather than static counts. Monthly distributions reveal whether engagement is rising or declining. Parallel regression (predicting activity intensity alongside the binary outcome) extracts additional signal from the same features and improves classification when used as an ensemble input.
FAQ
What is the key takeaway from "Zindi New User Engagement Prediction"?
For retention prediction, engineer features that capture engagement trajectories across multiple activity streams rather than static counts. Monthly distributions reveal whether engagement is rising or declining. Parallel regression (predicting activity intensity alongside the binary outcome) extracts additional signal from the same features and improves classification when used as an ensemble input.
Who wrote this and what is it about?
This was written by Mahmoud Trigui, Senior Data Scientist. Predicting whether new Zindi users will remain active for 2+ months. Classification pipeline with feature engineering on platform behavior data. Ranked Top 70% (210/1100 competitors) on Zindi, February 2023.