DigiCow Farmer Training Adoption — Zindi Competition

The Problem

Agricultural training adoption prediction requires understanding that farmer behavior depends heavily on topic relevance, geographic context, and the timing of the training delivery. The task was to predict the probability that a farmer would adopt a DigiCow-supported practice within 7, 90, and 120 days of their first training session. Three time horizons, three separate predictions, all using only information available at the moment of training.

The hardest part wasn't the modeling — it was the data. Training topics were entered as free text by field trainers, riddled with typos, abbreviations, and inconsistent naming. "Artificial Insemination," "AI," "artficial insem," and "A.I. cattle" all referred to the same topic. Without normalizing these into canonical forms, the feature space would be impossibly fragmented — thousands of unique strings where fewer than 120 actual topics existed. Any model built on raw topic strings would learn noise.

The three time horizons added a structural constraint: adoption probability must be monotonically non-decreasing. The probability of adoption within 120 days must be at least as high as within 90 days, which must be at least as high as within 7 days. A model that violates this constraint produces logically impossible predictions — and gradient boosting models have no built-in mechanism to enforce monotonicity across multiple targets.

My Approach

The first battle was NLP topic normalization. I built a pipeline that fixed typos, expanded abbreviations, and then applied Levenshtein fuzzy matching to merge remaining variants into 117 canonical topics. This isn't a standard NLP problem — it's a specialized deduplication task where domain knowledge matters more than language models. Knowing that "AI" in agricultural context means "Artificial Insemination" rather than "Artificial Intelligence" requires domain-aware rules before fuzzy matching handles the residual cases.

From the normalized topics, I engineered features that capture topic-level adoption patterns: per-topic adoption rates (what fraction of farmers trained on this topic eventually adopt?), topic quality bins (Low/Medium/High/VeryHigh based on historical adoption rates), and topic category counts (how many dairy vs. poultry vs. livestock topics in this session). These features encode the fundamental insight that some topics are inherently more adoptable than others — a practical skill like "milking hygiene" has higher adoption rates than an abstract concept like "farm business planning."

Geographic context proved equally important. I computed adoption rates at county, subcounty, and ward levels — capturing the reality that farming communities influence each other. If your neighbors adopted a practice, you're more likely to adopt it too. Kenya's agro-ecological zones (High-potential, Medium, Semi-Arid) added another geographic layer, reflecting that practices viable in high-rainfall areas may be irrelevant in arid regions. Interaction terms between zone and training characteristics captured these geographic moderating effects.

The modeling used Bayesian optimization across six model families (LightGBM, XGBoost, Ranger, CatBoost, H2O AutoML, GLMNet), with six different sampling strategies to handle the 11% class imbalance in the 7-day target. A freeze set matched to the test distribution provided unbiased model comparison. The final step enforced monotonicity — if the 90-day prediction was lower than the 7-day prediction, it was corrected upward. This post-hoc fix ensured logical consistency without constraining the models during training.

Key Decisions

Key Takeaway

This project demonstrated that data cleaning can be the highest-leverage modeling decision. The NLP topic normalization — reducing thousands of noisy strings to 117 canonical topics — created the foundation for every useful feature downstream. Without clean topic labels, per-topic adoption rates are meaningless, topic categories are fragmented, and the model learns from noise rather than signal. In messy real-world data, the preprocessing pipeline that fixes input quality is often worth more than the modeling pipeline that fits the patterns.