Soil Moisture Prediction — Time Series Forecasting Pipeline
This competition is hosted on Zindi, a machine learning platform for data science challenges.
TL;DR: In IoT time series with varying data density across units, build independent models per unit and let complexity scale with data availability. Multi-scale lag and rolling features capture temporal dynamics without requiring sequence models, and strictly chronological splits are the only honest validation — anything else creates an illusion of accuracy that disappears at deployment.
The Problem
Low-cost IoT sensors in agricultural fields produce noisy, incomplete readings. The challenge was building models that could predict soil moisture 24 hours and 6 days ahead from this imperfect signal. Four irrigation fields, each with different soil properties, crop types, and irrigation schedules, needed independent predictions. The data was hourly time series with gaps, sensor drift, and no guarantee that the patterns from one field would transfer to another.
What makes agricultural time series different from financial or industrial forecasting is the physics. Soil moisture doesn't follow random walks — it follows deterministic processes (evaporation, drainage, irrigation events) overlaid with noisy measurements. A field that was irrigated yesterday will have a predictable moisture trajectory for the next 24 hours, regardless of what happened last week. The challenge is extracting this deterministic signal from noisy sensor readings with significant gaps.
The four fields had wildly different data availability — Field 1 had nearly 9,000 observations while Field 3 had barely 1,100. Building a single model would dilute Field 1's rich signal with Field 3's sparse noise. Building four models means each must work with its own data limitations. The multi-horizon prediction requirement (24 hours and 6 days) added another layer: features that predict well at short horizons may be useless at longer ones.
My Approach
I built independent XGBoost models for each field, respecting the reality that each field has unique soil characteristics and irrigation patterns that a pooled model would average away. Per-field modeling means Field 1's 9,000 observations train a deep, well-regularized model while Field 3's 1,100 observations train a shallower, more constrained one — each model's complexity matches its data availability.
The feature engineering centered on capturing temporal dynamics at multiple scales. Lag features at 1, 3, 6, and 12 hours capture immediate moisture decay patterns. Rolling statistics (mean, standard deviation, median, variance) over 3, 6, and 12-hour windows smooth out sensor noise while preserving trends. Irrigation lag features capture how recent watering events propagate through the soil profile over subsequent hours. Together, these create a 25+ feature representation that encodes both the recent trajectory and its volatility.
The validation strategy was strictly chronological — no shuffled cross-validation that would leak future information into training. For each field, the last 20% of observations formed the test set, ensuring the model proves it can predict forward in time rather than interpolate between known points. This is non-negotiable for time series: if your validation lets the model see tomorrow's data while predicting today, your error estimates are meaningless.
XGBoost was configured with a low learning rate (0.01) and deep trees (max_depth=12) to capture complex temporal patterns, with early stopping at 10 rounds preventing overfitting. I also explored GRU and LSTM architectures as alternatives, using a 288-step lookback window (12 hours at the observation frequency), though the gradient-boosted models proved more robust given the data sizes available.
Key Decisions
Per-Field Independent Models
Each field has unique soil properties, crop cover, and irrigation schedules that make pooled modeling counterproductive. Independent models let each field's complexity match its data availability — deep trees for data-rich fields, constrained models for sparse ones. The alternative (one model with field as a feature) averages away field-specific dynamics.
Multi-Scale Lag and Rolling Features
Lag features at 1, 3, 6, and 12 hours capture immediate dynamics while rolling statistics smooth sensor noise. The combination gives the model access to both the recent trajectory and its volatility — knowing that moisture is 40% means nothing without knowing whether it's been rising or falling and how stable the readings are.
Strictly Chronological Validation
No shuffled CV. The last 20% of each field's timeline forms the test set. This prevents temporal leakage — the silent error mode where models appear accurate because they've seen nearby future observations during training. In time series, the only honest validation is forward prediction.
Deep XGBoost with Early Stopping
A learning rate of 0.01 with max_depth=12 allows the model to capture complex temporal interactions (irrigation events compound over multiple time steps), while early stopping at 10 rounds prevents the deep trees from memorizing training patterns. This configuration balances expressiveness with generalization for small-to-medium time series.
Key Takeaway
Agricultural IoT forecasting rewards simplicity matched to data constraints. Per-field models with well-engineered temporal features and strict chronological validation outperformed more complex approaches (neural networks, pooled models) because they respect three realities: each field is physically different, time series validation must be forward-looking, and model complexity must scale with data availability rather than problem ambition.
Design insight: In IoT time series with varying data density across units, build independent models per unit and let complexity scale with data availability. Multi-scale lag and rolling features capture temporal dynamics without requiring sequence models, and strictly chronological splits are the only honest validation — anything else creates an illusion of accuracy that disappears at deployment.
FAQ
What is the key takeaway from "Soil Moisture Prediction"?
In IoT time series with varying data density across units, build independent models per unit and let complexity scale with data availability. Multi-scale lag and rolling features capture temporal dynamics without requiring sequence models, and strictly chronological splits are the only honest validation — anything else creates an illusion of accuracy that disappears at deployment.
Who wrote this and what is it about?
This was written by Mahmoud Trigui, Senior Data Scientist. Predicting soil humidity in 5-minute increments using low-cost DIY IoT sensor data. Ranked Top 37% (96/700 competitors) on Zindi, October 2019.