Semi-Automatic Forecasting Framework

The Problem

The planning teams at Sofrecom needed reliable forecasts to drive their annual budgeting, resource allocation, and staffing decisions. But "reliable forecasts" meant something different for each department. Some needed 12-month ahead predictions at monthly granularity for strategic planning. Others needed 52-week forecasts to schedule field teams and manage operational capacity. And each department had multiple KPIs they tracked independently—each one essentially its own time series with its own patterns and behaviors.

The existing process was largely manual: analysts would pull data, run a few models in Excel or basic statistical tools, pick the one that "looked right," and present it. This approach had two fundamental problems. First, it didn't scale—with dozens of series to forecast, each needing careful attention to seasonality, trends, and special events, the manual workload was crushing. Second, it wasn't reproducible—different analysts made different choices, and when forecasts needed to be updated, there was no guarantee of consistency.

What the business really needed was a system that could handle the diversity of their forecasting needs (different series, different granularities, different horizons) while being rigorous enough that stakeholders could trust the numbers for high-stakes decisions like hiring plans and budget allocations. The system needed to be semi-automatic: smart enough to handle the technical complexity without constant human intervention, but transparent enough that an analyst could understand and override its choices when domain knowledge dictated something different.

The Approach

I architected the framework around a 3-step optimization process that mirrors how an experienced forecaster would approach each series, but automated to handle dozens of series efficiently. The key insight was that you can't just throw AutoML at time series and hope for the best—the optimization needs to respect the sequential nature of the problem and the different roles that lags, features, and model architecture play.

Step one identifies the optimal lag structure for each series. Not every series benefits from the same lookback window—some have strong weekly seasonality requiring at least 52 lags, while others only need 3-4 recent values. I implemented an automated search that evaluates different lag configurations using time-series cross-validation, measuring forecast accuracy on held-out future periods rather than random splits.

Step two introduces exogenous features: date-based variables (day of week, month, holidays), decomposition components (trend, seasonality, residual), and any business-specific indicators available for each series. The framework tests each feature's marginal contribution and retains only those that improve out-of-sample accuracy—preventing the overfitting that plagued the manual approach.

Step three tunes the model architecture and hyperparameters. The framework evaluates multiple model families (LightGBM, XGBoost, linear models) and applies Bayesian optimization for hyperparameter tuning within each family. The final prediction is an ensemble of the top-performing models, weighted by their cross-validation scores—this smoothing step consistently outperformed any single model by 5-10% on the test sets.

Implementation Details

The feature engineering pipeline is comprehensive but disciplined. Date-based features capture calendar effects (month, week of year, day of week, holiday indicators). Decomposition features use STL (Seasonal and Trend decomposition using Loess) to extract trend, seasonal, and remainder components—the trend component is particularly valuable for series with structural shifts. I also implemented Fourier features at multiple periodicities to capture complex seasonal patterns that simple indicators miss.

Target transformation was critical for series with multiplicative seasonality or heteroscedastic variance. I implemented log transforms, Box-Cox transforms, and differencing as configurable options that the optimization loop evaluates automatically. The framework applies the inverse transform to produce forecasts in the original scale, with proper uncertainty propagation so confidence intervals remain meaningful.

The ensemble approach deserves special attention. Rather than a simple average, I implemented a stacking approach where a meta-learner combines base model predictions using their cross-validated accuracy patterns. Some models are better at capturing trends while others handle seasonality well—the ensemble learns to leverage each model's strengths. I also implemented a "smoothing" post-processing step using exponential moving averages on the ensemble output, which reduces the jagged artifacts that tree-based models sometimes produce in multi-step forecasts.

For the visualization layer, I built interactive PowerBI and LookerStudio dashboards that let stakeholders explore forecasts at different levels of detail. They can see the overall trajectory, drill into specific departments or KPIs, compare forecast vs. actuals for past periods (building trust), and examine the confidence intervals that communicate uncertainty. This wasn't just a nice-to-have—it was essential for adoption. A forecast that lives in a Jupyter notebook doesn't drive decisions; a forecast in a dashboard that updates automatically becomes part of the planning workflow.