Co-Financing Prediction Model (Dataiku Migration)
I took an existing Python-based co-financing prediction model and re-architected it into a production-grade Dataiku workflow—prioritizing SQL-native operations for performance, designing an efficient Analytics Base Table process, and rigorously validating that the migration preserved (and in some cases improved) model accuracy.
The Problem
Co-financing in telecom infrastructure refers to arrangements where multiple operators share the cost of deploying physical network infrastructure—fiber ducts, poles, conduits. Predicting which infrastructure segments will attract co-financing interest (and from whom) is a high-value forecasting problem: accurate predictions allow better negotiation positioning, optimal deployment sequencing, and more accurate financial planning for capital-intensive fiber rollout projects.
A working prediction model already existed. It had been developed as a collection of Python scripts that ran locally, processed data through pandas DataFrames, applied MLforecast for time-series-aware predictions, and output results to CSV files. The model worked—its accuracy was acceptable and the business was using its predictions. But it was fragile. It ran on a single data scientist's machine, broke when data formats changed, couldn't be scheduled for automatic refresh, and was opaque to anyone who hadn't written the code. In short, it was a prototype masquerading as a production system.
The mandate was clear: migrate this model to Dataiku, the organization's standard ML platform, without losing accuracy. But I saw this as an opportunity to do more than a lift-and-shift. The original code had accumulated technical debt—inefficient data transformations that could be replaced with SQL, redundant processing steps, and feature engineering that happened in memory when it should happen in the database. A thoughtful migration could make the system not just more maintainable but faster and more scalable.
The Approach
I approached this as an architecture optimization project rather than a simple code translation. The first step was understanding the existing model deeply—not just what it did, but why each step existed and what the implicit assumptions were. I spent time with the original developer to document the data flow, the feature logic, and the modeling decisions that had been made iteratively but never written down.
The original Python scripts loaded raw data into pandas and performed joins, aggregations, and feature calculations in memory. This worked for the prototype but was a scalability ceiling. I redesigned the data preparation layer to use SQL recipes in Dataiku, pushing computation to the database engine where it belongs. This reduced the Analytics Base Table creation time from 45 minutes (Python/pandas) to under 3 minutes (SQL), eliminated memory constraints, and made the feature logic auditable by anyone who reads SQL—not just Python developers.
The migration followed a rigorous validation protocol. At each stage, I compared outputs between the old Python implementation and the new Dataiku workflow, ensuring numerical equivalence within floating-point tolerance. This wasn't just about the final predictions—I validated intermediate feature values, training set construction, and model coefficients to catch any divergence early rather than wondering why the final forecast looked slightly different.
The original model used MLforecast as a black box. During migration, I took the opportunity to validate its contribution rigorously. I tested the model with and without MLforecast's lag features, compared against simpler baselines, and evaluated whether additional predictive variables (that were available but unused in the original) could improve accuracy. This comparative analysis confirmed that MLforecast was adding genuine value (not just complexity) and identified two additional features that improved the model by 1.5%—a "free" improvement that came from questioning assumptions during migration.
Implementation Details
The Analytics Base Table (ABT) design was the architectural centerpiece. In the original code, the ABT was implicitly constructed through a chain of pandas operations—joins scattered across multiple scripts, feature calculations mixed with data cleaning, and no clear separation between raw data and derived features. I redesigned this as an explicit, documented ABT creation process: a sequence of SQL recipes that start from raw source tables and produce a single, well-documented analytical table with all features materialized and validated.
The Dataiku workflow is organized into distinct flow zones: data ingestion (connecting to source databases), data preparation (SQL-based cleaning and joining), feature engineering (both SQL-based and Python-based for complex temporal features), model training (using Dataiku's visual ML with custom code recipes for MLforecast), and output publication (predictions pushed to downstream systems). Each zone has input/output validation checks that verify data quality expectations are met before proceeding.
I tested alternative modeling approaches as part of the validation exercise. Beyond the original MLforecast implementation, I evaluated Dataiku's native AutoML models, gradient boosting with manual feature engineering, and a simplified statistical baseline. The comparative analysis revealed that the MLforecast approach genuinely outperformed alternatives by 3-5% on the co-financing prediction task—its temporal feature generation (automatic lag and rolling window calculations) captured patterns that static feature sets missed. This validated the original modeling choice and gave stakeholders confidence that the solution was robust, not just familiar.
The production deployment includes automated scheduling (weekly model refresh), performance monitoring (accuracy drift detection), and a fallback mechanism that uses the previous week's model if the current training fails quality checks. I also built a model comparison dashboard that shows the migrated model's performance against the historical Python implementation, making it easy for stakeholders to verify that the migration maintained or improved quality over time.
Key Takeaway
A migration project is a rare opportunity to challenge accumulated assumptions. The original developer made reasonable choices at the time, but those choices were never re-evaluated as the data grew and the platform matured. By treating migration as architecture optimization rather than code translation, I achieved three things simultaneously: better performance (15x faster data preparation), better maintainability (SQL is more accessible than pandas chains), and better accuracy (validated existing choices, found free improvements). Never just translate—always re-evaluate.
Related Insights
Adversarial Validation: Detect Data Shift Before Your Model Fails in Production →LightGBM vs XGBoost in 2026 — Practical Guide for Tabular ML →StatsForecast, MLforecast & NeuralForecast: Three Forecasting Jobs in One Ecosystem →isnull() Is a Feature — Missingness as Behavioral Signal →
Comments
FAQ
What is the key takeaway from "Co-Financing Prediction Model (Dataiku Migration)"?
Architecture optimization project converting Python forecasting scripts to production-grade Dataiku workflows, with SQL-first design and comprehensive model validation.
Who wrote this and what is it about?
This was written by Mahmoud Trigui, Senior Data Scientist. Architecture optimization project converting Python forecasting scripts to production-grade Dataiku workflows, with SQL-first design and comprehensive model validation.