AI Workflow · Feature Engineering · Tier B

AI Pre-Flight Review Before You Write Model Code

Before I write serious model code, I use an AI assistant to challenge five assumptions — problem framing, features, validation, leakage, and the baseline. It does not build the project; it makes the plan reviewable.

Problem FramingFeature InventoryLeakage ReviewBaseline First

View on LinkedIn →

TL;DR: Use AI early to challenge assumptions, not late to justify results. An AI assistant is a useful second set of eyes on problem framing, feature candidates, validation design and leakage risk — but the business context decides the problem, the data-generating process decides the split, and the human expert verifies availability, timing and data lineage. The value is not AI-generated answers; it is reaching the modelling stage with assumptions that are visible, challengeable, and easy to test.

Visual Summary

Before training a model, ask AI to challenge these 5 assumptions — not to decide, but to make the plan reviewable

Start by questioning the problem, not the algorithm: what decision are we improving, and which feature categories are missing

Design the evaluation before building the model: time split, group split, or rolling-origin backtest

Treat AI as a second set of eyes on feature logic, not as a leakage detector

Define what success means before chasing a score: baseline to beat, decision metric, minimum useful lift, operational constraints

AI can challenge the plan, but humans must verify it — frame, inventory, validate, check leakage, define success, then model

The Problem

There is a familiar order of operations in a modelling project: load the data, build features, train something, tune it, and only then ask whether the problem was framed correctly. It is efficient to execute and expensive to unwind. A model can be perfectly well built around the wrong decision, validated on a split that does not match deployment, and reported with a metric that nobody in the business actually cares about.

I started using an AI assistant earlier in that process — not to write the pipeline, but to challenge my first assumptions before I invest hours in it. The assistant is a fast, tireless second reader. What it is not is a decision maker. The value is not in the answers it produces; it is in reaching the modelling stage with assumptions that are written down, visible, and challengeable.

The Approach

My modelling checklist has five gates, and I run them before any serious model code. The assistant proposes; I verify. Together they are the validation strategy for the plan itself, not just for the model.

1. Question the problem, not the algorithm

Start from the decision: what decision are we improving? Then check whether this is even a classification problem. It might be ranking, survival analysis, forecasting, anomaly detection, or a rules problem that no model should be solving at all. The assistant is good at surfacing alternatives; the business context is what decides between them.

2. Inventory feature categories, then verify each one

Given the available schema, ask which categories of feature might be missing — behavioural changes, historical aggregates, relative-position features, calendar effects, entity relationships, external signals. Then do the part the model cannot: check availability, feasibility, meaning, and point-in-time correctness — does the feature exist at the moment the prediction is made? Suggested does not mean available, valid, or useful.

3. Design the validation environment before building

Ask what a realistic evaluation environment looks like, then let the data-generating process decide. Time-ordered data wants a rolling-origin backtest or a future holdout. Grouped entities want a group-aware split so the same customer cannot appear on both sides. Independent observations can take a random split. And a gap is sometimes required, when labels overlap or data arrives late.

4. Use the assistant as a second set of eyes, not a leakage detector

I ask it to inspect feature logic and raise questions: could this use information unavailable at prediction time, is a target-derived aggregate fitted safely, does a rolling feature look into the future? It is genuinely useful at surfacing suspicious patterns, and it turns data leakage from an invisible risk into a named one. It is not proof of safety. A plausible answer is not a guarantee, and the verification of data lineage and timing stays with the human expert.

5. Define the baseline and success criteria first

Before modelling, write down the simple baseline model to beat, the metric that actually matters at the relevant horizon, the operational constraints (latency, cost, availability, ownership), and the minimum improvement worth shipping. Metric selection is a modelling decision, not a reporting detail. Without those four, a model can improve a score that changes no decision. An assistant can help structure the criteria; it cannot judge whether the expected result is credible.

Outcome

The practical effect is that arguments happen earlier and cheaper. A framing dispute that would have surfaced three weeks later, after a pipeline, a tuning sweep and a reporting slide, now happens on a whiteboard before any of that exists. The five gates are also reusable: they are the same review I apply to a colleague's design, so AI-assisted development of this kind does not skip the human step, it preps it. In an MLOps workflow it is cheap to run and expensive to omit, because it changes what model selection ever gets to compare.

Suggested is not verified

The failure mode to watch is treating plausible output as a review result. The assistant widens the option set; it does not confirm availability, timing, or business meaning. Every suggestion still has to be checked against the schema, the data lineage, and the decision it is meant to support.

Use it early, not as justification

The same assistant asked to review a finished model tends to produce a polished confirmation of what already exists. Asked before the pipeline, it produces questions worth answering. The timing is the whole difference.

This is not a case for letting AI run the project. It is a case for spending ten minutes early on the five questions I would otherwise skip. Start coding with a clearer plan, then go and test every assumption in it.

Key Takeaway

Design insight: Use AI early to challenge assumptions, not late to justify results. An AI assistant is a useful second set of eyes on problem framing, feature candidates, validation design and leakage risk — but the business context decides the problem, the data-generating process decides the split, and the human expert verifies availability, timing and data lineage. The value is not AI-generated answers; it is reaching the modelling stage with assumptions that are visible, challengeable, and easy to test.

AI Can Review Feature Code. It Cannot Approve It → Not every analytics question is about what drives the outcome → A predictive model is not a decision system → The most dangerous label in ML is the one that looks correct but isn't! → Seed Sensitivity Is Part of Model Selection → Foundation Models Raise the Baseline → Encoding is a modeling decision, not a preprocessing checkbox → When the Label Does Not Exist: Define the Behaviour First → A Strong AutoML Baseline Can Beat Hand-Tuned Models → Automate Maintenance, Keep Judgment Human →

← PreviousAll Insights Next →Make a Simple Forecasting Baseline Fail First

FAQ

What is the key takeaway from "AI Pre-Flight Review Before You Write Model Code"?

Use AI early to challenge assumptions, not late to justify results. An AI assistant is a useful second set of eyes on problem framing, feature candidates, validation design and leakage risk — but the business context decides the problem, the data-generating process decides the split, and the human expert verifies availability, timing and data lineage. The value is not AI-generated answers; it is reaching the modelling stage with assumptions that are visible, challengeable, and easy to test.

Who wrote this and what is it about?

This was written by Mahmoud Trigui, Senior Data Scientist. AI as a pre-flight review layer: challenge problem framing, features, validation, leakage and the baseline before you build the pipeline.

Can an AI assistant replace the human review of a machine learning project?

No. It widens the option set and surfaces questions worth asking, but it does not decide. The business context chooses the problem, the data-generating process chooses the split, and a human expert verifies availability, point-in-time correctness and data lineage. Treat its output as suggestions to verify, never as a review result.

What should I check before writing my first model pipeline?

Five things: what decision the model is meant to support, and whether it is even a modelling problem; which feature categories are missing, and whether each one exists at prediction time; what a realistic validation environment looks like, such as a rolling-origin backtest, a group-aware split, or a random split; where data leakage could enter; and the baseline model plus the success criteria to beat. Answering these first turns hours of rework into a short design review.

Machine Learning Feature Engineering MLForecast Time Series Decomposition Forecasting LightGBM XGBoost Catboost Clustering Segmentation NLP LLMs Web App R Markdown SQL Oracle DB SAS-Guide SAS E-Miner Dataiku BigQuery GCP Python R CRISP-DM Hypothesis Testing ANOVA Data Analytics Dimensionality Reduction Recommendation System Network Analysis Geospace Analysis Spatial Data Embedding Sampling Techniques Decision Rules Data Storytelling CVM Churn Fraud Detection Sentiment Analysis Topic Modeling IBM Watson PowerBI Looker Studio VBA Statistical Learning Ensemble Modeling Stacking Cross-Validation Profiling ABT Construction Plumber Tidyverse Shiny Prophet Deep Learning Scikit-Learn JSON SAS Programming Git VS Code CSS Styling Automated Reporting Outlier Detection Temporal Clustering Startup Survival Pre-Valuation Modeling K-Means Decision Trees Data Science Predictive Modeling SVM LDA Text Classification Weight Prediction Pattern Recognition Real-Time Detection Community Detection Pipeline Automation Data Quality Checks Data Reliability Specification Mapping Business Strategy Marketing Campaigns Try & Buy Frameworks KPI Dashboards Network Quality Sales Analytics Mentoring Statistics Lecturer Remote Work Hybrid Work Consulting Contract Full-Time Freelance Sofrecom Orange Group Tunisia Telecom Kiota Intelligence VC Analytics Series A Prediction Pre-Valuation Modeling Production ML Applied AI Prompt Engineering Business Forecasting Decision Systems Graph Analytics Household Detection Multi-SIM Detection FTTH Forecasting Audit Extraction Infrastructure Classification Pydantic GPT-4 OpenAI API Base64 Classification Zindi Codementor LAAS-CNRS ESSAI MIT xPRO Tunisia ML Competition Cell Tower Analysis Uber Logistics Uber Cape Town Necessary Condition Analysis Behavioral Signals Spike Smoothing Observation Unit Design Dendrogram Ward Clustering VIF Target Encoding Machine Learning Feature Engineering MLForecast Time Series Decomposition Forecasting LightGBM XGBoost Catboost Clustering Segmentation NLP LLMs Web App R Markdown SQL Oracle DB SAS-Guide SAS E-Miner Dataiku BigQuery GCP Python R CRISP-DM Hypothesis Testing ANOVA Data Analytics Dimensionality Reduction Recommendation System Network Analysis Geospace Analysis Spatial Data Embedding Sampling Techniques Decision Rules Data Storytelling CVM Churn Fraud Detection Sentiment Analysis Topic Modeling IBM Watson PowerBI Looker Studio VBA Statistical Learning Ensemble Modeling Stacking Cross-Validation Profiling ABT Construction Plumber Tidyverse Shiny Prophet Deep Learning Scikit-Learn JSON SAS Programming Git VS Code CSS Styling Automated Reporting Outlier Detection Temporal Clustering Startup Survival Pre-Valuation Modeling K-Means Decision Trees Data Science Predictive Modeling SVM LDA Text Classification Weight Prediction Pattern Recognition Real-Time Detection Community Detection Pipeline Automation Data Quality Checks Data Reliability Specification Mapping Business Strategy Marketing Campaigns Try & Buy Frameworks KPI Dashboards Network Quality Sales Analytics Mentoring Statistics Lecturer Remote Work Hybrid Work Consulting Contract Full-Time Freelance Sofrecom Orange Group Tunisia Telecom Kiota Intelligence VC Analytics Series A Prediction Pre-Valuation Modeling Production ML Applied AI Prompt Engineering Business Forecasting Decision Systems Graph Analytics Household Detection Multi-SIM Detection FTTH Forecasting Audit Extraction Infrastructure Classification Pydantic GPT-4 OpenAI API Base64 Classification Zindi Codementor LAAS-CNRS ESSAI MIT xPRO Tunisia ML Competition Cell Tower Analysis Uber Logistics Uber Cape Town Necessary Condition Analysis Behavioral Signals Spike Smoothing Observation Unit Design Dendrogram Ward Clustering VIF Target Encoding