← All Insights
Data Quality · AI Workflows · Production ML · Tier B

AI-Generated Validation Can Pass While Data Is Still Wrong

An AI assistant generated sensible checks for a customer table — nulls, ranges, types, uniqueness, allowed values. Everything passed. The data was still wrong: the missing rule was not technical, it was semantic.

Customer Lifecycle Semantic Rules State Transitions Humans Define Quality
View on LinkedIn →

TL;DR: An AI-generated validation suite can pass while the data is still wrong. Technical validity is necessary, but business validity is the real goal — AI can generate checks, humans define quality.

The Problem

I asked an AI assistant to write a data-validation suite. It generated sensible checks: no nulls in key IDs, revenue within a valid range, unique records where expected, valid date formats, and accepted categorical values. Then I ran the validation. Everything passed.

And the data was still wrong. The missing rule was not technical — it was semantic. The dataset contained a customer state that was impossible in the real business process: a customer marked as churned also had a new activation event recorded earlier in the same lifecycle.

The Approach

No null check, range check, type check, or autogenerated expectation can reliably infer that rule without someone explaining how the business process actually works. That is the right way to think about AI-generated validation code.

AI is useful for generating the scaffold: repetitive validation syntax, standard null, type, range, and uniqueness checks, initial test-suite structure, and documentation and naming conventions.

But the highest-value quality rules usually require context. In this case, the rule was a lifespan constraint: a reactivated status can only follow a churned status through an approved transition. After a customer reaches churned, a later activated event is allowed only if an approved reactivated transition exists in between.

Outcome

Adding the semantic rule made the impossible record fail correctly: FAIL — lifecycle_transition_violation. The same principle applies to every business question: can this status follow the previous status? Can a contract end before it starts? Can revenue exist before activation? Can this event occur after an account is closed?

The assistant can express a rule. It cannot reliably decide which business rules matter. So yes: let AI remove validation boilerplate — but keep ownership of the definition of “valid” with the people who understand the data and the process behind it.

Key Takeaway

Design insight: An AI-generated validation suite can pass while the data is still wrong. Technical validity is necessary, but business validity is the real goal — AI can generate checks, humans define quality.

Related

Adversarial Validation: Detect Data Shift Before It Hurts →AI Can Review Feature Code. It Cannot Approve It →When the Label Does Not Exist: Define the Behaviour First →Foundation Models Raise the Baseline →LLM Semantic Features: Propose, Then Earn Their Place →From Cluster to Operational Segment: Assignment First →Feature Engineering Is Assembling Evidence Across Systems →Not Every Analytics Question Is About What Drives Outcome →

← PreviousAll Insights Next →From Cluster to Operational Segment: Assignment First

FAQ

What is the key takeaway from "AI-Generated Validation Can Pass While Data Is Still Wrong"?

An AI-generated validation suite can pass while the data is still wrong. Technical validity is necessary, but business validity is the real goal — AI can generate checks, humans define quality.

Who wrote this and what is it about?

This was written by Mahmoud Trigui, Senior Data Scientist. AI validation can pass while the data is still wrong: the missing rule is semantic — and only business owners can define it.

Comments

Machine Learning Feature Engineering MLForecast Time Series Decomposition Forecasting LightGBM XGBoost Catboost Clustering Segmentation NLP LLMs Web App R Markdown SQL Oracle DB SAS-Guide SAS E-Miner Dataiku BigQuery GCP Python R CRISP-DM Hypothesis Testing ANOVA Data Analytics Dimensionality Reduction Recommendation System Network Analysis Geospace Analysis Spatial Data Embedding Sampling Techniques Decision Rules Data Storytelling CVM Churn Fraud Detection Sentiment Analysis Topic Modeling IBM Watson PowerBI Looker Studio VBA Statistical Learning Ensemble Modeling Stacking Cross-Validation Profiling ABT Construction Plumber Tidyverse Shiny Prophet Deep Learning Scikit-Learn JSON SAS Programming Git VS Code CSS Styling Automated Reporting Outlier Detection Temporal Clustering Startup Survival Pre-Valuation Modeling K-Means Decision Trees Data Science Predictive Modeling SVM LDA Text Classification Weight Prediction Pattern Recognition Real-Time Detection Community Detection Pipeline Automation Data Quality Checks Data Reliability Specification Mapping Business Strategy Marketing Campaigns Try & Buy Frameworks KPI Dashboards Network Quality Sales Analytics Mentoring Statistics Lecturer Remote Work Hybrid Work Consulting Contract Full-Time Freelance Sofrecom Orange Group Tunisia Telecom Kiota Intelligence VC Analytics Series A Prediction Pre-Valuation Modeling Production ML Applied AI Prompt Engineering Business Forecasting Decision Systems Graph Analytics Household Detection Multi-SIM Detection FTTH Forecasting Audit Extraction Infrastructure Classification Pydantic GPT-4 OpenAI API Base64 Classification Zindi Codementor LAAS-CNRS ESSAI MIT xPRO Tunisia ML Competition Cell Tower Analysis Uber Logistics Uber Cape Town Necessary Condition Analysis Behavioral Signals Spike Smoothing Observation Unit Design Dendrogram Ward Clustering VIF Target Encoding Machine Learning Feature Engineering MLForecast Time Series Decomposition Forecasting LightGBM XGBoost Catboost Clustering Segmentation NLP LLMs Web App R Markdown SQL Oracle DB SAS-Guide SAS E-Miner Dataiku BigQuery GCP Python R CRISP-DM Hypothesis Testing ANOVA Data Analytics Dimensionality Reduction Recommendation System Network Analysis Geospace Analysis Spatial Data Embedding Sampling Techniques Decision Rules Data Storytelling CVM Churn Fraud Detection Sentiment Analysis Topic Modeling IBM Watson PowerBI Looker Studio VBA Statistical Learning Ensemble Modeling Stacking Cross-Validation Profiling ABT Construction Plumber Tidyverse Shiny Prophet Deep Learning Scikit-Learn JSON SAS Programming Git VS Code CSS Styling Automated Reporting Outlier Detection Temporal Clustering Startup Survival Pre-Valuation Modeling K-Means Decision Trees Data Science Predictive Modeling SVM LDA Text Classification Weight Prediction Pattern Recognition Real-Time Detection Community Detection Pipeline Automation Data Quality Checks Data Reliability Specification Mapping Business Strategy Marketing Campaigns Try & Buy Frameworks KPI Dashboards Network Quality Sales Analytics Mentoring Statistics Lecturer Remote Work Hybrid Work Consulting Contract Full-Time Freelance Sofrecom Orange Group Tunisia Telecom Kiota Intelligence VC Analytics Series A Prediction Pre-Valuation Modeling Production ML Applied AI Prompt Engineering Business Forecasting Decision Systems Graph Analytics Household Detection Multi-SIM Detection FTTH Forecasting Audit Extraction Infrastructure Classification Pydantic GPT-4 OpenAI API Base64 Classification Zindi Codementor LAAS-CNRS ESSAI MIT xPRO Tunisia ML Competition Cell Tower Analysis Uber Logistics Uber Cape Town Necessary Condition Analysis Behavioral Signals Spike Smoothing Observation Unit Design Dendrogram Ward Clustering VIF Target Encoding