The most dangerous label in ML is the one that looks correct but isn't!
In applied ML, label design is often more important than algorithm choice.
The Problem
In a startup survival project (Crunchbase, ~50K funding rounds), the question was: Will this company raise another round within 36 months?
At first glance, labeling sounds simple: raised again = Yes, did not raise = No. But that logic breaks if the company has not been observable for the full 36-month horizon.
A startup founded in 2019, observed at cutoff 2021, looks like a failure. It is labeled "No," but it is not a failure. It simply hasn't had enough time. That is not a model problem — it is a label validity problem.
Two Design Decisions
Eligibility window
Only train on companies with 36+ months of observable history. If the prediction horizon is 36 months and the observation window is shorter, the negative label is not trustworthy.
Asymmetric holdout
Most startups fail. Validation should mirror deployment prevalence between survivors and failures. Not a balanced 50/50 split. This ensures the model's probability calibration matches the real world.
Neither of these are model choices. They are label hygiene — the unglamorous work that determines whether the pipeline predicts something real or something circular.
Key Takeaway
Design insight: Once temporal contamination enters the training set, the model learns recency instead of risk. Label design is often more important than algorithm choice.