Seed Sensitivity Is Part of Model Selection
A configuration that wins once is not necessarily a good configuration: when the search process contains randomness, seed sensitivity is part of model selection.
TL;DR: A configuration that wins once is not necessarily a good configuration. When a forecasting-search workflow contains randomness, evaluate each promising configuration across multiple seeds and select on a robust aggregate such as the median result — because the goal is not the most impressive run but the configuration that remains useful when randomness changes.
The Problem
In a forecasting-search workflow, some candidate configurations looked excellent in one run. Then the seed changed. And the result changed too.
That is a problem when the search process itself contains randomness: stochastic model fitting, random sampling, evolutionary or genetic search, random feature or lag exploration, and resampled validation procedures all introduce variation between runs. If a configuration is selected from one run only, it may be selected because it got lucky.
The Approach
Instead of treating the single best score as the winner, I evaluated promising configurations across multiple seeds, then selected based on a robust aggregate such as the median result.
Where the randomness comes from
Stochastic fitting, random sampling, evolutionary or genetic search, random feature or lag exploration, and resampled validation all move the result between runs.
Judge configurations, not runs
A single score is evidence about one seed. A distribution across seeds is evidence about the configuration.
Pick the robust aggregate
The median result across seeds is more stable than the maximum: the flashy single-run winner can disappear, but what stays strong across repeated runs becomes easier to trust.
Outcome
The outcome was often less dramatic. The flashy single-run winner could disappear.
But the configurations that stayed strong across repeated runs became easier to trust — and that reliability is what makes a configuration worth deploying.
Key Takeaway
Design insight: Seed sensitivity is part of model selection when randomness drives the search process. Instead of selecting the most impressive single-run result, evaluate each promising configuration across multiple seeds and choose the robust aggregate such as the median — the goal is the configuration that remains useful when randomness changes, not the one that won once.
Related
LightGBM vs XGBoost in 2026: A Practical Tabular Guide →A Reranker Cannot Rank What It Never Receives →Foundation Models Raise the Baseline →Zero-Shot Forecasting Changes the Baseline →Conformal Prediction: When the Model Is Uncertain →Adversarial Validation: Detect Data Shift Before It Hurts →The Hardest Forecasting Decision Is Which Model to Use →MLforecast Forecasting Pipeline →
FAQ
Why is seed sensitivity part of model selection?
When a forecasting-search workflow contains randomness - stochastic fitting, random sampling, evolutionary or genetic search, random feature or lag exploration, resampled validation - a configuration selected from one run may simply have gotten lucky. Seed sensitivity is part of model selection because the single best score is not reliable evidence when the seed changes.
How should you pick a configuration when the seed changes the result?
Evaluate each promising configuration across multiple seeds and select on a robust aggregate such as the median result. The goal is not the most impressive run but the configuration that remains useful when randomness changes.