An LLM-Derived Feature Is Still a Feature
Adding an LLM-derived feature to a forecasting pipeline is easy. Evaluating it with proper time-series discipline is the part many teams skip - and that is where risk enters.
TL;DR: An LLM-derived feature is still a feature. Ship it only when it improves the model you would actually deploy, stays valid at the forecast origin, and survives production conditions - cost, latency, stability, and failure handling. Novelty is not evidence; point-in-time backtesting is. Treat LLM features like every other feature, and the validation bar stays exactly where it belongs.
Visual Summary
The Problem
Adding an LLM-derived feature to a forecasting pipeline is easy. Evaluating it with proper time-series discipline is the part many teams skip. That is where risk enters.
An LLM-generated signal - sentiment, semantic label, embedding, event score, or text summary - does not receive special treatment because it looks sophisticated. It is still a feature.
And it should face the same forecasting rigor as every lag, rolling statistic, or external regressor.
The Approach
My evaluation sequence is simple: four checks before an LLM feature ships.
1. Backtest it point in time
Use the same rolling-origin or walk-forward validation as any forecasting feature. At every forecast origin, generate or retrieve the LLM feature using only information available at that point in time.
2. Check feature availability explicitly
Was the source text available before the forecast timestamp? Could retrieval return documents published later? Did an aggregate accidentally include future observations? Are the prompt, model version, and retrieval index versioned for reproducibility?
3. Compare against the real baseline
Do not compare with a naive forecast. Compare with the best model you would actually deploy, and check lift by forecast horizon, segment, and series - not only the global average.
4. Check operational viability
Measure cost per refresh or batch, latency, failure rate, stability across reruns, and monitoring and fallback behaviour.
Outcome
An LLM feature should ship only if it improves the existing baseline, remains valid at forecast time, and works at production scale.
Novelty does not lower the bar. It raises the number of things worth checking.
Key Takeaway
Design insight: An LLM-derived feature is still a feature. Ship it only when it improves the model you would actually deploy, stays valid at the forecast origin, and survives production conditions - cost, latency, stability, and failure handling. Novelty is not evidence; point-in-time backtesting is. Treat LLM features like every other feature, and the validation bar stays exactly where it belongs.
Related
LLM Semantic Features: Propose, Then Earn Their Place →Seed Sensitivity Is Part of Model Selection →The Hardest Forecasting Decision Is Which Model to Use →MLforecast Made Me Rewrite My Forecasting Pipeline →StatsForecast, MLforecast & NeuralForecast →LLM Fallback Strategy: What Happens When the Model Fails? →Enterprise LLM Wrapper →
FAQ
What is the key takeaway from "An LLM-Derived Feature Is Still a Feature"?
An LLM-derived feature is still a feature. Ship it only when it improves the model you would actually deploy, stays valid at the forecast origin, and survives production conditions - cost, latency, stability, and failure handling. Novelty is not evidence; point-in-time backtesting is. Treat LLM features like every other feature, and the validation bar stays exactly where it belongs.
Who wrote this and what is it about?
This was written by Mahmoud Trigui, Senior Data Scientist. An LLM-derived feature is still a feature: backtest it point in time, check leakage, compare against the production baseline, then verify operational viability.