Conformal Prediction: when the model is uncertain, the interval tells you
In production forecasting, point predictions without uncertainty are incomplete. The interval is the signal that tells you when to trust the forecast.
The Problem
In a telecom demand forecasting project, I built an XGBoost model to predict hourly data usage. The model was accurate on average.
But when I looked at the residuals, something was wrong: in peak hours (evening, weekends), the model's errors were much larger than in off-peak hours.
Yet standard confidence intervals treated all predictions the same way. Peak hours looked certain when they were actually very uncertain. Off-peak forecasts looked uncertain when they were actually quite reliable.
The business was making inventory decisions based on false confidence.
The Solution: Conformal Prediction
I wrapped the model with conformal prediction — a distribution-free uncertainty quantification method.
- Step 1: Fit your model normally (XGBoost, neural net, whatever)
- Step 2: On a calibration set, compute residuals (actual - predicted)
- Step 3: Sort residuals, find the quantile matching your desired coverage (e.g., 90%)
- Step 4: For new predictions: prediction ± that calibrated residual quantile
The key insight: the interval width adapts to the data. In peak hours, the interval widened automatically (because residuals were larger). In off-peak hours, it narrowed.
The Business Impact
Before
Peak hour forecast: 1200 units ± 100 — same width as off-peak. False confidence.
After
Peak hour: 1200 units, 90% confident between 950 and 1450. Off-peak: 400 units, 90% confident between 380 and 420.
No distributional assumptions. No 'assume normal.' Just empirical coverage. That changed the business conversation entirely.
Key Takeaway
Design insight: Point predictions without uncertainty are incomplete. Conformal prediction gives you calibrated intervals without assumptions. The interval is not a nice-to-have — it's the signal that tells you when to trust the forecast.