Why Two Weaker Models Beat a Stronger One Through Error Diversity
Two weaker models can sometimes outperform one stronger model — not through magic, but because their mistakes are complementary.
TL;DR: The best ensemble is not the one with the best individual models — it is the one whose errors combine well enough to improve the real decision. Error diversity drives the gain; evaluate on held-out predictions and reject if operational cost outweighs measured improvement.
The Problem
Chasing the single "best" model often ignores that different models fail in different places. Combining them can reduce errors if their residuals are not too correlated.
The Approach
Focus on error diversity. Generate candidates from different families, feature views, training variations, or assumptions. Verify in held-out (out-of-fold) predictions before learning blend weights. Start with a simple average; only move to weighted blending/stacking if the held-out gain justifies extra latency, maintenance, monitoring, and explainability cost.
Visual Summary
Outcome
Strong baseline → complementary candidates → verify diversity → simple average first → evaluate as its own model on held-out predictions → accept only if the real decision gain justifies operational complexity.
Key Takeaway
Design insight: The best ensemble is not the one with the best individual models — it is the one whose errors combine well enough to improve the real decision. Error diversity drives the gain; evaluate on held-out predictions and reject if operational cost outweighs measured improvement.