Should this be a rule, a model, or an LLM?
One of the most useful AI architecture questions in 2026 is still very simple: should this be a rule, a model, or an LLM? Many teams start with the most powerful tool — that is often the wrong starting point. A simpler decision framework works better.
Visual Summary
The Core Question
One of the most useful architecture questions in 2026 is still very simple: should this be a rule, a model, or an LLM?
Many teams start with the most powerful tool. That is often the wrong starting point. The most expensive tool in the stack is not always the best fit — it is the one that is easiest to justify in a demo and hardest to run in production.
A simpler decision framework works better: match the tool to the nature of the problem, its latency budget, and who owns the logic.
Use RULES when...
- Logic is simple and deterministic
- You can enumerate all cases
- Explainability is non-negotiable (regulatory)
- Speed matters (nanoseconds, not seconds)
- The business owns the logic (not data-driven)
Example
if amount > 10,000€ AND country = high-risk → flag for review
Rules are cheap, transparent, and auditable. If you can enumerate the cases up front, there is no reason to train anything.
Use a MODEL (ML) when...
- Patterns exist in historical data
- Rules would be too many or too complex
- You need to RANK or SCORE (not just yes/no)
- You can validate with ground truth
- Latency budget: milliseconds
Example
Predict churn probability for each customer based on 50 behavioral features
A model shines when the decision is a score, the patterns live in the data, and you have labels to validate against. It generalizes where a hand-written rule table would collapse under its own complexity.
Use an LLM when...
- Input is unstructured (text, documents, images)
- The task requires REASONING or INTERPRETATION
- Rules are impossible to enumerate (too many edge cases)
- You need FLEXIBILITY (new categories without retraining)
- Latency budget: seconds (acceptable)
Example
Extract contract parties, obligations, and deadlines from legal documents
An LLM earns its place when the input resists a fixed schema and the task is interpretation, not arithmetic. It buys flexibility at the cost of latency, cost, and guardrails.
Where many teams go wrong
- Using an LLM for something a rule could do cheaply
- Forcing rules onto something that should be learned from data
- Using ML on unstructured reasoning tasks it is not shaped for
Each tool has a natural lane. The failure mode is picking the tool first and the problem second.
The hybrid (production reality)
In production, most systems are layered. Each layer handles what it is best at and escalates only when it cannot decide:
Input → Rules (fast, cheap, deterministic)
↓ (rules can't decide)
→ Model (scored, ranked, data-driven)
↓ (model confidence low)
→ LLM (reason about the edge case)
↓ (LLM uncertain)
→ Human review
Not every system needs all four layers, but the principle holds: start with the simplest tool that fits the problem, and escalate only when it stops being enough.
Key Takeaway
Design insight: Start with the simplest tool that fits the problem — a rule, then a model, then an LLM, then a human. Escalate only when it stops being enough. The cheapest correct architecture is the one that wins in production.