Use the Smallest Model That Reliably Finishes the Job

The most capable model is a useful benchmark. It is rarely the right default for every request.

Routine classification, extraction, formatting, and policy lookup may not need frontier-level reasoning. Complex planning, ambiguous decisions, or high-consequence analysis may. Sending both through the same expensive path creates unnecessary cost and latency—or unnecessary quality risk if the default model is chosen only for price.

OpenAI's current model-selection guidance recommends optimizing for accuracy until the use case reaches its target, then maintaining that accuracy with the cheapest and fastest suitable model. That order matters.

Establish the quality floor

Define what “reliably finishes” means for the task:

• required accuracy or completion rate;
• prohibited error types;
• escalation behavior;
• latency threshold;
• maximum cost per successful outcome; and
• performance on difficult and adversarial cases.

Start with a capable model to learn what good looks like and to produce high-quality examples. Then compare smaller models, lower reasoning effort, prompt changes, retrieval improvements, or distillation against the same evaluation set.

Route by task, not by user

One workflow may contain several levels of complexity.

A support process could use:

1. deterministic rules to validate identifiers;
2. a small model to classify intent;
3. retrieval to assemble policy and account context;
4. a stronger model for ambiguous resolution; and
5. a person for exceptions with material consequence.

Routing should depend on measurable signals such as task type, ambiguity, risk, context size, required tools, and confidence—not on a broad assumption that one department deserves a stronger model.

Make fallback explicit

If the primary path fails, the system should know what happens next:

• retry with corrected context;
• escalate to a stronger model;
• reduce capability;
• queue the request; or
• route to a person.

Do not silently switch to a weaker model for high-risk work merely to preserve availability.

Log routing decisions so operators can explain which path handled a case and why.

Audit the rule against outcomes, not assumptions.

Watch total workflow cost

A smaller model is not cheaper if it creates more retries, longer prompts, manual rework, incorrect actions, or support cases.

Measure the full path:

Model + retrieval + tools + infrastructure + review + exception handling ÷ successful outcomes

This also prevents a large model from looking expensive when it eliminates several failed intermediate calls.

Preserve portability where it earns its cost

Keep model-specific behavior behind a tested interface and rerun the evaluation suite when providers or versions change. Do not build an elaborate multi-provider abstraction before a business need exists. Portability has value for continuity, negotiation, regulatory needs, or material cost differences—not as architecture theater.

The right model strategy is a tested portfolio, not a permanent vendor ranking.

Cayru helps companies build evaluation-based routing and LLMOps controls that balance quality, latency, resilience, and complete workflow economics.

Baseline one AI workload and test whether a smaller model can meet its real acceptance threshold.

Related: Use evaluation-driven development and manage AI through a product P&L.