Model selection is premature when the business has not defined what the complete workflow must accomplish.
Start with a workflow contract: a short, testable definition of responsibility, authority, and acceptance. It should remain stable even if the team later changes the model, retrieval system, or orchestration framework.
For customer onboarding, “use AI to review documents” is too vague. A useful contract might require the system to reduce preparation time, improve completeness, attach source evidence to every recommendation, keep activation under human approval, and stay below an agreed cost per completed case.
That contract determines the evaluation suite.
| Test case | What the system must prove |
|---|---|
| Expected request | Completes the workflow correctly |
| Incomplete or conflicting data | Identifies the gap and avoids inventing facts |
| Policy conflict | Applies the rule or escalates |
| Unauthorized request | Refuses the action and records the denial |
| Tool or model failure | Enters the approved degraded mode |
| Adversarial input | Protects data, tools, and policy boundaries |
Measure correctly completed workflows, correct tool selection, policy violations, false denials, human-review rate, latency, retries, rollback, and cost. Generic benchmark scores cannot replace these business-specific measures.
Run the suite whenever a model, prompt, retrieval configuration, tool, agent skill, or policy changes. If a change cannot pass the same contract, it is not a safe production release.
Cayru helps product and engineering leaders turn broad AI ideas into bounded workflows, acceptance criteria, and reusable evaluation harnesses.
Define the first production workflow before investing further in models or infrastructure.