When an AI project should take a simpler route or stop

Why AI projects rarely stop on their own

Every iteration of an AI project produces a small improvement and a plausible next idea: a better prompt, another tool, a newer model. Demos tend to show the cases that work. Without dates and thresholds, nobody is responsible for saying that the project has reached its limit. The NIST AI Risk Management Framework treats this as an explicit decision: whether the system achieves its intended purpose and whether its development or deployment should proceed. Someone has to make that determination on a schedule.

Signals that call for a decision

• Quality has plateaued below the required level for several iterations, especially on the case types that matter most. Before concluding this, read failure transcripts. Anthropic's evaluation guidance notes that a grader can reject valid answers, so a plateau may be a measurement problem.
• Single runs pass but repeated runs disagree, so results cannot be relied on case by case.
• The cost per handled case stays above today's process at measured review and escalation rates, with no credible lever left. The FinOps Foundation describes unit costs as a way to spot where costs outpace the value delivered.
• Review effort is not falling, because reviewers still redo the work rather than check it.
• Complexity grows faster than results: each fix adds a step, an agent or a special case. Anthropic's 2024 article on agents advises adding complexity only when it demonstrably improves outcomes.

Options between continuing and stopping

1. 1

   Narrow

   Keep the case types that meet the bar and route the rest to people.

2. 2

   Simplify

   Replace an agent with a fixed workflow or a single model call, or use rules with a model only for exceptions. NIST asks that viable non-AI alternatives be considered as well.

3. 3

   Change approach

   Try a different model, retrieval design or fine-tuning only when failure analysis points to a cause that change would address.

4. 4

   Pause

   Wait for something specific, such as recorded outcomes, an API or an approval, with a date to look again.

5. 5

   Stop

   Switch off and retire the system, with the responsibilities NIST describes for deactivating systems whose outcomes do not match their intended use.

A hypothetical month-four review

Assume a project that routes incoming support tickets and drafts replies. The agreed bar was 90 percent correct routing. After four months, routing reaches 93 percent on routine tickets but has stayed around 60 percent on billing disputes for three iterations, and transcripts show the disputes need account history the system cannot see. Agents rewrite most drafted replies, so review takes about as long as writing them. Measured cost per handled ticket is slightly above today's process.

The sponsor narrows and stops in the same meeting. Routing continues for routine tickets. Billing disputes go to people until account history is available through an approved API, which becomes a pause with a date. Drafting stops. The cost per case is measured again six weeks later on the narrower scope, and that number decides the next review.

Taking the decision without sunk-cost pressure

Set the review dates and thresholds when the project starts, so that the conversation is about evidence rather than effort. At each review, ask whether you would fund the next stage today, knowing what you now know, and leave money already spent out of the answer. Include someone who is not on the project team. A documented stop is a result: the test set, the failure analysis and the cost model all inform the next attempt.

Retiring what you stop

NIST's framework asks for processes to phase out AI systems safely. In practice that means revoking the service accounts and API keys, deleting or archiving data according to the agreed retention, telling users what replaces the system, and filing the test set and the decision record where the next team will find them. When we build process agents and a plain workflow would do the job, we build that instead.

What an AI pilot must proveThe full cost of an AI workflow