Build, buy, automate, or stop: an AI decision framework

Decision summary

Write the decision in operational terms: who does what work today, at what volume, delay, error rate, cost, and risk; what outcome must change; and what evidence would justify investment. If the baseline is unknown, a projected AI return is usually precision without measurement.

Then compare real alternatives. A policy clarification, interface improvement, data cleanup, rules engine, workflow integration, or existing product may solve the bottleneck with less uncertainty. Custom AI or an agent is warranted only when model-dependent judgment creates enough additional value to cover its evaluation and operating burden.

SOURCES: [1] OpenAI · [4] National Institute of Standards and Technology

Frame the job before the solution

Capture the trigger, actor, input, decision, systems touched, completion state, exceptions, and consequence of error. Separate repetitive handling from genuine judgment. Identify which information already exists in structured form and where policy is ambiguous or contradictory.

Define a baseline and a target range for quality, cycle time, review, cost, adoption, and serious failures. Include a “do nothing yet” option. Deferral can be rational when provider capability, regulation, data readiness, or the underlying process is changing faster than the organization can responsibly absorb.

SOURCES: [4] National Institute of Standards and Technology · [5] National Institute of Standards and Technology

Compare seven options

The alternatives are not mutually exclusive. A strong design often combines policy change, deterministic software, and a narrow model-assisted step.

• Fix the process: clarify policy, ownership, incentives, or handoffs.
• Build conventional software: encode stable rules, state, calculations, and interfaces.
• Automate the workflow: connect systems with events, APIs, queues, and human exception handling.
• Buy a packaged AI product: adopt an existing workflow with vendor controls and constraints.
• Integrate model capability: add classification, extraction, retrieval, or generation inside owned software.
• Build an agent: allow bounded planning and tool selection where variation requires it.
• Defer or stop: preserve evidence and revisit only when named conditions change.

SOURCES: [1] OpenAI

Score value, feasibility, risk, and reversibility

Value includes avoided delay, better decisions, quality, capacity, user experience, or new revenue—but should account for adoption and the part of the workflow that remains manual. Feasibility covers data access, integration, task performance, latency, ownership, and the team's ability to operate the result.

Risk includes impact on people, security, privacy, compliance, reliability, and reputation. Reversibility asks whether the organization can change models, providers, permissions, and workflow without losing its data, evaluations, or operating knowledge. A lower-performing but reversible option can be strategically superior to a tightly coupled demonstration.

SOURCES: [4] National Institute of Standards and Technology · [6] National Institute of Standards and Technology

Architecture, security, and compliance implications

The option changes the control boundary. Buying a product transfers some implementation work but creates a vendor, data, identity, evidence, and exit dependency. A custom integration preserves more control but adds delivery and operational ownership. An agent adds variable execution paths and therefore needs narrower tools, explicit authority, evaluation of trajectories, and stronger recovery.

Before selecting an option, map data flows, user and service identities, actions, affected people, provider roles, retention, logging, human oversight, and the consequence of error. Determine legal applicability separately with qualified advisers. A voluntary framework or provider statement can inform the decision but does not establish that the deployed system is compliant.

SOURCES: [1] OpenAI · [4] National Institute of Standards and Technology · [6] National Institute of Standards and Technology

Price the complete operating system

Total cost includes discovery, process redesign, data preparation, integration, identity, evaluation, security, review, observability, incident response, exceptions, training, provider use, and maintenance. Model use may be visible while reviewer and exception costs remain hidden.

Estimate ranges under realistic volumes and failure rates. Include the cost of a serious error, provider outage, model migration, and a manual fallback. Do not treat a pilot's clean inputs or engineering attention as the normal production environment.

SOURCES: [7] DORA · [2] OpenAI

Run a decision-changing proof sprint

The proof should target the most important uncertainty, not maximize feature count. If task quality is unknown, test representative cases. If integration or permission is the risk, build the narrow end-to-end path. If review load determines economics, measure reviewer corrections and time.

Define pass, redesign, and stop conditions before results arrive. Use a baseline and holdout cases. Record model, tools, instructions, retries, time, cost, and human assistance so that the outcome is not mistaken for a property of the model alone.

SOURCES: [2] OpenAI · [3] OpenAI

Failure modes and limits

Common decision failures include using a benchmark as a task test, accepting a vendor demonstration as production evidence, ignoring exception handling, valuing generated output instead of accepted workflow outcomes, and treating sunk pilot effort as a reason to continue.

Another failure is false convergence: choosing one provider too early or building a multi-agent platform before proving the first job. Capabilities and terms change; the durable assets are workflow understanding, representative cases, permission design, integration contracts, and operating evidence.

SOURCES: [1] OpenAI · [2] OpenAI · [3] OpenAI

Worked example: inbound document operations

A company wants an agent to process inbound documents. The baseline shows that most documents use three stable templates; a minority are ambiguous exceptions. The defensible design uses deterministic parsing and validation for the templates, a model to classify or extract the long tail, and human review when confidence or policy conditions fail.

The proof samples the real distribution and measures accepted extraction, manual corrections, latency, and exception cost. A packaged product and a thin custom integration are compared against a fully custom agent. The company can now decide with evidence, and it may conclude that a hybrid automation creates most of the value with less autonomy.

SOURCES: [1] OpenAI · [2] OpenAI

Implementation checklist

The decision record should contain the following.

• Current workflow, baseline, owner, volume, exceptions, and consequence of error
• Measurable target and the conditions for doing nothing
• Comparison of process, software, automation, buy, integrate, agent, and defer options
• Value, feasibility, risk, reversibility, and full operating-cost ranges
• Representative proof cases and a manual or current-system baseline
• Predefined pass, redesign, and stop conditions
• Provider, data, permission, security, compliance, and operational dependencies
• Ownership of prompts, evaluations, code, data, traces, and migration artifacts
• A phased roadmap that requires evidence before increasing scope or authority

SOURCES: [4] National Institute of Standards and Technology · [5] National Institute of Standards and Technology · [2] OpenAI

How Nautilus can help

Nautilus can map the workflow, compare the alternatives, build the decision-changing proof, and produce an executive recommendation with architecture, costs, risks, evidence, and stop conditions. A recommendation to buy, simplify, defer, or stop is an acceptable engagement outcome.