AI-assisted software development: an operating model beyond code generation

Decision summary

Adopt AI-assisted development as a controlled change to the engineering operating model, not an individual autocomplete procurement. Define which work is eligible, what context may be shared, how generated changes are validated, who owns the result, and which delivery and quality outcomes will determine expansion.

The central question is whether the team produces accepted, maintainable software faster without increasing downstream instability or risk. Faster first drafts are valuable only when review, integration, testing, security, deployment, and support do not absorb the gain.

SOURCES: [1] DORA · [2] Google Cloud · [3] METR

What the current evidence says

DORA's 2025 research drew on nearly 5,000 respondents and more than 100 hours of qualitative data. Ninety percent reported using AI at work, more than 80% perceived productivity gains, and 30% reported little or no trust in AI-generated code. The study found positive associations with delivery throughput and product performance, alongside a negative association with delivery stability.

These are primarily survey and observational results, not proof that AI caused each outcome. DORA's “amplifier” framing is the useful operating conclusion: strong platforms, workflows, tests, version control, and feedback loops can turn local assistance into better delivery, while weak systems can accelerate defects and instability.

SOURCES: [1] DORA · [2] Google Cloud

What the METR studies do—and do not—show

METR's randomized early-2025 study found that experienced open-source developers working in repositories they already knew took 19% longer when allowed to use the tested AI tools. That is an important counterexample to universal speed claims, but its population, tasks, repositories, and tool generation are specific.

METR's February 2026 update says its later measurement was affected by severe self-selection and time-measurement problems. METR believes early-2026 tools likely improve speed, but the reported evidence was weak and confidence intervals crossed zero. The responsible conclusion is contextual: task, developer, repository, tool, workflow, and measurement method materially affect the result.

SOURCES: [3] METR · [4] METR

Reference operating model

Set policy by work type and risk. Low-risk transformations, test scaffolding, explanation, and constrained refactoring may receive broader assistance. Authentication, authorization, cryptography, migrations, infrastructure, billing, privacy logic, and safety-critical code should receive stronger context restrictions and review.

Keep an accountable human owner for every change. The owner must understand the requirement and be able to defend the implementation. Use retrieval and repository instructions to provide current local conventions, but never assume model familiarity with the codebase is complete. Generated changes should pass the same or stronger gates as human-authored changes.

• Plan: clarify requirement, acceptance examples, risks, and allowed context.
• Generate: work in small diffs with relevant repository evidence.
• Verify: compile, lint, test, scan, inspect behavior, and examine the diff.
• Review: assess design, security, maintainability, tests, and operational impact.
• Release: canary where appropriate and observe user and system outcomes.
• Learn: convert defects, reviewer corrections, and incidents into repository guidance and tests.

SOURCES: [1] DORA

Evaluation and metrics

Compare teams or workflows against a baseline over enough time to observe downstream effects. Track accepted cycle time, change failure rate, escaped defects, rework, review time, rollback, support load, and developer experience. Segment results by task type; an average can hide strong gains in one class and serious losses in another.

For AI-enabled product or agent code, use task-specific evaluations in addition to normal software tests. OpenAI's evaluation guidance recommends real distributions, edge and adversarial cases, human calibration, failure logging, and continuous evaluation when the system changes.

• Time from accepted task to accepted production behavior
• Review effort and number of material corrections
• Test effectiveness, escaped defects, incidents, and rollback
• Maintainability and ownership after the original author leaves
• Developer confidence, cognitive load, and ability to explain the change
• Model use, latency, and cost as supporting—not primary—metrics

SOURCES: [1] DORA · [5] OpenAI

Failure modes, security, and limits

Common failures include plausible code built on a false API assumption, unnecessary architectural complexity, duplicated existing capability, weak tests that confirm the generated implementation rather than the requirement, and large diffs that reviewers cannot meaningfully inspect. Local task speed can also create a review bottleneck or increase deployment instability.

Control what source, secrets, customer data, logs, and incident material may be sent to each provider. Treat packages, commands, migrations, generated tests, and security-sensitive logic as untrusted until independently verified. AI assistance does not move accountability from the engineering organization to the model.

SOURCES: [1] DORA · [3] METR · [4] METR

Worked rollout

A platform team can begin with three bounded classes: test generation for existing behavior, migration from one approved API version to another, and small internal-tool enhancements. Establish a four-week baseline, define eligible repositories and data boundaries, train reviewers, and require small diffs with explicit acceptance tests.

Run a controlled pilot with outcome metrics segmented by task class. Review samples where the assistant was not used, inspect material corrections, and interview developers about understanding and review burden. Expand only the classes that improve accepted cycle time without degrading stability, security, or maintainability.

SOURCES: [1] DORA · [4] METR

Implementation checklist

A responsible enablement program should be able to show the following.

• Approved use cases, repositories, providers, and data boundaries
• Repository guidance and architecture context kept under version control
• An accountable human owner for every generated change
• Small-diff, test, review, security, and deployment requirements
• Extra controls for identity, billing, data, infrastructure, and safety-critical code
• Baseline and pilot measures that include stability and rework
• Task-class segmentation instead of a universal productivity claim
• A correction loop from review and incidents into tests and guidance
• A stop or rollback condition when quality or stability deteriorates

SOURCES: [1] DORA · [3] METR · [5] OpenAI

How Nautilus can help

Nautilus can assess the delivery system, establish policy and context boundaries, configure the development workflow, create representative evaluations, train teams, and measure a controlled pilot. The objective is durable engineering capability inside the client team, not dependence on a particular assistant or an unsupported productivity promise.