AI production readiness: security, observability, and operational control

Decision summary

An AI feature is production-ready only when its owner can describe what it may do, prove how that authority is enforced, detect when behavior leaves the expected envelope, and contain the result. Model quality is one dimension. Identity, dependency behavior, data handling, operational evidence, and recovery are equally part of acceptance.

The readiness standard should scale with consequence. A drafting assistant that cannot publish may need output review and privacy controls. An agent that changes records or contacts customers needs scoped credentials, approval rules, idempotency, hard budgets, traceable state, and tested incident procedures.

SOURCES: [1] OWASP GenAI Security Project · [2] OWASP

Map trust boundaries before controls

Draw the path from user and trigger through context sources, model providers, tool gateways, internal systems, external APIs, queues, reviewers, logs, and storage. Mark where identity changes, where untrusted content enters, where data leaves a boundary, and where an irreversible effect can occur.

This map prevents a common error: securing the prompt while leaving the execution path broad. An instruction may be harmless until a tool interprets it with a powerful credential. Conversely, a model error can be contained when deterministic authorization and validation sit between proposal and execution.

• Users, service identities, and delegated authority
• Prompts, retrieval sources, memories, and tool-returned content
• Model and embedding providers
• Tool gateway, queues, databases, and external APIs
• Approval and exception interfaces
• Traces, metrics, logs, evaluation records, and retention boundaries

SOURCES: [1] OWASP GenAI Security Project · [6] OpenAI · [3] National Institute of Standards and Technology

Least agency as the central security pattern

OWASP describes excessive agency as a combination of excessive functionality, permissions, or autonomy. Reduce all three. Give the system only the operations needed for the named job, scope each credential to the relevant objects and actions, and require independent approval before material side effects.

Prefer purpose-built operations such as “create draft for case 123” over open-ended database, browser, shell, or messaging access. Enforce rate, turn, time, concurrency, and spend budgets outside the model. A model should not be able to expand its own role, mint broader credentials, bypass approval, or silently convert a read workflow into a write workflow.

SOURCES: [2] OWASP · [1] OWASP GenAI Security Project

Observability that explains behavior

A useful trace relates the top-level agent invocation to model calls, retrieval, tool execution, handoffs, guardrail decisions, approvals, errors, and final state. OpenTelemetry's GenAI semantic conventions provide a vendor-neutral direction for spans and metrics such as model identity, token use, finish reasons, latency, and tool execution.

Full prompts, documents, tool arguments, and results may contain credentials, personal information, customer data, or adversarial text. OpenTelemetry keeps sensitive content off by default, and OpenAI provides controls over sensitive trace capture. Capture enough metadata and structured evidence to investigate a run without turning the observability system into a secondary data leak.

• Correlation from user request to every model and tool span
• Model and configuration identity
• Tool name, authorization result, duration, status, and idempotency key
• Approval actor, decision, and evidence
• Tokens, latency, retries, finish reasons, and budget exhaustion
• Outcome, escalation, and final state

SOURCES: [4] OpenTelemetry · [5] OpenAI Agents SDK

Failure modes and limits

NIST's adversarial machine-learning taxonomy includes evasion, poisoning, privacy, and misuse attacks and emphasizes that mitigations have limits. Agentic systems add memory and context poisoning, identity abuse, insecure agent communication, tool exploitation, code execution, and cascading failures. Defense therefore needs multiple layers and an assumption that some controls will fail.

Prompt injection cannot be solved by asking the model to ignore malicious instructions. External content must be treated as untrusted data, and the equivalent-human test is useful: what systems and actions should a human temporary worker performing this job actually be allowed to access without separate approval?

SOURCES: [3] National Institute of Standards and Technology · [1] OWASP GenAI Security Project · [6] OpenAI

Operational readiness and incident response

Define observable service objectives for task success, serious-failure rate, latency, cost, review burden, and dependency health. Alerts should route to an owner who can pause the relevant tool or workflow without taking the whole application offline. Every write path needs a documented recovery model: reverse, compensate, quarantine, or escalate.

Exercise incidents before launch. Simulate a poisoned document, invalid tool result, revoked credential, provider timeout, duplicated event, spend spike, and unsafe proposed action. Confirm that the system stops inside its budget, operators can reconstruct the run, and replay does not duplicate external effects.

SOURCES: [1] OWASP GenAI Security Project · [4] OpenTelemetry · [5] OpenAI Agents SDK

Production-readiness checklist

The evidence package should include the following controls and test results.

• Named system owner, user population, data classification, and action inventory
• Trust-boundary diagram and agentic threat model
• Tool allowlist, typed contracts, least-privilege identities, and authorization tests
• Independent approval for high-impact or irreversible actions
• Input, output, tool, state, and business-rule validation
• Hard limits for turns, time, retries, concurrency, and spend
• Sensitive-data-aware trace and retention configuration
• Representative security, failure, and dependency tests
• Pause, revoke, recover, replay, and rollback runbooks
• Change and incident processes that update the evaluation suite

SOURCES: [2] OWASP · [1] OWASP GenAI Security Project · [4] OpenTelemetry · [3] National Institute of Standards and Technology

How Nautilus can help

Nautilus can assess an AI system against its actual authority and operating environment, produce the threat model and control architecture, implement observability and bounded execution, and run production-readiness exercises with the client's security and operations teams. Findings remain tied to evidence and remediation rather than a generic “AI-safe” label.