The AI agent evaluation playbook: from demonstrations to release evidence

Decision summary

An evaluation is useful only when it supports a decision. Begin with a falsifiable claim such as: “For these support cases, the system prepares an evidence-linked draft that a reviewer accepts without material correction, while never issuing a refund or changing an account without approval.” That claim determines the cases, annotations, graders, telemetry, and threshold.

Do not collapse success into one average score. Production decisions usually require several gates: task correctness, policy compliance, tool-path validity, escalation behavior, latency, cost, reviewer effort, and serious-failure rate. A system can improve average answer quality while becoming less safe or less economical.

SOURCES: [1] OpenAI · [4] OpenAI

Build a representative task suite

Sample work from the intended operating environment rather than writing only clean examples. Preserve the shape of the incoming distribution: common cases, long-tail exceptions, ambiguous requests, incomplete records, conflicting evidence, tool outages, and adversarial content. Attach the expected result, permitted path, forbidden actions, and escalation rule to each case.

Keep separate development, regression, and holdout sets. Development cases help the team understand failures. Regression cases preserve known lessons. Holdout cases reduce the temptation to tune the system to a visible test. Refresh the suite when workflow policy, data, users, tools, or model behavior changes.

• Typical cases weighted by real operating volume
• High-impact cases weighted by consequence, not frequency
• Boundary cases near policy and permission thresholds
• Adversarial instructions embedded in retrieved or tool-returned content
• Timeouts, malformed results, stale state, partial execution, and duplicate events
• Cases that should be refused, deferred, or escalated

SOURCES: [1] OpenAI · [2] OpenAI

Evaluate outcome and trajectory

Outcome grading asks whether the final state is correct. Trajectory grading asks how the system arrived there: which context it used, which tools it selected, whether arguments were valid, whether it respected approval, and whether it recovered safely. Both are necessary for an agent because the path itself can create risk or cost.

OpenAI's trace grading guidance describes structured labels or scores applied to an end-to-end execution log. That provides diagnostic evidence that a black-box final-answer score cannot: a failure may come from retrieval, planning, a tool contract, a handoff, a guardrail, or the model. The remediation should target the failing layer rather than adding undirected prompt text.

• Final-state correctness and evidence quality
• Tool selection, arguments, ordering, and unnecessary calls
• Permission and approval compliance
• Handoff and escalation correctness
• Turn, token, time, retry, and cost budgets
• Recovery from tool, network, or state failures

SOURCES: [3] OpenAI · [2] OpenAI

Use graders that match the question

Deterministic assertions are best for schemas, exact policy rules, state transitions, citations, permissions, and numeric limits. Human review is necessary where quality depends on domain judgment or user impact. Model graders can scale semantically rich checks, but they need explicit rubrics, examples, and calibration against qualified human labels.

OpenAI recommends pairwise comparison, classification, and specific criteria over unconstrained open-ended judging. Record grader disagreements instead of hiding them inside an average. Disagreement can reveal an ambiguous policy, weak rubric, unstable grader, or genuinely subjective task that needs a different operating control.

SOURCES: [1] OpenAI

Turn evaluation into an operating control

Evaluation belongs in the change process. A release candidate should be tested against the relevant suite when the model, instructions, tool definition, retrieval logic, permission, workflow policy, or dependency changes. Production sampling then checks whether the test distribution still represents reality.

Use tiered gates. A safety-critical regression can block release regardless of aggregate quality. A modest quality decline might allow a limited canary with additional review. A cost or latency change may require a routing or caching adjustment. The decision record should identify who accepted each residual risk and what would trigger rollback.

• Pre-merge checks for deterministic contracts and small regression subsets
• Pre-release full-suite and adversarial evaluation
• Canary release with heightened review and comparison to a baseline
• Production sampling for drift, new failure classes, and reviewer burden
• Incident-to-regression conversion so material failures become permanent tests

SOURCES: [1] OpenAI · [2] OpenAI

Security and compliance implications

Evaluation records can contain production prompts, retrieved documents, tool arguments, customer decisions, and examples of harmful behavior. Classify and minimize that material, restrict access, define retention, and use synthetic or de-identified cases where they preserve the behavior under test. An evaluation system should not become an ungoverned copy of sensitive production data.

A passing evaluation is evidence for a defined claim and configuration; it is not certification or a general compliance conclusion. Map each test to a risk, control, requirement, or release decision, preserve the tested model and system version, and make residual risk acceptance explicit. NIST AI RMF treats measurement as one part of a wider Govern, Map, Measure, and Manage process.

SOURCES: [1] OpenAI · [5] National Institute of Standards and Technology

Failure modes and limits of evaluation

A high score can be misleading when the test cases are unrepresentative, the harness differs from production, the model has seen the benchmark, the grader rewards style instead of correctness, or retry and tool budgets are omitted. OpenAI's 2026 discussion of third-party evaluations emphasizes that model, reasoning settings, tools, safeguards, budgets, and elicitation all affect the result.

Evaluation reduces uncertainty; it does not prove universal reliability. Rare harmful behavior is especially difficult to estimate from small samples. Preserve serious failures as named events, use consequence-aware thresholds, and keep human review or hard controls where the residual risk is unacceptable.

SOURCES: [4] OpenAI · [1] OpenAI

Worked example: a customer-operations agent

Suppose an agent investigates a billing complaint and drafts a proposed resolution. The evaluation record includes the customer request, account state, policy version, expected evidence, acceptable resolution classes, approval requirement, and prohibited actions. Some cases contain misleading instructions inside customer-supplied attachments; others simulate a billing API timeout or a stale account snapshot.

Deterministic graders confirm that the account was not changed, required citations resolve, and approval was requested when a credit crossed policy. A domain reviewer scores factual synthesis and customer usefulness. A trace grader checks whether the correct systems were consulted and whether the agent ignored untrusted instructions. Release requires no prohibited action, a defined acceptance rate, bounded review effort, and stable latency and cost.

SOURCES: [2] OpenAI · [3] OpenAI · [1] OpenAI

Implementation checklist

Before a production decision, confirm that the evaluation program has the following properties.

• A written claim, population, baseline, and release decision
• Typical, edge, adversarial, refusal, escalation, and dependency-failure cases
• Expected outcomes plus permitted and forbidden trajectories
• Deterministic checks for permissions, schemas, state, and budgets
• Human-calibrated semantic graders with recorded disagreements
• Separate development, regression, and holdout sets
• Reported model, tools, reasoning configuration, retries, and budgets
• Change-triggered regression testing and production sampling
• A path from incidents and reviewer corrections back into the suite

SOURCES: [1] OpenAI · [4] OpenAI · [3] OpenAI

A note on the OpenAI Evals platform

OpenAI's current evaluation guidance states that the legacy Evals platform will become read-only on October 31, 2026 and shut down on November 30, 2026. The evaluation discipline remains valid, but new programs should use current datasets and graders or a portable internal harness rather than create a hard dependency on the retiring platform.

SOURCES: [1] OpenAI

How Nautilus can help

Nautilus can turn a workflow and risk model into a representative task suite, implement deterministic and trace-based graders, establish release thresholds, and integrate evaluation with delivery and production review. The goal is a client-owned evidence system that continues to work when models or providers change.