AI Agent Evaluation: How to Test Production AI Systems

AI agent evaluation should establish whether a complete workflow produces an acceptable outcome under realistic conditions. A polished response is one piece of evidence. If the system retrieves sources, calls tools, changes records, or asks for approval, the evaluation must also inspect those behaviors and the state left behind.

The most useful starting point is a bounded business task with observable completion. Define the intended result, prohibited actions, acceptable escalation, and the evidence required to judge each. Then build a repeatable way to run representative cases against the actual model, prompts, tools, policies, and application configuration.

This guide uses an illustrative internal request-resolution workflow. Its examples and scorecard are proposed evaluation methods, not measured client outcomes or universal release thresholds. Use the structure to design evidence for your own Production Pilot.

Define the unit of evaluation

A test case should include the request, starting environment, available evidence, permitted actions, and acceptance criteria. A trial is one run of that case against a specific configuration. The result includes the generated output, relevant execution events, and final business state. Keep these concepts separate so a good-looking transcript cannot substitute for a verified outcome.

Anthropic's agent evaluation guidance makes this distinction between the transcript and resulting environment state, and describes code-based, model-based, and human graders. It also emphasizes repeated trials because behavior can vary. These are useful foundations; the workflow owner still needs to define what counts as acceptable work.

For example, a request-resolution case may require a correct draft with permitted sources and no external write. A later execution version may require an approved update confirmed in the target application. Reusing the same answer-quality rubric for both would leave the most important new capability untested.

Record the release configuration with every run. Include model and serving settings, prompt version, tool definitions, relevant policy version, retrieval configuration, and dataset version. Otherwise, a score difference may be impossible to attribute or reproduce well enough for a decision.

Build cases from the operating reality

Start with representative cases from the intended workflow, using data you are permitted to process for evaluation. Preserve the features that make them difficult: missing information, conflicting records, long documents, unusual formats, and ambiguous identifiers. Removing every complication during preparation can create a dataset that measures a cleaner task than the application will receive.

Separate frequent cases from consequential exceptions. A rare unauthorized action should not disappear inside a high average completion rate. Define slices for supported request types, source conditions, user permissions, and dependency states. Report the volume and outcomes for each slice so the team can see where the evidence is strong or thin.

Include cases where the correct result is to stop, ask for clarification, or route work to a person. Those outcomes should be judged as successful handling when the specification calls for them. At the same time, measure unnecessary escalation so a system cannot achieve apparent safety by refusing useful work indiscriminately.

Maintain a held-out set for decision-making and a development set for iteration. Record when a case becomes familiar to the team through repeated debugging. A fixed regression bank is valuable for preserving known behavior, but it is not the only evidence needed to understand generalization to new requests.

Specify outcomes before choosing graders

Scroll horizontally to see all columns.

Use independent checks where the environment exposes a definitive result. If the task must create a record, inspect the record. If the task must never send a message, inspect the action log or controlled test adapter. A language-model judge can assess nuanced writing quality, but it should not replace a direct check of an observable system change.

For subjective criteria, write a rubric with concrete examples. “Good answer” is not enough. Specify completeness, support, handling of uncertainty, and relevance to the user's request. Have domain reviewers score a sample independently and resolve disagreements. This exercise often reveals that the product requirement needs clarification.

Calibrate automated judgments

A model-based grader can help scale a well-defined review task, but its agreement with the intended human judgment must be checked. Give it the evidence required to make the judgment and evaluate it against reviewed cases. Inspect false passes as well as false failures. A grader that rewards polished language may overlook missing evidence or incorrect actions.

Keep the agent's output separate from trusted grading instructions. Outputs and retrieved documents may contain text that attempts to influence the evaluator. Treat that text as material to inspect, not as a change to the rubric. Where possible, use structured checks and protected test fixtures for the properties that determine release authority.

Version the rubric and grader configuration. When the scoring method changes, rerun the comparison baseline rather than presenting old and new scores as directly equivalent. Record disagreement rates and the cases that required adjudication. The evaluation process itself needs enough transparency for the release owner to understand its limitations.

Do not combine every dimension into a single number too early. A weighted average can conceal an unacceptable permission failure behind strong style and speed scores. Decide which properties are hard release conditions and which are tradeoffs to compare. Keep those decisions visible in the scorecard.

Test trajectories without forcing one exact path

Agents may reach the same valid outcome through different sequences. Evaluate required and prohibited behavior without unnecessarily requiring one complete transcript. For example, a case may require identity verification before any write, while allowing several legitimate ways to find supporting documentation.

Inspect the trajectory when it explains a failure or verifies a control. Did the system repeatedly query the same source without new information? Did it ignore a tool error? Did it attempt a write before approval? Did it use an inaccessible source? These observations can identify architecture problems that answer grading alone misses.

The agent architecture guide helps separate orchestration, model, retrieval, and tool responsibilities. Use that decomposition to classify failures. A missing document is different from a retrieved document being misread; a rejected API request is different from a model selecting the wrong action.

Prefer assertions about meaningful state transitions and business outcomes over brittle checks for a particular phrase. Exact output matching is appropriate for a required code or schema value, but often inappropriate for a correct natural-language explanation. Choose the grading method based on what the criterion actually means.

Exercise the boundaries and failure paths

Build controlled tests for revoked access, unsupported records, malformed tool parameters, and requests outside the first-release scope. Include misleading instructions inside source content and tool responses. OWASP's prompt-injection guidance describes direct and indirect attacks; translate relevant attack paths into cases for your application's actual tools and data.

Test a lost acknowledgment after a write, a duplicate event, and a dependency timeout. Confirm whether the application reconciles the outcome before repeating the operation. These tests belong in a safe environment with controlled adapters or fixtures. They should verify the real recovery contract without making uncontrolled changes to business systems.

Evaluate review behavior as part of the system. Use cases with stale proposals, expired approvals, unavailable reviewers, and rejected actions. Verify that a changed payload cannot inherit approval for an earlier proposal. The human-in-the-loop guide explains the review design behind these checks.

Record what is simulated and what is exercised against a real integration. A test double can prove application behavior for a specified API contract, while a sandbox integration check can reveal that the contract differs from the real service. Both can be useful, but they support different conclusions.

Use repeated trials and honest denominators

Run enough trials to reveal material variability in the cases that matter. Report the number of distinct cases and the number of attempts separately. Ten runs of one easy case do not provide the same coverage as ten distinct operating conditions. Preserve the failed attempts; selecting the best result from several runs changes the question being measured.

Define the denominator for every rate. Accepted completion may be reported across all supported cases, while reviewer acceptance may apply only to drafts sent for review. Those rates cannot be substituted for each other. If a case times out or fails before grading, retain it in the appropriate outcome category instead of silently dropping it.

For a small pilot, exact counts may be more informative than a percentage with many decimal places. Report the observed failures and the uncertainty caused by limited coverage. Expand sampling to resolve a specific remaining decision, such as whether a difficult request type is ready for the initial audience.

Compare candidates on the same cases and conditions. If one configuration gets a different tool budget, a fresher index, or a more permissive reviewer, document that difference. It may be a legitimate design choice, but the evaluation should show the complete configuration being compared.

Worked example: evaluate a request-preparation pilot

Suppose the first release prepares internal service responses but cannot update account permissions. The team defines acceptable completion as a relevant draft, supported by permitted sources, with unresolved identity or policy conflicts surfaced for review. The no-write boundary is a release condition.

The case set includes ordinary requests, duplicate names, conflicting policy versions, missing source documents, and requests from users with different access. Deterministic checks verify that no write adapter was invoked and that cited document identifiers came from the permitted evidence set. Domain reviewers judge whether the sources actually support the recommendation.

The team then inspects reviewer effort. A draft may be accurate but still require extensive reconstruction because it omits the facts needed for a decision. That is a product-quality finding, not merely a writing preference. The team can revise the review packet and rerun the same cases to determine whether the change helps.

Before enabling writes in a later release, the evaluation expands. It adds proposal versioning, approval eligibility, target-record validation, execution confirmation, and interrupted-action recovery. The draft pilot remains useful evidence, but it does not prove the newly introduced authority.

This worked example illustrates why evaluation should evolve with scope. Each expansion changes what the system can affect and therefore what must be verified. A good scorecard makes the newly required evidence explicit rather than carrying a general “pilot passed” label into production.

Tie measurements to business value

Measure the end-to-end work needed to produce an accepted result. Include preparation, review, correction, exception handling, and recovery. Lower model cost or faster generation can be useful engineering improvements, but neither establishes a better business outcome if downstream effort rises.

Connect the evaluation to the intended benefit. If the objective is faster response, measure time from arrival to accepted completion. If it is additional capacity, measure throughput at the required quality and review coverage. If it is consistent handling, inspect variation across teams, case types, and source conditions.

Keep expected benefits separate from observed results. A pilot may demonstrate reduced effort per case without proving a change in revenue or total staffing cost. Explain the operational step needed to realize the benefit, such as using released capacity to serve more demand. The ROI estimation guide develops that distinction.

Make release criteria and ownership explicit

Write the release decision against the actual version, audience, supported cases, and authority. Summarize evaluation results by dimension, record unresolved issues, and state how each issue affects scope. A restriction such as draft-only operation should be enforced in the deployed application and visible in the operating record.

Assign owners for the dataset, grading criteria, tool contracts, source freshness, and production feedback. Define which changes trigger which regression checks. A new model, retrieval pipeline, policy, integration, or permission rule can change behavior even if the user interface remains identical.

Use the Production AI Launch Checklist to connect evaluation with recovery, monitoring, and rollout. Where responsibility for approval and release decisions is unclear, AI Governance & Operating Model addresses that organizational question without replacing the technical test evidence.

Read a scorecard without hiding the failures

Imagine a small illustrative comparison with forty distinct cases. Candidate A produces thirty accepted completions, six appropriate escalations, two incorrect completions, and two unresolved technical failures. Candidate B produces thirty-two accepted completions, four appropriate escalations, one incorrect completion, and three technical failures. These invented counts demonstrate scorecard interpretation; they are not benchmark or client results.

The second candidate has more accepted completions, but that fact alone does not settle the release. Inspect the incorrect completion and the technical failures. If the incorrect completion crossed an authority boundary, it may require holding the affected capability regardless of the aggregate improvement. If the technical failures occurred only in an excluded dependency configuration, they may inform a scope restriction instead. The decision depends on what happened and what the release is intended to permit.

Now add effort. If Candidate B's accepted drafts require substantially more review, the additional completions may not improve total operating capacity. Record reviewer time and rework using comparable cases, then identify the bottleneck. The appropriate response could be a better evidence packet rather than a different model.

Also inspect coverage. Forty cases concentrated in one request type provide limited evidence for other types. Report the slices represented and the conditions not yet exercised. A release can remain useful by limiting its initial scope while collecting the missing evidence. This is more precise than presenting a single success percentage as a general property of the system.

Keep the scorecard close to the case records. Reviewers should be able to trace an aggregate result back to the failed task, configuration, expected outcome, and observed state. That connection turns measurement into an engineering decision rather than a presentation statistic.

Keep evaluation useful after launch

Production observations should feed new cases into the evaluation process. Capture incorrect accepted results, unnecessary escalation, source gaps, integration failures, and reviewer corrections with appropriate access and retention. Reproduce the relevant conditions, decide the intended behavior, and add a regression case when it will prevent a meaningful recurrence.

Avoid treating every user preference as a global rule. Investigate whether the issue reflects a product requirement, a local workflow variation, a source problem, or an isolated error. That classification helps the team change the correct part of the system and preserve behavior that already works.

Managed AI Operations can support this continuing cycle within an agreed scope. The objective is a maintained evidence base for the workflow: what it handles well, where it stops, what changed, and why the next release is justified.