Production AI Launch Checklist: Evals, Security, Controls, and Monitoring

Use this checklist to decide whether a specific AI workflow is ready for a controlled production release. Complete it with the process owner, engineering lead, security owner, and operational reviewer. For each item, record verified, open, or not applicable with a reason, together with the evidence and accountable owner.

The decision applies to a defined version, user group, case scope, and level of authority. A draft-only pilot and an automatic execution workflow require different evidence. Use the AI Agent Evaluation guide to design the test set and the Production Pilot to resolve a bounded implementation uncertainty.

Gate 1: scope and success are observable

Record the included inputs, users, sources, integrations, and outputs. List excluded cases and the route they take. State the difference between a successful result, a valid escalation, and an incomplete case. Name the system record that proves completion when an action changes another application.

Document acceptance criteria before the final evaluation. Include quality, permissible behavior, response time, total reviewer effort, and exception handling. Numerical thresholds should come from the workflow's consequences and operating needs. Do not copy a benchmark score into a release criterion without explaining its relevance.

Gate 2: evaluation covers the whole workflow

Scroll horizontally to see all columns.

Anthropic's agent evaluation guidance distinguishes the transcript from the resulting environment state and describes deterministic, model-based, and human grading. Apply that distinction to your workflow: a message claiming completion is different from a verified completed action.

Gate 3: authority and information boundaries hold

List the deployed identities and their allowed operations. Test access with a user who should be denied, an object outside scope, and a revoked permission. Verify that restrictions are enforced at the data or action boundary, even when the model proposes an unauthorized operation.

Test malicious or misleading instructions in retrieved documents and tool outputs. Review outbound destinations, file handling, and logs. Record the security tests and remaining restrictions relevant to the launch. The AI Agent Security article explains these boundaries; the OWASP prompt-injection guidance describes direct and indirect instruction attacks.

Gate 4: human review is an operating workflow

Name the review queue, coverage hours, eligible approvers, escalation owner, and expiration behavior. Show reviewers the exact proposed action, current evidence, and consequences. Record approvals, edits, rejection, and cancellation separately. Test that a changed proposal cannot use approval for an earlier version.

Measure whether reviewers can actually complete the work at expected volume. A system that routes every uncertain case to an unattended queue is not operationally complete. Include reviewer effort and queue age in the release evidence. Use the human-in-the-loop controls guide to design that interaction.

Gate 5: interruption and recovery are rehearsed

Stop the worker at a meaningful point, restart it, and inspect what happens. Test a lost response after an external write, a duplicate event, a timeout, a dependency outage, and a cancelled case. Confirm which actions already occurred before permitting a retry.

Record durable business state separately from conversational context. Define retry limits, reconciliation, manual recovery, and the point at which work stops. Rehearse restoration using the actual deployment configuration. Keep the recovery evidence with the release record so the procedure can be repeated after infrastructure changes.

Gate 6: observation leads to action

Scroll horizontally to see all columns.

Logs should identify a case, release version, relevant source versions, tool outcome, and state transition without collecting unnecessary sensitive content. Access to operational evidence needs its own permissions. Confirm that alerts reach an accountable person and that the person has a usable response procedure.

Gate 7: rollout and rollback have owners

Define the initial audience, traffic or case allocation, authority, observation period, and stop conditions. A small rollout should produce useful evidence about the actual release. Google's canary-release guidance explains using a limited deployment and comparison signals to inform a broader rollout; the exact design should reflect your workflow and traffic.

Rollback is more than selecting an older model. Record application, prompt, tool, policy, and state compatibility. Identify actions that cannot be undone automatically and the repair process for them. Verify that new intake can stop while existing cases remain visible to operators.

Complete the release record

Record the release identifier, evaluation summary, tested scope, deployment configuration, owners, open issues, restricted capabilities, and the decision to release or hold. Open items should have an explicit consequence for scope; they should not disappear inside a general approval statement.

In an illustrative request-routing pilot, a release might permit classification and draft preparation for one queue while keeping record updates disabled. The release record would state that boundary and the evidence needed before enabling writes. This makes progress concrete without claiming the pilot has proved every future capability.

Connect launch to continuing operations

The pilot-to-production article develops the architecture and handoff requirements. Managed AI Operations can support a defined operating scope after launch. If decision rights remain unresolved, AI Governance & Operating Model connects those responsibilities to the organization.

Discuss an AI project with the release record and the uncertainty you need to resolve. A useful launch decision says what can operate now, how it will be observed, and what evidence will justify the next expansion.