Long-Running AI Agents: Durable Workflows, State, Retries, and Recovery

Long-running AI agents need an execution model that survives waiting, interruption, and partial completion. A workflow may spend minutes gathering evidence, hours awaiting approval, or days coordinating with an external process. Keeping a model conversation open is not enough to establish what the business process has already done.

The design needs durable state, explicit action records, bounded retries, and a way to reconcile uncertain outcomes. It also needs ownership for cases that cannot recover automatically. These requirements become especially important when the system can change records, send communications, or create commitments outside its own database.

This guide develops an illustrative service-request workflow that prepares a proposal, waits for approval, and updates a target system. The sequence is an engineering example, not a client result. Use it to identify the failure paths your own implementation must handle.

Define what must survive

Start by listing the facts an operator would need after a worker stops. These usually include the case identifier, current stage, input version, relevant evidence, proposed action, approval decision, attempted operation, external confirmation, and next permitted step. Store them as explicit business state.

Conversation history can help reconstruct why a proposal was made, but it should not be the sole authority for whether a payment was submitted or a record changed. Generated summaries can omit details or combine old and new facts. The application needs a reliable record of state transitions and the evidence behind consequential actions.

LangGraph's persistence documentation distinguishes checkpointed thread state from cross-thread stores. It also notes that in-memory persistence does not survive a process restart. That framework distinction is useful, but application durability still depends on the selected storage, deployment, and business-state design.

Write down the failure scope you intend to survive: worker restart, host loss, database failure, deployment change, or regional disruption. These are different requirements. A checkpoint on a local disk may satisfy one and fail another. Choose and test the recovery boundary that the workflow actually needs.

Model business states explicitly

Scroll horizontally to see all columns.

Use states that explain what the system may do next. A generic “error” value often hides whether an operation was never attempted, rejected, or completed without acknowledgment. Those situations require different recovery decisions.

Keep cancellation and expiration explicit. A user may withdraw a request while a reviewer is considering it. A source record may change during a wait. The application should detect those events and prevent an obsolete proposal from continuing simply because an old worker resumes.

The agent architecture article explains how these states fit into graphs and loops. Durability concerns the process around model calls as well as the model's local reasoning context.

Separate a retry from a new business request

A retry attempts to complete the same intended operation after a failure or unknown response. A new request represents a new business intention. The system needs a stable identity for the first and a distinct identity for the second. Matching only on similar payload text can confuse legitimate repeated work with accidental duplication.

AWS's idempotent API guidance describes caller-provided request identifiers and the importance of handling repeated identifiers consistently. Apply that principle at the operation boundary: keep the same action identity for retries of the same intent and reject incompatible reuse with a changed payload.

In the example, an approved case-note action receives an operation identifier tied to the case and proposal version. A retry of that action keeps the identifier. A reviewer changing the note creates a new proposal and requires the corresponding approval decision. It should not silently reuse the old action's identity.

Do not promise exactly-once business effects merely because the orchestration engine stores checkpoints. The target system may process an operation before the caller records completion. The integration contract must explain how duplicate prevention and outcome reconciliation work across that boundary.

Classify failures before choosing recovery

Scroll horizontally to see all columns.

Retries are useful only when they address the failure. Repeating a forbidden operation does not make it authorized. Repeating a malformed payload may consume capacity without improving the case. The workflow should expose the reason for stopping so an operator can resolve the actual problem.

Set limits at both the individual operation and whole-case levels. A dependency may recover after several attempts, while an entire case can still exceed its business deadline. Record the elapsed wait, next scheduled attempt, and reason for deferral. This makes queue behavior visible instead of leaving users with an indefinite spinner.

Reconcile unknown outcomes

The hardest interruption often occurs between an external side effect and local confirmation. The target accepts an update, but the response is lost. The worker sees a timeout and cannot infer whether the change happened. A blind retry can duplicate the effect; declaring failure can leave a completed action unrecognized.

Design a reconciliation query or lookup using the stable operation identity or another reliable target-system record. If the target supports idempotency, use its documented contract and retention window. If it does not, identify what evidence can establish prior completion and where ambiguity still requires human investigation.

For the case-note example, the adapter might store a correlation reference that can be read back from the target. The workflow checks for that reference before repeating the write. If the target cannot expose a trustworthy completion record, the application should narrow automatic behavior or route uncertain cases to an operator.

Keep external actions individually identifiable. Updating a record and sending a notification are separate effects. If the update succeeds and the notification fails, recovery should repeat only the unfinished action. Treating the entire sequence as one retryable block creates unnecessary duplication risk.

Make waiting for people durable

An approval request should reference the exact proposal, evidence, and intended action. Store the proposal version and the review decision so a resumed worker can validate what was approved. The reviewer needs enough context to make a decision without reconstructing the entire execution history.

LangGraph's interrupt documentation illustrates pausing execution and resuming with external input. When adopting a framework, inspect its replay semantics and the placement of side effects around the interruption. Application-level approval and action records remain necessary to define what the resumed work is allowed to do.

Define expiration, cancellation, reassignment, and unavailable-reviewer behavior. A decision may be too old to use if the underlying account, policy, or request has changed. Revalidate material conditions before execution and obtain a new decision when required. A prior approval is not permanent permission for an evolving proposal.

The human-in-the-loop controls guide develops the review interface. For long-running work, add queue age and coverage to the operating design. Waiting is a first-class state with an owner and a deadline.

Control concurrent events

Two workers can receive the same event, a user can edit a case while a job runs, and a reviewer can act while a cancellation arrives. Define how the application detects competing updates. Use a state version or another concurrency mechanism appropriate to the storage layer and transition contract.

For example, approval of proposal version three should not advance a case that now contains version four. A completion event should not revive a cancelled case. Each transition should validate the current state and record the event it accepted. Repeated events should lead to a stable result rather than a second business action.

Avoid letting independent workers overwrite a shared free-form summary as the source of truth. Preserve task-specific results and merge them through an explicit rule. If several investigations contribute evidence, keep their provenance and completion status visible. The final proposal should identify which evidence version it used.

Concurrency design is easier when the application has a clear owner for each mutable business fact. Decide which component may change the proposal, which may record approval, and which may mark an external action complete. A general shared-memory channel should not obscure those responsibilities.

Plan for version changes while work is active

A deployment can occur while cases wait for review. The new code may interpret state differently, change a tool schema, or use a revised policy. Decide whether active cases continue on the original compatible version, migrate through a defined transformation, or pause for review.

Store enough version information to make that decision. Model, prompt, policy, tool, and state-schema changes can each affect resumption. A worker should not assume that a checkpoint created by an earlier configuration is safe to replay with arbitrary new behavior.

Test a representative in-flight case through the planned upgrade path. Include a pending proposal and an unknown action outcome, since these are sensitive transition points. Record how rollback affects those cases as well. Rolling back code does not automatically undo external actions or transform newer state into an older format.

The goal is not to freeze the system. It is to make change compatible with work already underway. A release procedure should say which active cases are protected, which are paused, and who resolves exceptions.

Worked recovery sequence

An employee submits a service request. The application records the case, gathers permitted evidence, and creates proposal version one. A reviewer approves it. The worker validates current conditions and submits an identified update to the service platform. The platform applies the update, but the worker loses its connection before receiving the response.

On restart, the application reads the action record and sees that the outcome is unknown. It does not ask the model whether the update probably succeeded. It queries the target using the action's correlation reference. The target confirms the update, so the application records completion and continues to the notification step.

Now assume the notification service is unavailable. The application records that the business update succeeded and schedules a bounded notification retry. The operator sees a completed update with pending communication, rather than a generic failed case. If the notification deadline passes, an assigned person can communicate the result through the defined fallback.

Finally, consider a different case where the target cannot confirm the write. The application preserves the proposal, approval, attempted payload, and available diagnostic evidence. It routes the case to reconciliation without submitting a second ambiguous update. This is a valid controlled outcome even though automatic completion was not possible.

Test recovery at meaningful boundaries

Interrupt execution before submission, after submission but before acknowledgment, after acknowledgment but before local completion, and during a waiting state. Inject duplicate events and concurrent changes. Inspect the resulting business state and target records, not only whether the worker restarted.

Use a safe integration environment or controlled adapters to exercise these conditions. Record what the test environment guarantees and what must still be checked against the actual service contract. A simulated timeout can validate application branching, while a sandbox exercise can reveal how the real target exposes completion evidence.

Measure recovery outcomes alongside task quality. Track cases resumed automatically, cases reconciled without duplication, cases requiring attention, and unresolved queue age. The evaluation guide connects these checks to release evidence. The Production AI Launch Checklist helps assign ownership before rollout.

Give operators a usable recovery path

An operator needs a clear case history, current state, confirmed actions, unresolved facts, and permitted next steps. Protect sensitive content, but retain the evidence necessary to diagnose the workflow. A runbook should explain when to retry, reconcile, cancel, request new approval, or route work manually.

Provide a way to stop new actions while preserving visibility into active cases. Make recovery actions auditable and subject to the same authority boundaries as normal execution. An emergency interface with unrestricted credentials can undermine otherwise careful tool controls.

Specify the recovery record an operator receives

A useful recovery packet begins with the case identifier and current business state. It then lists confirmed external effects, attempted operations with unknown outcomes, pending decisions, and the next permitted recovery actions. Include the relevant source and proposal versions so the operator can tell whether the evidence is still current. Keep secrets and unnecessary personal information out of the packet.

For the service-update example, the packet might show that proposal two was approved, the update was submitted with a particular operation reference, and no completion response was received. The first recovery action is to inspect the target using that reference. A generic button to rerun the case would be inappropriate because it hides the unresolved effect.

Record the operator's decision as a new event. If the target confirms completion, attach that confirmation and advance only the unfinished steps. If the target rejects the operation, record the reason and determine whether a revised proposal is needed. If the outcome remains unknown, keep the case in attention state with an assigned owner and follow-up time. An unresolved case should not disappear simply because an operator opened it.

Test this experience with someone who did not write the workflow. Ask them to explain what happened and choose the next action using the packet and runbook. Missing context in that exercise is a concrete handoff defect. Fix the information or procedure, then repeat the affected scenario.

This also provides a useful boundary for automation. Recovery steps with reliable evidence and a clear permitted action may become automated later. Steps that still require judgment about an ambiguous external effect should remain explicitly assigned. The distinction should follow observed recovery behavior rather than a desire to remove every manual exception.

Scope durability as part of the application

Long-running agents become dependable through explicit process state, integration contracts, review records, and tested recovery. Those responsibilities should appear in the implementation scope alongside model selection and prompting. Otherwise, they tend to emerge as urgent production repairs after the first interrupted action.

Bring the workflow's longest waits, consequential writes, target-system contracts, and current failure examples into a Custom AI Development discussion. They define the durability problem far more clearly than a request for an agent that can run indefinitely.