Durable Execution for Long-Running Agents

An agent works for forty minutes, finishes the difficult part of a migration, and loses its process before writing the final status.

What happens next reveals whether the system is an agent or merely a long prompt.

A fragile implementation starts over. It rereads the repository, rebuilds the plan, repeats tool calls, and hopes the outside world still matches its assumptions. A durable system resumes from an explicit record. It knows which step owned the work, what artifacts were produced, which effects were observed, and what remains unverified.

Long-running autonomy is not created by a larger context window. It is created by execution that survives interruption.

The Process Is Not the Workflow

The first architectural mistake is storing the workflow only in model context or worker memory.

Processes disappear. Containers restart. Browser sessions expire. Models hit limits. Queues redeliver messages. A human approval may arrive hours after the worker that requested it has gone away.

If the task exists only inside that worker, every infrastructure failure becomes amnesia.

The durable unit is an operation record outside the model. It contains the goal, current plan version, step states, artifact references, active authority, attempts, external operation identities, and completion criteria. A worker may hold a convenient in-memory view, but it is never the sole source of truth.

The model participates in the workflow. It does not contain the workflow.

Checkpoint at Meaningful Boundaries

Saving a transcript after every token is not durable execution. It is expensive journaling without a recovery model.

Checkpoint when the system crosses a boundary that a replacement worker must understand: a plan is accepted, an artifact is created, a tool effect is requested, an approval is granted, verification passes, or ownership moves to another worker.

A useful checkpoint answers four questions:

• what was the step trying to establish?
• what evidence and artifacts now exist?
• what external effects may have occurred?
• what condition permits the next transition?

This record should be compact enough to load and structured enough to validate. The full trace can remain available for investigation, but resumption should not require replaying an hour of conversation as literature.

Make Steps Resumable, Not Merely Retryable

Retrying means running the same instruction again. Resuming means inspecting the recorded state and continuing from the correct boundary.

The difference is critical for effectful work. If a deployment request timed out, retrying may start a second deployment. Resuming first reconciles the operation identity with the deployment service. If code was written but tests never ran, resuming begins with the existing artifact and the pending verification step.

Each step needs a recovery policy. Pure reads can often be repeated. Deterministic transformations can be recomputed when inputs are versioned. External effects require idempotency and reconciliation. Human decisions must be bound to the artifact that was actually approved.

“Run this step again” is not a universal recovery strategy.

Use Leases for Ownership

Durability creates another problem: more than one worker may believe it should continue the same task.

A queue message can be delivered twice. A slow worker can look dead to a monitor. An operator may manually resume a task that is still active elsewhere. Without ownership control, two competent agents can create conflicting artifacts or duplicate effects.

Use a lease with a bounded lifetime. A worker acquires the step, renews ownership while making progress, and releases it on a durable transition. If the lease expires, another worker may take over, but it begins by reading the operation record and reconciling any in-flight effects.

The lease prevents simultaneous execution. It does not prove that the previous worker did nothing. Recovery still needs evidence about the outside world.

Record Events, Derive Current State

A single mutable status field such as running or failed throws away the path that produced it.

An append-only event history gives the workflow a durable memory: step proposed, lease acquired, tool call attempted, acknowledgement lost, state reconciled, artifact verified, approval consumed. The current view can be derived from those events while the history remains available for debugging and audit.

This does not require a fashionable event-sourcing platform. It requires stable identities, ordered transitions, and enough information to reject impossible state changes.

Events also make automation safer. A monitor can react to “lease expired with an unresolved external operation” differently from “worker failed before any effectful step.” The word failed alone cannot support that distinction.

Persist Artifacts, Not Just Summaries

Agent handoffs often lose the work while preserving a polished description of it.

The system says a migration plan was completed, but the actual plan lived in a temporary directory. A reviewer receives a summary without the diff. A resumed worker knows that research happened but cannot inspect the source manifest.

Artifacts need durable locations, versions, hashes, ownership, and retention. A checkpoint should refer to the exact repository commit, document, query result, test report, or prepared action that the next step will use.

Summaries help a model orient. Artifacts let a system continue.

This becomes more important after context compaction. A concise state description can link back to authoritative work products instead of trying to encode all substance inside a smaller prompt.

Treat Waiting as a Real State

Long-running workflows spend much of their life waiting: for a build, a human, an external system, a scheduled window, or a change in source data.

Keeping a model invocation alive while nothing can happen wastes resources and makes recovery harder. Waiting should be a durable state with a reason, wake condition, deadline, and escalation path.

The workflow can suspend after requesting approval and resume when a signed decision event arrives. It can wait for a deployment to reach a terminal state without holding a conversation open. It can schedule a follow-up check when an external service reports pending.

An agent that can stop cleanly is more autonomous than one that must keep talking to prove it is still working.

Separate Workflow Time From Model Time

A task may last three days while using only twenty minutes of model inference.

This distinction improves both architecture and cost. Workflow time includes human review, external processing, backoff, and scheduled observation. Model time should be spent only when interpretation, planning, or generation is needed.

Triggers should wake the right step with the smallest sufficient context. A deployment completion event may require a deterministic verifier first; the model only returns if the result is ambiguous. A changed file can invalidate one artifact without forcing the whole plan to restart.

Durability makes intelligence intermittent. The workflow persists even when no model is running.

Define Honest Terminal States

Success and failure are too crude for long-running work.

A task may be completed and verified, completed with a warning, safely stopped before effects, blocked on missing authority, or unresolved because an external action may have occurred. These states have different operational consequences.

The system should never mark a task complete merely because the model produced a final answer. Completion is a transition backed by required evidence. It should never label an ambiguous payment failure as safely failed. Uncertainty is a state worth preserving.

Honest terminal states prevent automation from turning missing information into false confidence.

Test Recovery as Aggressively as the Happy Path

Kill the worker after it writes an artifact but before it records the checkpoint. Drop the acknowledgement after an external action commits. Expire the lease during a slow tool call. Deliver the same wake event twice. Resume with a newer model and a compacted context. Revoke approval while the task is waiting.

Then ask whether the workflow produces one correct effect, stops with a legible unresolved state, or corrupts its own story.

These tests are not edge cases. They simulate the environment long-running agents are built to inhabit.

Persistence Changes What an Agent Can Be Trusted to Do

The promise of agentic systems is not that a model can think for a long time without interruption. It is that useful work can continue across interruptions without losing identity, evidence, or control.

Externalize the operation. Checkpoint meaningful transitions. Use leases. Persist artifacts. Reconcile effects. Model waiting explicitly. Require evidence for completion.

The durable agent is not the one that never stops.

It is the one that can stop anywhere, return later, and still know exactly what must happen next.