Durable Execution: The Missing Primitive for Reliable AI Agents

AI agents can do more than answer questions. They can search the web, call APIs, write code, process documents, and ask a human for approval before taking an action. To complete a single task, an agent might perform dozens of these operations over several minutes, hours, or even days.

But what happens if the process running the agent crashes halfway through a task?

The usual answer is to start the task again. This may be acceptable for a chatbot generating a response, but it becomes a problem when the agent has already performed real-world actions. It may have sent an email, charged a card, created a support ticket, or received an approval from a human. Starting again could repeat those actions, while simply continuing requires us to know exactly where the agent stopped.

The problem is not specific to AI powered systems, all sort of systems that need to perform task reliably also suffer from this problem. Durable Execution is a solution that can help systems solve this problem.

In this article, we’ll learn what durable execution is, why it is particularly useful for AI agents, and how we can use it to build an agent that continues working after failures.

A simple agent that does not always finish

Let’s imagine an agent that researches a topic and produces a report. To complete the task it must:

• create a research plan
• research each part of the plan
• analyze its findings and make a recommendation
• ask a human to approve the recommendation
• generate the final report and send a notification

We could represent the workflow with a function like this:

def create_report(topic):
    plan = create_research_plan(topic)
    findings = research_subtopics(topic, plan)
    analysis = analyze_research(topic, findings)
    recommendation = create_recommendation(topic, analysis)
    approval = request_approval(recommendation)

    if approval["decision"] == "rejected":
        return recommendation

    report = generate_report(topic, analysis, recommendation)
    send_notification(report)
    return report

The code is easy to understand, but its execution is fragile. All of its progress exists in the memory of the process running it.

If the process crashes after request_approval, we lose the local variables and the position of the program. When it restarts, it cannot tell that the research was already completed or that a human already approved the recommendation. Calling create_report again repeats the workflow from the beginning.

We could manually save the result of every step in a database and write logic that determines where to resume. But as the workflow grows, we also have to handle retries, timeouts, duplicate operations, application deployments, and steps that may take days to complete. Our agent logic becomes mixed with the code required to keep it alive.

What is durable execution?

Durable execution is a way of running code so that its progress survives process crashes, restarts, and other transient failures.

Instead of relying only on the process’s memory, a durable execution system records the progress of a function as it runs. If execution is interrupted, another process can use that recorded history to reconstruct the previous state and continue from where it stopped.

From the application’s point of view, we still write a sequence of operations:

From the durable execution system’s point of view, each completed operation and its result become part of the workflow’s history:

If the worker crashes at this point, the workflow does not have to perform the first four operations again. The system can recover their results and resume at the next operation.

Durability does not mean that a process never fails. It means the execution is designed with the expectation that processes will fail, while the overall workflow can still make progress.

To cut it short, Durable Execution is “crash-proof” execution.

Why AI agents need durability

Traditional request-response applications usually complete work within the lifetime of an HTTP request. AI agents often operate outside this model.

Agents are long-running

An agent may wait for an API rate limit, a background job, another agent, or a human decision. The longer it runs, the more likely it is to encounter a process restart, network failure, deployment, or timeout.

Agents have non-deterministic steps

Calling a language model twice with the same prompt can produce different results. Restarting an agent from the beginning can change its plan and actions, even when the original execution had already made useful progress.

Agents cause side effects

An agent becomes useful when it can act on the world, but those actions introduce risk. Retrying a model call may only cost more tokens. Retrying an operation that sends an email, transfers money, or deletes a resource can have a much greater cost.

Agents need to wait for humans

Human-in-the-loop workflows do not fit neatly into a process that must remain alive. A reviewer may respond in five minutes or two days. A durable workflow can suspend while it waits and continue after the response arrives without holding a server process open the entire time.

Durability is more than retrying

Retries are an important part of reliability, but they solve a smaller problem. A retry repeats a failed operation. It does not tell us which operations completed before the process crashed, restore their results, or prevent a completed side effect from happening twice.

A message queue can make sure work is delivered to a worker, but once the worker receives a message, we still need to track its progress. If the message describes the entire research task, redelivery may run the entire task again. We could instead create a message for every step, but then we have to coordinate their order, pass results between them, schedule retries, and decide when the workflow is complete.

We can also write checkpoints to a database ourselves. This works, and it is what the example in this article will do to expose the mechanics. But a complete durable execution system is more than a current_step column. It needs to persist the input and output of each step, retry state, waiting state, execution history, and the identity of side effects.

The main features we want are:

• Persistence: completed step outputs survive process failure
• Replay: the workflow can start from its first line and reuse completed results
• Retries: temporary failures are retried according to a durable schedule
• Suspension: a workflow can wait without occupying a worker
• Concurrency control: only one worker executes a workflow at a time
• Idempotency: repeated attempts do not duplicate an external effect

Together, these features allow us to keep the workflow readable as a sequence of operations while its execution is distributed across different processes and points in time.

Building a durable research agent

To see how this works, I built a small research agent using Flask, PostgreSQL, and a separate Python worker. The project does not use a durable execution framework. Instead, it implements a small executor so we can see the parts that frameworks such as Temporal, Restate, DBOS, and Azure Durable Functions normally provide.

The default language model provider is deterministic and runs offline. This keeps our attention on execution rather than model setup, API keys, or network access.

The application has the following architecture:

The separation between the API and the worker is important. The HTTP request only creates the workflow and returns its ID. The task does not need to finish before the request ends. An independent worker polls PostgreSQL, claims available workflows, and executes them.

The workflow

Our research workflow is still written as normal Python code. Each operation that may fail, take time, produce a non-deterministic result, or cause a side effect must cross a durable step boundary. In this project, those operations are passed to execute_step:

def run_research_workflow(executor, workflow):
    llm = get_llm_provider()
    topic = workflow.input["topic"]

    plan = executor.execute_step(
        workflow.id,
        "create_plan",
        {"topic": topic},
        lambda: create_research_plan(topic, llm),
    )
    first = executor.execute_step(
        workflow.id,
        "research_1",
        {"subtopic": plan["subtopics"][0]},
        lambda: research_subtopic(topic, plan["subtopics"][0], llm),
    )
    second = executor.execute_step(
        workflow.id,
        "research_2",
        {"subtopic": plan["subtopics"][1]},
        lambda: research_subtopic(topic, plan["subtopics"][1], llm),
    )
    analysis = executor.execute_step(
        workflow.id,
        "analyze",
        {"findings": [first, second]},
        lambda: analyze_research(topic, [first, second], llm),
    )

The workflow continues by creating a recommendation, waiting for approval, generating the report, and sending a notification. Reading it from top to bottom still tells us what the agent does. The durability logic stays in the executor.

Code outside execute_step is replayed normally. It should therefore be deterministic and must not cause an untracked side effect. Step names also become part of the workflow’s durable contract: a completed step is found by its name and its stored output is returned without running the activity or comparing its new input. Renaming a step or changing what an existing step means requires a versioning strategy for workflows that are already running.

Persisting workflow state

PostgreSQL stores two main records for execution. A workflow record contains its input, status, final output, and the next time it is eligible to run. A step record contains a unique step name, input, output, number of attempts, error, and retry schedule.

workflows
  id, status, input, output, current_step, next_run_at

workflow_steps
  workflow_id, name, status, input, output,
  attempts, error, next_attempt_at

There is a unique constraint on (workflow_id, name). This gives every logical step a stable identity. The executor can now answer a crucial question during recovery: Has this step already completed for this workflow?

The application also stores an append-only event history. The history is useful for the workshop UI and observability, but it is different from the step state used to resume execution. Logs tell us what happened; persisted step outputs give replay the values it needs to continue.

Replay

When a worker claims a workflow, it calls run_research_workflow from the first line every time. The workflow does not jump directly to current_step. Instead, execute_step checks the persisted record for each operation:

if step.status == "completed":
    logger.info("Skipping completed step: %s", name)
    return step.output

step.status = "running"
step.input = step_input
step.attempts += 1

result = activity()

step.status = "completed"
step.output = result
return result

The actual implementation uses short database transactions and row locks around these state changes, but this is the central idea. A completed step returns its stored output instead of running the activity again. That output becomes the value of the local variable, allowing the workflow to reconstruct its previous state as it replays.

Suppose the worker crashes while generating the recommendation. When another worker claims the workflow, replay behaves like this:

create_plan    -> return persisted plan
research_1     -> return persisted findings
research_2     -> return persisted findings
analyze        -> return persisted analysis
recommend      -> run the incomplete activity again

The new worker does not need the memory of the old worker. Everything needed to reach the interrupted operation is in PostgreSQL.

Activities have at-least-once execution semantics in this example. A process can die after an activity does its work but before the result is committed. The executor cannot know whether that operation completed, so it must run it again. This is why replay and idempotency must be considered together.

Crash recovery and concurrency

Two workers should not recover and run the same workflow simultaneously. The project uses a PostgreSQL session advisory lock based on the workflow ID. Under normal operation, the worker holds this lock while running the workflow.

The lock belongs to the database connection rather than the worker’s memory. If the worker process dies, PostgreSQL closes the connection and immediately releases the lock. Another worker can then claim the workflow. Short row locks protect individual state transitions, while the advisory lock provides workflow-level exclusion for this demonstration.

This is not a complete production fencing mechanism. If a live worker loses only its lock connection, another worker could acquire the lock while the first activity continues running. A production design needs leases with renewal and ownership checks, fencing tokens, or the concurrency controls supplied by a durable execution platform.

We can demonstrate recovery by slowing down the next activity:

curl -X POST http://localhost:5050/demo/slow-next-step \
  -H 'Content-Type: application/json' \
  -d '{"seconds":15}'

After creating a workflow, kill the worker while the slow activity is running and restart it:

docker compose kill worker
docker compose up -d worker

The persisted execution history will show that the workflow was recovered and the interrupted step was resumed. The worker logs will show Skipping completed step for outputs returned during replay. We have lost a process, but not the workflow.

Durable retries

Some failures do not require a full workflow recovery. An API may be temporarily unavailable or reject a request because of a rate limit. When an activity raises a temporary error, the executor calculates a bounded exponential backoff:

delay = min(base_backoff * (2 ** (attempt - 1)), max_backoff)
retry_at = utcnow() + timedelta(seconds=delay)

step.status = "pending"
step.next_attempt_at = retry_at
workflow.status = "pending"
workflow.next_run_at = retry_at

The retry time is stored in PostgreSQL. The worker does not call sleep for the duration of the backoff and it does not need to remain alive. It releases the workflow and can execute other work. Once next_run_at is reached, any worker can claim and replay it.

We can deterministically fail the next activity once with:

curl -X POST http://localhost:5050/demo/fail-next-step

The history will show the first attempt, the scheduled delay, and the second attempt. Because the retry schedule is durable, restarting every process during the delay does not reset it.

Waiting for human approval

After producing a recommendation, our agent must wait for a person to approve or reject it. The executor stores the recommendation as the approval step’s input, changes the workflow status to waiting, and returns the worker to its polling loop.

step.status = "waiting"
step.input = {"recommendation": recommendation}
workflow.status = "waiting"
raise WorkflowWaiting()

The workflow now consumes no worker resources. The application can be offline for minutes or days without losing the recommendation. When the approval endpoint is called, it stores the decision as the step’s output and changes the workflow back to pending. A worker then replays the workflow, recovers the earlier results, reads the approval, and generates the report.

This same pattern can be used when waiting for a webhook, a scheduled time, another workflow, or an external event. Waiting becomes persisted state rather than a process blocked in memory.

Idempotent side effects

Persisting step outputs prevents completed model and research activities from being repeated during replay, but external side effects need additional protection.

Imagine the agent sends its final notification and crashes before execute_step stores the completed result. On recovery, the notification step still appears to be running. Running it again could send the same message twice.

The sample project creates an idempotency key from the workflow and operation:

key = f"{workflow_id}:final_notification"

Before sending, it inserts this key into a table where the key is the primary key. If replay attempts the notification again, the insert violates the unique constraint and the duplicate is skipped.

This demonstration provides at-most-once delivery because it reserves the key before sending. It prevents duplicates, but introduces another failure window: the process could die after reserving the key and before sending the notification. In that case, the notification would be lost.

For production systems, it is better to pass the idempotency key to an external provider that supports it, or use a transactional outbox with an idempotent consumer. A durable execution engine cannot make an arbitrary external API part of its database transaction. We still need to define the delivery guarantee for every side effect.

The failure window can be demonstrated by arming a one-shot crash before approving a waiting workflow:

curl -X POST http://localhost:5050/demo/crash-after-notification

After approval, the worker reserves the idempotency key, sends the notification, and exits before the step is marked as completed. Restarting the worker causes it to replay send_notification. The existing key is found, so the duplicate notification is skipped and the step can complete.

Running the example

The complete application is available on GitHub. Clone the repository and run it with Docker Compose:

git clone https://github.com/Xavier577/deflaskcon2026.git
cd deflaskcon2026
docker compose up --build

Open http://localhost:5050, enter a research question, and watch each step and event as they are persisted. The PostgreSQL volume remains across container restarts, which allows us to stop the entire application while it is waiting for approval and continue later.

The workflow can also be created from the command line:

curl -s -X POST http://localhost:5050/workflows \
  -H 'Content-Type: application/json' \
  -d '{"topic":"PostgreSQL vs DynamoDB for financial transactions"}'

The API returns a workflow ID. We can use it to inspect the workflow and its durable history:

curl -s http://localhost:5050/workflows/WORKFLOW_ID
curl -s http://localhost:5050/workflows/WORKFLOW_ID/history
curl -s -X POST http://localhost:5050/workflows/WORKFLOW_ID/approve

What durable execution does not solve

Durable execution makes an agent’s progress reliable, but it does not make the agent’s decisions correct. A workflow can reliably execute a bad plan, use an inaccurate model response, or call the wrong tool.

We still need to consider:

• Model quality: prompts, evaluation, grounding, and output validation
• Permissions: an agent should only have access to the tools and data required for its task
• Guardrails: high-risk actions may need policy checks or human approval
• Observability: operators need to understand what the agent did and why
• Idempotency: external side effects need an explicit delivery strategy
• Versioning: changing workflow code or the meaning of a step name can change replay behavior for workflows that are already running
• Cost: retries and repeated incomplete activities can result in additional model and API usage

Our implementation is also a teaching tool, not a general workflow platform. It does not implement heartbeats, cancellation, payload versioning, history retention, scalable task queues, workflow-code versioning, or fencing if a live worker loses its lock connection. Production durable execution systems exist because handling these concerns correctly for many workflows is a substantial engineering problem.

The goal of building the small executor is not to suggest that every team should build its own. It is to make the primitive visible. Once we understand the state, replay, retries, and idempotency involved, the value provided by a production runtime becomes much easier to evaluate.

Conclusion

AI agents turn a model call into a long-running sequence of computations, tool calls, external actions, and human decisions. Every additional step creates another place where the process can fail and another result we may not want to compute twice.

Durable execution changes the unit of reliability from the process to the workflow. Processes can crash, workers can restart, deployments can happen, and humans can take days to respond. The workflow retains its identity, completed results, retry schedule, and waiting state throughout all of them.

The important ideas are straightforward:

• give every workflow and step a durable identity
• persist step outputs before relying on them
• replay workflows and reuse completed results
• schedule retries in durable storage rather than sleeping
• suspend workflows while waiting for external events
• make side effects idempotent and define their delivery guarantees

This does not solve every problem involved in building reliable AI agents, but it provides the foundation on which those agents can do useful work without losing their progress whenever a process disappears. If an agent is expected to act over time, durability should not be an afterthought. It should be part of its execution model.

Further reading

• What is Durable Execution? by Temporal. An introduction to durable execution, replay, and failure recovery.
• Durable execution concepts by Restate. An explanation of journals, replay, durable state, and resilient communication.
• What are AI agents? by IBM. Background on AI agents, their components, and common applications.
• A practical guide to building agents by OpenAI. Practical guidance on tools, orchestration, guardrails, and human intervention.