An Agent That Can't Resume After a Crash Isn't Production-Ready

An agent run is a long-lived process. Picture an agent that analyzes a support ticket, retrieves relevant docs, drafts a response, waits for a manager's approval, then sends the email. That's five steps, spread across tools, retrieval, and a human who might approve two hours later or two days later. Any one of deploys, OOM kills, spot instance preemptions, provider 429s or 5xx errors, or that human delay will interrupt the run.

The naive fix is "retry the whole thing." That is how you double-email a customer, double-issue a refund, or re-run an expensive retrieval you already paid for. Logs might say the call succeeded, but if the process died before the checkpoint was written, the next worker doesn't know that.

Durable execution is the system I build around an agent so a crash doesn't mean start over. It means: checkpoint every step, make every side effect idempotent, keep a ledger that remembers what already happened, and test the resume path before you ship. This post is about those four layers and when they matter.

The failure modes, concretely

Lost progress. The agent spends tokens on a plan, retrieval, and tool calls. It crashes before writing the checkpoint. The next invocation starts from scratch and re-pays for all of it.

Duplicated side effects. The agent calls an API. The API returns success. The process crashes before the step is recorded as complete. The resume logic sees an incomplete step and retries the call. Now the effect has happened twice: two emails, two charges, two record updates.

Zombie runs. The agent crashes. Two workers both decide the run needs resuming. They both load the same checkpoint and both try to continue from step three. Now you have two concurrent executions writing to the same thread, racing each other, and producing inconsistent state.

Stuck waits. The agent waits for a human approval. The human never answers. Nothing times out. The run holds state forever, or worse, a process slot if you didn't design the wait to release resources.

These are not edge cases. They are what happens when you run agents long enough for the world to interrupt them.

Layer 1: Step checkpoints

The first layer is state persistence at every step boundary. The checkpoint store records what the agent has decided and done so far. If the process dies, a new one can load that state and continue.

In LangGraph, you get this with a checkpointer. The production one is PostgresSaver from the langgraph-checkpoint-postgres package. The in-memory checkpointer loses everything on restart.

from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.graph import StateGraph

DB_URI = "postgresql://user:pass@localhost/agentdb"

with PostgresSaver.from_conn_string(DB_URI) as checkpointer:
    checkpointer.setup()
    graph = builder.compile(checkpointer=checkpointer)

Every run is keyed by a thread_id. You pass it in the config when you invoke:

config = {"configurable": {"thread_id": "run_abc123"}}
result = graph.invoke(user_input, config)

LangGraph saves checkpoints at node boundaries, not inside a node. If a node crashes halfway through, the resume will re-run the entire node from the beginning. That is why I design nodes to be small and do one side effect each, as the last thing in the node, so a retry is safe.

The durability mode controls when the checkpoint is written. There are three:

• durability="exit" — persist only when the graph exits: success, error, or interrupt. No mid-run crash recovery. Lowest overhead.
• durability="async" — persist while the next step runs. Small risk of losing a checkpoint if the process dies during the write.
• durability="sync" — persist before the next step starts. Most durable. Some overhead.

For runs with side effects, I use sync:

result = graph.invoke(user_input, config, durability="sync")

To resume a crashed run, you invoke with None as the input, same thread ID:

result = graph.invoke(None, config)

LangGraph loads the latest checkpoint and retries the unfinished work. If the run was interrupted via interrupt(), you resume with a Command:

from langgraph.types import Command

result = graph.invoke(Command(resume=approval_data), config)

The interrupted node starts again from its beginning, so the code before the interrupt() must be idempotent.

One edge case the LangGraph docs warn about: a crash before the first checkpoint is durable leaves the thread indistinguishable from never invoked. Calling invoke(None, config) raises EmptyInputError. The fix is to record admission outside the checkpointer. I write an outbox row to my own store before calling invoke the first time. That row is proof the run was admitted.

Checkpoints give you crash recovery at step boundaries. They do not by themselves prevent duplicate side effects. That is layer two.

Layer 2: Idempotency keys on every side effect

An idempotency key is a stable identifier for a logical action. The rule: derive it deterministically from (run_id, step, logical action, stable args hash). Never from uuid4() at call time. Never from timestamps. If the same step re-runs, it must produce the same key.

Example:

import hashlib
import json

def make_idempotency_key(run_id: str, step: str, action: str, args: dict) -> str:
    canonical = json.dumps(args, sort_keys=True)
    args_hash = hashlib.sha256(canonical.encode()).hexdigest()[:16]
    return f"{run_id}:{step}:{action}:{args_hash}"

You pass this key through to providers that support idempotency. Many payment APIs accept an Idempotency-Key header. If the call is repeated with the same key, the provider returns the original result instead of executing again.

For providers that don't support it, the effect ledger is your dedupe. That is layer three.

Layer 3: Effect ledger and outbox

The effect ledger is a Postgres table that records every side-effecting action the agent attempts. Before the agent calls an external API, it inserts a row. If a row with that idempotency key already exists and the effect is marked as sent or confirmed, the agent returns the stored result and skips the call.

Here is the schema:

CREATE TABLE effects (
    idempotency_key TEXT PRIMARY KEY,
    run_id TEXT NOT NULL,
    step TEXT NOT NULL,
    tool TEXT NOT NULL,
    args_hash TEXT NOT NULL,
    status TEXT NOT NULL CHECK (status IN ('pending', 'sent', 'confirmed', 'failed')),
    provider_ref TEXT,
    result JSONB,
    attempts INT DEFAULT 0,
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_effects_run_id ON effects(run_id);
CREATE INDEX idx_effects_status ON effects(status);

The flow:

1. Reserve: INSERT INTO effects (...) VALUES (...) ON CONFLICT (idempotency_key) DO NOTHING
2. Check: If the row already exists with status sent or confirmed, return the stored result. Skip the call.
3. Perform: Call the external API.
4. Record: UPDATE effects SET status = 'sent', provider_ref = ?, result = ?, attempts = attempts + 1 WHERE idempotency_key = ?

When the provider confirms the effect succeeded, update the status to confirmed. If it failed, record failed and the error.

Here is a Python wrapper:

import hashlib
import psycopg2
import json

def run_effect(conn, key: str, run_id: str, step: str, tool: str, args: dict, fn):
    args_hash = hashlib.sha256(json.dumps(args, sort_keys=True).encode()).hexdigest()[:16]
    
    with conn.cursor() as cur:
        cur.execute("""
            INSERT INTO effects (idempotency_key, run_id, step, tool, args_hash, status)
            VALUES (%s, %s, %s, %s, %s, 'pending')
            ON CONFLICT (idempotency_key) DO NOTHING
        """, (key, run_id, step, tool, args_hash))
        conn.commit()
        
        cur.execute("SELECT status, result FROM effects WHERE idempotency_key = %s", (key,))
        row = cur.fetchone()
        
        if row[0] in ('sent', 'confirmed'):
            return json.loads(row[1])
        
        result = fn()
        
        cur.execute("""
            UPDATE effects
            SET status = 'sent', result = %s, attempts = attempts + 1, updated_at = NOW()
            WHERE idempotency_key = %s
        """, (json.dumps(result), key))
        conn.commit()
        
        return result

For providers that do support idempotency keys, you still record in the ledger. The provider is authoritative for "did this happen," but the ledger is your audit trail and your reconciliation source. If a row is stuck in pending and you do not know whether the call landed, query the provider by the idempotency key or the provider_ref before retrying.

Transactional outbox is a variant: the agent writes the intent row in the same transaction as its own state change. A separate dispatcher worker reads pending rows and sends them. This decouples side effects from the agent process, which helps with long waits or rate limits, but it adds complexity. I reach for it when the agent must not block on the external call.

Layer 4: Resume path and run leases

Checkpoints and idempotency keys are passive infrastructure. The resume path is the code that uses them when a run crashes.

On worker boot or via a background sweeper, I find runs whose lease has expired. A lease is a row or a lock that says "this worker is handling this run." If the worker dies, the lease expires. A new worker acquires it and resumes.

Postgres advisory locks work:

import hashlib

def lock_key(run_id: str) -> int:
    return int.from_bytes(hashlib.sha256(run_id.encode()).digest()[:8], 'big', signed=True)

def try_acquire_lease(conn, run_id: str) -> bool:
    with conn.cursor() as cur:
        cur.execute("SELECT pg_try_advisory_lock(%s)", (lock_key(run_id),))
        return cur.fetchone()[0]

def release_lease(conn, run_id: str):
    with conn.cursor() as cur:
        cur.execute("SELECT pg_advisory_unlock(%s)", (lock_key(run_id),))

Python's built-in hash() on strings is randomized per process (PYTHONHASHSEED), so two workers get different lock IDs for the same run. The lock_key helper uses a stable SHA-256 hash instead.

Or a lease table with expiry:

CREATE TABLE run_leases (
    run_id TEXT PRIMARY KEY,
    worker_id TEXT NOT NULL,
    expires_at TIMESTAMPTZ NOT NULL
);

The sweeper logic:

from datetime import datetime, timedelta, timezone

def resume_crashed_runs(conn, worker_id: str):
    now = datetime.now(timezone.utc)
    
    with conn.cursor() as cur:
        cur.execute("""
            SELECT run_id FROM run_leases
            WHERE expires_at < %s
            FOR UPDATE SKIP LOCKED
        """, (now,))
        
        for (run_id,) in cur.fetchall():
            cur.execute("""
                UPDATE run_leases
                SET worker_id = %s, expires_at = %s
                WHERE run_id = %s
            """, (worker_id, now + timedelta(minutes=5), run_id))
            conn.commit()
            
            config = {"configurable": {"thread_id": run_id}}
            try:
                graph.invoke(None, config)
            except Exception as e:
                log_error(run_id, e)

The FOR UPDATE SKIP LOCKED ensures only one worker acquires each lease.

Before the first invoke, record the admission. A row in the outbox or a separate admission log. This is the fix for the LangGraph docs warning about the empty-thread case.

Interrupted nodes re-run from the top when you resume. The code before the interrupt() must be idempotent. Do not increment counters or append to lists before the interrupt unless you are prepared for them to happen twice.

The chaos test I run: kill the worker process between "API call returned" and "checkpoint written." Assert that exactly one effect row exists in the ledger for that idempotency key, with status sent or confirmed. If you find duplicates, your keys are not stable. If you find zero, your effect recording is racing the checkpoint.

Layer 5: Human-in-the-loop waits and timeouts

An interrupt() in LangGraph pauses the run and releases the process. The run does not hold a worker while waiting for a human. When the human approves, you invoke with Command(resume=approval_payload).

Timeouts matter. An approval request should not wait forever. I set a deadline: if the human does not respond within some threshold, the run escalates to another channel, auto-cancels, or degrades to read-only.

For long tool calls, heartbeats keep the system aware the work is alive. Temporal has this built in: an activity sends periodic heartbeats to the server. If heartbeats stop arriving, the server assumes the worker crashed and reschedules the activity. The timeout is HeartbeatTimeout, separate from StartToCloseTimeout.

LangGraph does not have this. You build it yourself: the tool call writes a heartbeat row every few seconds. A monitor checks that the row timestamp is fresh. If it goes stale, the run is marked for resumption.

Approval payloads should carry the idempotency key so a double-click approve does not double-execute. The approval handler checks the effect ledger: if the key is already sent or confirmed, return the stored result. Otherwise proceed.

LangGraph checkpointer vs Temporal

LangGraph with a checkpointer gives you state persistence at node boundaries. You orchestrate the leases, the sweeper, the timeouts, and the heartbeats yourself. You control the schema, the storage, and the replay logic.

Temporal is a durable workflow engine. It owns retries, timeouts, heartbeats, signals, and event history. Activities are the unit of work. Temporal records when an activity completes. A crashed activity is retried. A completed activity is not re-executed on workflow replay. The workflow's event history is append-only and deterministic.

Temporal has more machinery. It gives you a hosted or self-hosted server, a worker SDK, history replay, and visibility into long-running workflows. Activities still need idempotency. Temporal delivers at-least-once: an activity is not recorded in history until it returns or errors. If it crashes before returning, it is retried.

LangGraph gives you a library. You bring your own database, your own lease protocol, and your own sweeper. You have full control over checkpoints, redaction, and the replay path. You pay the setup cost.

When I reach for each:

LangGraph checkpointer: The agent is already in LangGraph. I want fine-grained control over the checkpoint schema and replay behavior. I have Postgres. I am willing to build the sweeper and the lease logic. The agent is part of a larger service and I do not want a separate workflow engine.

Temporal: The workflow spans services, languages, or teams. I need built-in retries, timeouts, heartbeats, and signals without building them myself. The workflow is long-running and complex enough that the event history and replay debugger are worth the operational overhead. I am okay with a hosted or self-hosted Temporal cluster.

When this is overkill

A single short LLM call does not need durable execution. Neither does a read-only agent that fetches data, summarizes, and returns. If the agent crashes, rerun the call. It is cheap.

A job that is naturally idempotent because the underlying store dedupes on a unique key does not need an effect ledger. A plain queue with retries and an idempotency key passed through to the provider is enough.

Decision list:

• Single LLM call, no tools, no state: No checkpoints needed.
• Read-only, short-running, no side effects: Retry on crash. Checkpoints optional.
• Multiple steps, side effects, but each step is idempotent: Checkpoints + idempotency keys. No ledger if the provider enforces idempotency.
• Multiple steps, side effects, non-idempotent providers: Checkpoints + idempotency keys + effect ledger.
• Human-in-the-loop, long-running, multi-hour waits: Add timeouts, heartbeats, and lease-based resumption.
• Multi-service, multi-language, complex state machines: Consider Temporal or similar.

The hard line is side effects. Once the agent can send, charge, or update, you need idempotency and a ledger. Everything else is tuning.

Checklist to audit an agent for crash-safety

• Does the agent persist state after every step, or only at the end?
• If a node crashes halfway, does it re-run the whole node, and is that safe?
• Are idempotency keys derived deterministically, or generated with uuid4() at call time?
• Does the ledger record effects before the call, or only after?
• Can two workers resume the same run concurrently, or is there a lease?
• Is there a sweeper to find and resume crashed runs?
• Do human-in-the-loop waits have timeouts and escalation paths?
• Do long tool calls have heartbeats so you know they are alive?
• If a process dies after "API call succeeded" but before "checkpoint written," does the next resume duplicate the effect?
• Is admission recorded before the first invocation so an empty-thread error does not lose the run?

If the answer to any of these is "I am not sure," test it. Kill the process at that boundary. Check the ledger. Check for duplicates. Replay the trace.

I ship replayable traces alongside durable execution. When a resume path is wrong, the trace tells me which step diverged. When an idempotency key is wrong, the ledger tells me which duplicate got through. Observability is not optional for durability. You cannot trust a system you cannot inspect.

When to call me

If you are running agents that span minutes or hours, wait for humans, call external APIs with side effects, or operate in regulated domains where a duplicate charge or a duplicate email is an incident, I can help. I design and retrofit durable execution architectures, effect ledgers, lease protocols, and resume chaos tests for production agents.

I am a freelance AI engineer. I work with LangGraph, FastAPI, Postgres, and production agent stacks. I do not build prototypes. I build the systems that let you ship the prototype to production and keep it running when the world interrupts it.

If that is where you are, let's talk: https://www.bilaltahseen.com/hire-ai-engineer.