Technology note — this article reflects LangGraph 1.2 (released May 2026), Agent Server releases through the 0.8.x line, and the LangSmith runtime capabilities available or announced as of July 2026. Runtime guarantees are stated per documented semantics and marked where they depend on configuration; managed-runtime features are labeled with their maturity. Code samples were written and import-checked against langgraph==1.2.10 on Python 3.11.
Engineering AI for Production
In Part 1 of Engineering AI for Production, we looked at the retrieval layer beneath RAG systems.
Part 2 moved into the agent development loop, showing how prompts, evaluations, baselines and production feedback can make change safer and more measurable.
Part 3 looked at the platform underneath that workflow, comparing self-hosted, hybrid and SaaS deployment models.
Part 4 then moved into long-horizon agents: systems that work across hundreds of steps, manage growing context and rely on planning, persistent workspaces and subagents to stay on course.
The final part asks a different question:
What happens when that agent needs to keep running for twenty minutes, an hour or longer, while the infrastructure underneath it is changing?
At that point, the problem starts to look much less like prompt engineering and much more like distributed systems engineering.
A long-running agent has state that must survive failures. Tool calls may need to be retried without repeating real-world side effects. Human approvals may need to survive a deployment or even a weekend. Pods need to shut down without destroying in-flight work. Generated code needs an execution boundary. And the runtime needs to know what to do when workers disappear halfway through a task.
In this final article, Ernesto Martinez looks at the runtime layer that makes long-running agents genuinely durable: checkpointing, idempotency, graceful draining, persistent interrupts, queue architecture and sandboxed execution.
The aim is not simply to make an agent capable of running for an hour.
It is to make sure that hour of work can survive the kinds of failures, retries and deployments that are normal in production software.
Engineering AI for Production · Part 5 of 5
An agent that runs for an hour is a distributed-systems problem wearing an AI costume.
Strip away the model calls and look at what remains: a stateful, resumable, interruptible workload that must survive deploys, retries and partial failures without duplicating side effects or losing an hour of work. Every guarantee it needs — durability, exactly-once effects, clean shutdown, human approval that outlives a restart — is a classic infrastructure problem. The previous post designed the workload; this one builds the floor under it.
Durability: checkpoint the state machine, then choose when
LangGraph's execution model is what makes durability tractable: a graph advances in discrete supersteps, and with a checkpointer configured, state is persisted at those boundaries. A crashed or evicted run resumes from its last checkpoint on the same thread — not from zero.
The durability parameter is a knob most teams don't know they have: "sync" persists before proceeding (strongest guarantee, highest latency), "async" persists while the next step runs (a crash can lose the in-flight step), "exit" persists only at completion. Our rule: sync for anything with side effects, async for read-only research work, exit only for the disposable.
Then comes the cost problem nobody meets in a demo: for chat-heavy agents, classic checkpointing re-serialises an ever-growing message history every super step —O(history) writes per step, quadratic over a long run. This is exactly what LangGraph 1.2's DeltaChannel (beta, May 2026) addresses: channels that persist incremental deltas instead of full accumulated values, turning per-step write cost from "the whole conversation" into "what changed. "For hour-long, message-heavy runs this is the difference between checkpointing being invisible and being your write bottleneck — measure it on your own workload before and after; it is beta, and its benefit scales with how append-heavy your state is.
Idempotency: the guarantee the framework cannot give you
Resume-from-checkpoint has a sharp edge: the boundary between "crashed after the tool call" and "crashed after the checkpoint" is where duplicates are born. A payment issued, an email sent, a ticket created — replayed on resume because the step's effect happened but its completion was never persisted.
No framework can solve this generically, because it lives inside your tools. The pattern is the same one payment APIs have used for a decade — derive an idempotency key from what is stable across a replay (the thread and the step's semantic identity, not a timestamp), and make the effectful call conditional:
Combine this with durability="sync"on effectful nodes and per-node retry policies for the transient failures, and replays become harmless. Two adjacent 1.2 additions complete the operational picture: per-node timeouts (wall-clock and idle) so one hung API call cannot absorb a run forever, and per-node error handlers so a failed step degrades according to policy instead of exploding the run. All three are declared where they belong — on the node:
Shutdown and interrupts: the runtime meets thedeploy pipeline
The most common killer of long runs is not a crash — it is your own deploy. Kubernetes sends SIGTERM, the grace period expires, and every in-flight rundies mid-superstep. Resumable, yes (that was the point of checkpointing) — but resumable is the floor. The clean pattern is drain on-deploy: stop accepting new work, let in-flight work reach a checkpoint boundary, then exit. LangGraph1.2 gives this a first-class API — RunControl.request_drain()— which requests that work wind down gracefully rather than being severed; wireit to SIGTERM and size your termination Grace Period Seconds to your longest supersteps, not to the default 30 seconds. Drain semantics stop new work and let current work reach a boundary — they do not magically finish an hour of remaining plan; resumability covers the rest.
The wiring is short enough to show in full — a signalhandler on the worker's side, and a grace period on Kubernetes' side that respects your actual superstep length:
Worth pairing with a readiness probe that fails once drain is requested, so the load balancer stops routing new runs to a pod that is winding down. With this in place, a deploy during a hundred in-flight runs is an observable non-event: run-age metrics dip as workers drain, checkpoints land, new pods pick the threads back up.
Human-in-the-loop interrupts are the same discipline pointed at a longer clock. When a graph hits an interrupt() — an approval gate before that refund, say — the run parks at a checkpoint indefinitely. The guarantee that the approval survives a weekend and a deploy holds only if three things are true: a durable checkpointer (not the in-memory one that quietly works in dev), a stable thread_id the resuming request can present, and retention that outlives your slowest approver. Test it the honest way: trigger an interrupt, kill the process, deploy a new version, then approve — if Command(resume=...) picks the run up correctly, you have durable HITL; if not, better to learn it in staging.
The queue and the security boundary
Above single-run correctness sits fleet mechanics: something must accept runs, schedule them, survive worker death, and back-pressure under load. That layer is a genuine build-vs-buy decision — and the managed path's own evolution is instructive: the Agent Server (the deployable runtime behind LangSmith Deployment) moved its run queue from Postgres polling to Red is in the 0.8 line (April 2026) and adopted delta-based checkpointing in its Postgres checkpointer shortly after. Read that as field-tested confirmation of where the pressure points are at scale: queue throughput and checkpoint write volume. If you self-host this layer, you will meet both — our own wish list for it is exactly a worker heartbeat/reclaim protocol, queue depth and run-age metrics, crons, and double-execution protection, which is precisely the feature list the managed server advertises.
Long-running agents also increasingly execute code — and an agent that runs generated code needs an execution boundary stronger than a container in your app cluster, because prompt injected code is untrusted code with your network attached. The pattern we consider table stakes, whether assembled from Firecracker-style microVMs or bought (LangSmith Sandboxes went GA in May 2026 with this architecture —vendor-described): hardware-virtualized isolation; snapshot/restore so a prepared environment starts warm and can fork per run; deny-by-default egress; and — the piece most home-built setups miss — credentials outside the sandbox, held by an authenticating proxy that injects them into approved outbound calls, so exfiltration has nothing to steal. Model traffic gets the same treatment at the fleet level: an LLM gateway in front of providers (LangSmith's is private beta as of May 2026; the pattern is portable) enforcing hard spend caps per team, PII/secret redaction before requests leave your perimeter, and policy violations as observable events. An agent that can spend money and run code should not be one config error away from doing either without limit — runaway-cost stories are an agent-era common place; caps at the boundary are what make them a non-event.
Self-hosted or managed: where the break-even actually sits
Everything above can be assembled from OSS LangGraph on your own Kubernetes — persistence, drain, queue, sandbox, gateway are all buildable. The honest question is whether they are worth building, and the answer follows the same logic as the platform decision in post two. What tilts toward self-hosting the runtime: hard residency or network-isolation requirements for agent state; deep integration with an existing job platform; genuinely unusual execution semantics. What tilts toward managed: the queue-and-fleet layer being undifferentiated heavy lifting; the sandbox boundary being easy to get subtly wrong; feature velocity landing managed-first.
The break-even is not a constant — it is a function of your team's platform bench, workload volume, and compliance posture, and it moves as the managed layer matures. What we can offer is the shape: the components in this post are the checklist below either way, and the build vs-buy line should be drawn per component, not for the stack as a whole. Plenty of sound architectures run OSS LangGraph with their own Postgres and drain wiring — and buy the sandbox.
Takeaway: the runtime requirements checklist
Before trusting any agent with hour-long work — self-hostedor managed — demand a concrete answer to each:
- Durability: Which checkpointer, which durability mode per node class, and what is the measured checkpoint write cost at P99 history length?(Delta-based checkpointing evaluated?)
- Replay safety: Does every effectful tool carry an idempotency key stable across resume? Which test proves it?
- Failure policy: Per-node timeouts, retry policies scoped to transient errors, error handlers for the rest —declared on every node that talks to the world?
- Deploy behavior: Is drain wired to SIGTERM, is the grace period sized to superstep length, and does a mid-run deploy provably lose zero work?
- Interrupts: Does an approval survive process death, a deploy, and a weekend — verified in staging, with retention configured?
- Queue: What reclaims a run when aworker dies, what protects against double execution, and what metric pages you when run-age grows?
- Execution boundary: Is generated codeisolated at the VM level, is egress deny-by-default, and are credentials held outside the sandbox?
- Spend boundary: Are per-team hard caps and redaction enforced at a gateway — and are violations observable events, notlog lines?
Eight answers, and the twenty-minute agent from the previous post becomes a workload you can deploy on a Tuesday afternoon without watching it. That — not any single feature — is what durable by design means:the run outlives the pod, the approval outlives the weekend, and the failure modes are the ones you chose in advance.
References
- LangGraph persistence, durability modes andcheckpointers — docs.langchain.com (durability modes: LangGraph 0.6, 2025; DeltaChannel beta, per-node timeouts, error handlers, RunControl.request_drain(): LangGraph 1.2.0 release, May 12, 2026)
- LangGraph interrupts and human-in-the-loop — docs.langchain.com
- Agent Server changelog — docs.langchain.com/langsmith(Redis run queue: 0.8.0, April 16, 2026;
- Delta Channel in the Postgres checkpointer: 0.8.6, May2026)
- LangSmith Sandboxes (GA May 2026; architecture vendor-described) — LangChain blog, March 17 and May 13, 2026
- LangSmithLLM Gateway (private beta) — LangChain blog, May 13, 2026
- LangSmith Deployment / managed runtime options —docs.langchain.com/langsmith (verify entitlements per plan)
.png)
