Long-running AI agents need three things that serverless request handlers do not give them: a process with no short execution cap, state checkpointed outside the process so a restart resumes from the last step, and presence in more than one location so a provider outage does not stop the agent. Most teams find these gaps after a prototype works in development. In production it dies mid-run when a function times out at 15 minutes, a deploy kills the container, or a zone goes offline.

Disclosure: Control Plane publishes this guide.

TL;DR

Long-running AI agents fail in production when the infrastructure treats them like short-lived request handlers.

  • Common serverless functions cap a single invocation or request at 5 to 60 minutes, which ends agents mid-task.
  • Checkpoint every meaningful step to a durable store outside the process so a crash resumes from the last completed step.
  • Make every tool call and side effect idempotent with stable operation keys.
  • Give each run a single owner through an expiring lease, and check a fencing token on every checkpoint write so a replica that lost its lease cannot commit progress.
  • Handle SIGTERM by finishing or checkpointing the current step within the termination grace period.
  • Run agent replicas in two or more locations, across providers if a provider outage must not stop the agent, and keep checkpoints in a store every location can reach.
  • Control Plane runs agents as long-running standard or stateful workloads across AWS, GCP, Azure, and Oracle Cloud Infrastructure locations and routes traffic to healthy replicas when one location fails.

How do you run long-running AI agents in production?

Production agents need a runtime that does not stop them after a few minutes, a state layer that survives process death, and enough redundancy to stay up through infrastructure failures.

  1. Pick a runtime with no short execution cap. Serverless functions such as AWS Lambda (15-minute cap) or Azure Functions on the Consumption plan (10-minute maximum) terminate an agent mid-run. Use a long-running container process, a job runtime that allows hours or days, or a durable execution engine that manages the lifecycle.
  2. Persist state per step to a durable store, with one owner per run. After each meaningful unit of work, write the agent’s progress, tool outputs, and context to a database in one transaction. Each run is owned by one replica through a lease that expires and is renewed while the replica works. Every checkpoint write carries the lease’s fencing token, and the store rejects writes with an older token. If the process dies, the lease expires and another process claims the run, loads the last checkpoint, and continues.
  3. Make tool calls idempotent with stable keys. If a crash happens after sending an email but before recording success, the agent must not send the email again on recovery. Side effects run at least once, so give each one an idempotency key the downstream API honors, or record the key before the call and check the provider for a prior send before retrying.
  4. Handle SIGTERM and finish or checkpoint within the grace period. Deploys, scale-in events, and node drains send SIGTERM. The agent should stop claiming new work, complete or checkpoint the current step, and release its lease before the grace period ends. Set the grace period longer than the longest single step plus any preStop delay; on Kubernetes the default is 30 seconds.
  5. Retry with backoff on model and tool errors. Rate limits, transient network failures, and tool timeouts are routine. Use exponential backoff with jitter, circuit breakers that open after repeated failures, and a fallback such as a second model provider or a paused run.
  6. Run replicas in two or more locations. A single-region deployment stops when that region fails. Deploy to at least two locations, ideally across cloud providers, and route traffic to whichever location is healthy.
  7. Monitor with per-run IDs and alerts on stuck runs. Carry a run ID through logs and traces, alert on runs that have not checkpointed within an expected interval, and track completion rate, retries, tokens, and cost per run.

How long can an agent run on common runtimes?

Execution limits vary widely, and hitting a cap mid-run loses work unless a durable layer sits on top.

Runtime Execution limit What happens at the limit or on interruption Durable option
AWS Lambda 15 minutes per invocation Invocation is terminated AWS Lambda durable functions: runs up to 1 year; progress is checkpointed and replayed, skipping completed steps
AWS Step Functions (Standard) Up to 1 year Execution fails with a States.Timeout error Built-in state management
AWS Step Functions (Express) Up to 5 minutes Execution fails with a States.Timeout error Use Standard for long runs
Google Cloud Run services 60-minute maximum request timeout Request is terminated None built in
Google Cloud Run jobs 10 minutes per task by default, up to 168 hours (7 days) Task is terminated; Google states the timeout is not a guarantee of uninterrupted execution None built in
Azure Functions (Consumption) 5-minute default, 10-minute maximum Function is terminated Azure Durable Functions
Azure Functions (Flex Consumption and Premium) 30-minute default, no maximum Scale-in and platform updates can still stop a run after a grace period Azure Durable Functions
Cloudflare Workflows No timeouts on durable multi-step execution; pauses for events or approvals Steps retry automatically Built in
Kubernetes Deployment or StatefulSet No cap on the process; 30-second default graceful termination period Pod receives SIGTERM, then SIGKILL after the grace period StatefulSet gives stable identity and storage; checkpointing is up to the application
Control Plane standard and stateful workloads No cap on the process; HTTP request timeout up to 600 seconds, or up to 3,600 seconds with the cpln/timeoutSecondsOverride tag; 90-second default termination grace period Replica is removed from load balancing, a default preStop hook waits half the grace period, then the container receives SIGTERM Stateful workloads keep a stable identity and a persistent volume with snapshots; checkpointing is up to the application

Short-lived serverless functions need a durable execution layer on top, and long-running container runtimes still need graceful shutdown and external state.

Where should agent state live?

The right state layer depends on your existing infrastructure, how complex the workflow is, and how long the agent needs to pause.

Tool What it checkpoints Where state is stored How it resumes after a crash Best fit
Temporal Workflow event history Temporal’s persistence layer Rebuilds state by replaying the event history, so LLM and tool calls belong in activities, whose recorded results are reused on replay; activities retry automatically by default Complex orchestration that runs from seconds to years
LangGraph checkpointer Graph state at every super-step, grouped by thread_id In memory (lost on restart) or a database such as Postgres, SQLite, or MongoDB (separate packages) Resumes from the last successful step without re-running completed nodes LangGraph agents that need step-level recovery
Inngest Each step as a retriable unit of work Inngest Completed steps are not re-run; a run can sleep for days Event-driven workflows with long pauses
Restate Agent, workflow, and service progress Restate runtime Durable processes recover from persisted state Agents and services that need durable process semantics
DBOS Workflows and steps Postgres Recovers from Postgres checkpoints after crashes or restarts Teams with Postgres that want minimal new infrastructure
AWS Step Functions Workflow state transitions AWS-managed Retries failed states per their Retry policy; failed Standard executions can be redriven from the failed state AWS-native orchestration
Azure Durable Functions State, checkpoints, retries, and recovery Azure-managed Recovery is managed by the runtime Azure-native workflows
Cloudflare Workflows and Durable Objects Multi-step execution state; Durable Objects hold strongly consistent, transactional storage per object Cloudflare Automatic retries; pauses and resumes for events or approvals Workflows with approval gates

A team that already runs Postgres can add durability with DBOS or the LangGraph Postgres checkpointer and little new infrastructure. Complex multi-day orchestration with many branches fits Temporal. The OpenAI Agents SDK documents integrations with Temporal, Restate, DBOS, and Dapr for long-running agents, and the OpenAI Responses API background mode runs long requests that are polled or delivered by webhook. A single cloud’s durable service keeps agent state inside that provider’s failure domain, so a provider outage stops the workflow along with the agent.

What stops a long-running agent?

Eleven failure modes regularly stop production agents, and each has a control that prevents lost work.

Failure What happens without a control Control
Runtime execution cap Agent is terminated mid-step and in-memory state is lost Run on a runtime with no short cap, or use a durable engine that checkpoints before the cap
Container restart or crash Process dies and unfinished work is lost Checkpoint to an external store after each step; the new process resumes from the last checkpoint
Duplicate runner Two replicas, or a cut-off replica and its replacement, run the same run and repeat side effects An expiring lease per run in a strongly consistent store, renewed while working, with a fencing token checked on every checkpoint write
Rolling deploy of a new version The running container receives SIGTERM and is replaced Handle SIGTERM by finishing or checkpointing within the grace period; limit how many replicas roll at once
Node drain or scale-in The container is evicted or removed A PodDisruptionBudget for node drains and cluster autoscaler node removal (it does not cover a Horizontal Pod Autoscaler reducing replicas), a graceful shutdown handler, and a checkpoint before exit
LLM API rate limit or timeout The step fails and may retry endlessly or crash Exponential backoff with jitter, a circuit breaker, a fallback provider, or a paused workflow
Tool call failure The side effect may or may not have happened Idempotency keys on side-effecting calls and a reconciliation check before retry
Credential expiry Calls fail with authentication errors Short-lived, automatically rotated credentials and alerts before expiry
Zone outage Replicas in that zone are unreachable Replicas across zones, with traffic routed to healthy zones
Region outage Replicas in that region are unreachable Replicas in two or more regions that resume from checkpointed state
Cloud provider outage Replicas on that provider are unreachable Replicas across providers, with checkpoints in a store the healthy locations can reach

Every control depends on state living outside the process and outside any single failure domain.

How do you keep an agent running through a deploy or a provider outage?

Graceful shutdown, rolling updates, multi-location replicas, and checkpoint resume work together.

When a deploy, scale-in, or node drain happens, the orchestrator sends SIGTERM to the container. The agent should catch it, stop picking up new steps, and complete or checkpoint the current step. A preStop hook runs before SIGTERM, and its time counts against the termination grace period, which defaults to 30 seconds on Kubernetes.

Rolling updates keep capacity during deploys by replacing replicas in batches (on Kubernetes, by default no more than 25% unavailable at once) and waiting for each new replica to pass readiness probes. A PodDisruptionBudget limits how many replicas are down at once during voluntary disruptions such as node drains; it does not throttle a Deployment’s own rolling update.

Multi-location replicas protect against zone, region, and provider outages. If the agent runs in two locations and one fails, the other keeps serving. Checkpointed state must live in a store the surviving replicas can reach, such as a Postgres database replicated across regions. Asynchronous replication can lose the latest checkpoints on failover, so the resumed step must be safe to repeat, and leases belong on one strongly consistent primary at a time. When that primary’s location fails, promote a replica in a healthy location and advance the fencing epoch so writes from the old primary’s era are rejected.

Resuming from checkpoints ties these together. A replica in a healthy location loads the last checkpoint and continues. Steps whose completion was recorded are skipped, the interrupted step runs again, and idempotency keys let that retry run without duplicate side effects. Routing users to the nearest healthy region covers the traffic side.

Where does Control Plane fit for long-running agents?

Control Plane handles Day 2 operations for you, including autoscaling, built-in observability, and security.

It runs five workload types: serverless, standard, stateful, cron, and VM. On Control Plane, standard workloads are long-running processes, so an agent runs continuously with no execution cap. Stateful workloads give each replica a stable identity and its own persistent volume; a replica keeps the same identity (such as agent-0) when it is restarted or rescheduled, each replica is bound to its own volume from a volume set, and volume sets support snapshots. An inbound HTTP request times out after at most 600 seconds, or up to 3,600 seconds with the cpln/timeoutSecondsOverride workload tag, so run longer agent tasks as background work in the workload, with clients polling for status or receiving a callback.

A Global Virtual Cloud (GVC) is Control Plane’s named set of locations across AWS, GCP, Azure, and Oracle Cloud Infrastructure regions, plus your own Kubernetes clusters. A workload runs in every location of its GVC. Each workload’s endpoint routes clients to the nearest location where it passes readiness probes, so when one location fails, traffic reaches the healthy ones. In-flight work in the failed location resumes elsewhere only if its state was checkpointed to a store the other location can reach and its lease has expired; the fencing token on checkpoint writes keeps a cut-off replica from committing progress for the same run, and idempotency keys keep its side effects from repeating.

For that store, Control Plane’s Template Catalog includes PostgreSQL, PostgreSQL HA, and multi-location databases such as Postgres Multi-Location and pgEdge, which can back DBOS, the LangGraph Postgres checkpointer, or a custom checkpoint table; keep lease and fencing writes on a single primary, because multi-master replication resolves conflicts after the write. Temporal is also in the Template Catalog for teams that orchestrate with it. By default each replica gets a 90-second termination grace period, configurable in the workload’s rollout options. When a replica shuts down, Control Plane first removes it from load balancing, a default preStop hook then waits half the grace period (45 seconds at the default), and the container then receives SIGTERM, leaving the rest of the grace period for the agent to checkpoint. A queue-driven agent should also stop claiming runs when the preStop hook starts, since removal from load balancing only stops new HTTP requests.

Control Plane’s Universal Cloud Identity lets workloads call AWS, GCP, and Azure services with no long-lived cloud keys in your code. Control Plane Sandboxes run in Kata Containers microVMs and are persistent; they are for agents that build and run untrusted code, while the production agent runs as a workload. The Control Plane MCP server lets an agent operate infrastructure through the same Control Plane API as the CLI and console. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant.

During the October 20, 2025 AWS us-east-1 outage, 20% of Control Plane customers had AWS us-east-1 in their GVC. Their workloads were active and running in healthy locations within 10 seconds, with no customer downtime, according to Control Plane’s internal incident telemetry.

Related reading: where to run AI agents in production, 10 AI agent infrastructure platforms compared, the cost to run AI agents in production, and letting AI agents operate production infrastructure safely.

FAQ

These answers cover the questions engineering teams ask when they move AI agents from prototype to production.

How do you run long-running AI agents in production?

Run them as long-lived processes on a runtime with no execution cap, checkpoint state to a durable store after every meaningful step, make side-effecting tool calls idempotent, handle SIGTERM during deploys and scale-in, retry model and tool errors with backoff, and run replicas in two or more locations.

Can AWS Lambda run long-running AI agents?

Only with a durable layer on top, because a single Lambda invocation stops at 15 minutes. AWS Lambda durable functions extend runs to up to 1 year by checkpointing progress and replaying after failures, and AWS Step Functions Standard workflows also run up to 1 year.

How do I keep an AI agent running if the server restarts?

Checkpoint the agent’s state to an external durable store after each completed step, then load the last checkpoint when the process restarts. Steps recorded as complete are skipped, and idempotency keys let the interrupted step run again without duplicate effects.

What is durable execution for AI agents?

Durable execution persists a workflow’s progress so that after a crash, restart, or deploy, it resumes from the last completed step. Temporal rebuilds state by replaying event history, Inngest treats each step as a checkpointed unit of work, and DBOS checkpoints workflows and steps to Postgres.

Where should AI agent state be stored?

In a durable external store that survives restarts and that every location running the agent can reach. Postgres is common, used directly by DBOS and the LangGraph Postgres checkpointer, while Temporal keeps state in its own persistence layer.

How do AI agents survive a cloud provider outage?

They run replicas in locations on more than one cloud provider and keep checkpointed state in a store reachable from all of them. When one provider fails, traffic moves to healthy locations, and a replica there claims the run, loads the last checkpoint, and continues. On Control Plane, a Global Virtual Cloud spanning AWS, GCP, Azure, and Oracle Cloud Infrastructure locations routes clients to healthy locations automatically.

Do I need Temporal to run long-running agents?

No. Temporal suits complex multi-day orchestration, and Inngest, Restate, DBOS, AWS Step Functions, Azure Durable Functions, and Cloudflare Workflows also provide durable execution with different trade-offs. A simple workflow can also use custom checkpoint logic on Postgres.

How does Control Plane run long-running AI agents?

As standard or stateful workloads with no execution cap, running in every location of a Global Virtual Cloud across AWS, GCP, Azure, and Oracle Cloud Infrastructure. Each workload’s endpoint routes to the nearest location that passes readiness probes, stateful replicas keep persistent volumes with snapshots, and a 90-second default grace period, configurable per workload, lets in-flight work drain and gives the agent time after SIGTERM to checkpoint.

What is the difference between agent memory and durable execution state?

Agent memory is the context an agent reasons with, such as conversation history and preferences, while durable execution state records which steps finished, what they returned, and what runs next. Memory can be summarized over time; execution state must be exact so a crashed agent resumes without repeating or losing work.

How do rolling deploys affect a running AI agent?

A rolling deploy replaces replicas in batches, and each old replica gets SIGTERM and a limited grace period. An agent that checkpoints on SIGTERM and releases its run continues on another replica from the last checkpoint; one that ignores SIGTERM loses its in-flight work.

Talk to the Control Plane team to run long-running AI agents across clouds, or sign up free to start building.

Originally published on the Control Plane blog.