Running AI Agents on Kubernetes: Isolation and State

Why Running AI Agents on Kubernetes Is a Different Problem

Running AI agents on Kubernetes is not the same job as hosting a web API. A typical REST service is stateless: a request comes in, a response goes out, and the pod that handled it can be destroyed a second later with nothing lost. An AI agent breaks that model. It holds a working directory full of cloned repositories, half-finished edits, cached embeddings, and a running task that might span twenty minutes or twenty hours. Kill the pod naively and you throw all of that away.

That single difference (agents are long-running and stateful) drives every design decision in this article. Kubernetes gives you three things that matter here: isolation between workloads, persistence for the data an agent accumulates, and lifecycle primitives that assume some pods keep their identity across restarts. Get those three right and running AI agents on Kubernetes becomes predictable. Get them wrong and you get corrupted workspaces, agents that talk to services they should never reach, and a 4am restart that silently deletes a day of work.

This is the pillar piece. If you want the smaller-scale version first, read our walkthrough on how to run an AI agent on a VPS, or the beginner path in deploying your first agent. Here we go deeper into the cluster.

Isolation: Namespaces, NetworkPolicy, and Resource Limits

Isolation is the first pillar because agents run untrusted code by design. An agent that can execute shell commands, install packages, and hit arbitrary URLs is, functionally, a small remote code execution surface you are choosing to host. You want tight blast-radius control.

Namespaces separate tenants and teams

A Kubernetes namespace is the coarse boundary. Put each customer, team, or agent class in its own namespace so that names, secrets, and quotas do not collide. Namespaces are also where you attach a ResourceQuota to cap how much CPU and memory a whole group of agents can consume, which stops one runaway loop from starving the rest of the cluster.

NetworkPolicy controls what an agent can reach

By default, every pod in a Kubernetes cluster can talk to every other pod. For agents, that default is dangerous. An agent running a research task has no business reaching your internal billing service. A per-instance NetworkPolicy fixes this by declaring exactly which egress and ingress traffic is allowed. The Kubernetes documentation on network policies walks through the full selector syntax; the short version is that you write a policy that denies all egress, then explicitly allow DNS and the specific API endpoints the agent needs.

Resource requests and limits keep agents honest

Every agent container should declare both requests (what the scheduler reserves) and limits (the hard ceiling). Requests let Kubernetes place pods sensibly. Limits stop a memory leak in a model client library from taking down the node. Without limits, one agent that fumbles a large file into memory can trigger an out-of-memory kill on unrelated pods sharing the machine.

Here is a concrete manifest that ties isolation and persistence together. It uses a StatefulSet so the agent keeps a stable identity, mounts a PersistentVolumeClaim at /workspace, and sets resource bounds:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: agent-workspace
  namespace: agents-team-a
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: coding-agent
  namespace: agents-team-a
spec:
  serviceName: coding-agent
  replicas: 1
  selector:
    matchLabels:
      app: coding-agent
  template:
    metadata:
      labels:
        app: coding-agent
    spec:
      containers:
        - name: agent
          image: registry.example.com/openclaw:1.4.2
          env:
            - name: ANTHROPIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: agent-secrets
                  key: anthropic-api-key
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2"
              memory: "4Gi"
          volumeMounts:
            - name: workspace
              mountPath: /workspace
      volumes:
        - name: workspace
          persistentVolumeClaim:
            claimName: agent-workspace

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: agent-workspace
  namespace: agents-team-a
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: coding-agent
  namespace: agents-team-a
spec:
  serviceName: coding-agent
  replicas: 1
  selector:
    matchLabels:
      app: coding-agent
  template:
    metadata:
      labels:
        app: coding-agent
    spec:
      containers:
        - name: agent
          image: registry.example.com/openclaw:1.4.2
          env:
            - name: ANTHROPIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: agent-secrets
                  key: anthropic-api-key
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2"
              memory: "4Gi"
          volumeMounts:
            - name: workspace
              mountPath: /workspace
      volumes:
        - name: workspace
          persistentVolumeClaim:
            claimName: agent-workspace

The secretKeyRef matters: API keys belong in a Kubernetes Secret, not baked into the image or passed as a plain literal. Combine that with a NetworkPolicy in the same namespace and you have a workload that is fenced in on the network, capped on resources, and reading its credentials from a managed store.

Persistence: PVCs and Why /workspace Must Survive Restarts

The second pillar is persistence, and it is where most first attempts fail. A container's root filesystem is ephemeral. Anything written inside the container that is not on a mounted volume disappears the moment the pod restarts, and pods restart constantly: node drains, image updates, out-of-memory kills, and rolling deploys all recreate the container.

PersistentVolumeClaims give an agent durable storage

A PersistentVolumeClaim is a request for a chunk of durable disk that outlives the pod. The cluster satisfies it from a PersistentVolume, usually backed by cloud block storage such as an EBS volume or a DigitalOcean volume. You mount the PVC at the path your agent treats as home, typically /workspace, and now the cloned repos, the vector cache, and the in-progress edits survive a restart. The upstream reference on persistent volumes covers access modes and reclaim policies in detail.

The access mode matters. ReadWriteOnce binds the volume to a single node, which is correct for a single-replica agent that owns its disk. If two pods tried to mount the same ReadWriteOnce volume across nodes, one would fail to start. That constraint is a feature for agents: you almost never want two agent processes writing the same working directory at once, because they would race on the same files.

StatefulSet vs Deployment for agents

This is the choice people get wrong most often. A Deployment treats its pods as interchangeable cattle: any replica can be replaced by any other, and volume claims are shared or templated loosely. A StatefulSet gives each pod a stable network identity and, through volumeClaimTemplates, a stable one-to-one binding to its own PVC.

For agents, prefer a StatefulSet when each agent instance owns durable state that must follow it. The stable name (coding-agent-0) and the sticky volume mean that when the pod restarts, it comes back as the same logical agent with the same /workspace. A plain Deployment is fine only for genuinely stateless helper agents that recompute everything on each run and store results in an external database or object store.

A quick decision table:

State: Why Agents Are Long-Running, Not Stateless Handlers

The third pillar is conceptual, and it explains the other two. Web services are modeled as stateless request handlers on purpose, because that makes them trivial to scale horizontally. AI agents violate that assumption in three ways.

First, they carry conversational and task state. An agent working through a multi-step plan holds context that is expensive or impossible to reconstruct from scratch. Second, they do real filesystem work. A coding agent clones a repo, runs a build, and edits files across many turns; that directory is the agent's memory of the job. Third, their runs are long. A research agent might crawl and summarize for an hour. A request-response mental model would time out and lose everything.

Once you accept that agents are long-running and stateful, the Kubernetes choices follow naturally. You reach for StatefulSets because identity matters. You attach PVCs because the working directory is precious. You set generous terminationGracePeriodSeconds so an agent can checkpoint its progress before the pod is killed during a rolling update, rather than being cut off mid-write. You add a preStop hook if the agent needs to flush state to disk on shutdown.

There is also a scheduling consequence. Because agent pods are heavier and stickier than stateless replicas, you want them to reschedule cleanly when a node dies. Set a PodDisruptionBudget so voluntary disruptions (node maintenance, cluster autoscaler scale-down) do not evict all your agents at once, and make sure the PVC's storage class supports being reattached on a new node.

The Honest Tradeoffs of Kubernetes for Agents

Kubernetes solves the isolation, persistence, and state problems well. It also carries a real cost, and pretending otherwise helps nobody.

The learning curve is steep. To run agents safely you need working knowledge of namespaces, RBAC, NetworkPolicy, PVCs and storage classes, StatefulSets, resource tuning, secrets management, and the CNI plugin that actually enforces your network rules. Each of those has sharp edges. A NetworkPolicy silently does nothing if your CNI does not support it. A PVC can get stuck in Pending because no storage class is set as default. A StatefulSet will refuse to reschedule if the volume cannot detach from a failed node.

The operational cost is ongoing. Someone patches the control plane, rotates certificates, upgrades node pools, watches for CrashLoopBackOff, and gets paged when the cluster autoscaler makes a bad call at 3am. For a solo builder or a small team, running a production-grade control plane is often more work than the agents themselves. We spelled out that same maintenance tax at smaller scale in the VPS write-up, and it only grows with a full cluster.

Here are the recurring items you own when you run the cluster yourself:

• Control plane upgrades and API deprecation tracking across versions

• Node OS patching and kernel security updates

• Certificate rotation and secret rotation

• Storage class and CSI driver maintenance

• Monitoring, alerting, and log aggregation for every pod

• Capacity planning so PVCs do not fill and nodes do not thrash

None of this is a reason to avoid Kubernetes. It is a reason to be clear-eyed about whether operating it is the work you want to be doing.

Where a Managed Platform Fits

This is the honest place for Sokko to enter the story. Sokko is a managed platform built specifically for AI agents, without you running any infrastructure yourself. Each agent gets its own private space, its files and memory survive restarts, your keys stay in secure storage, and team roles are built in rather than hand-rolled. You deploy a runtime like OpenClaw, Hermes, Paperclip, or OpenSRE from the dashboard in a couple of clicks. If you want to see that flow end to end, our guide on how to deploy a personal AI agent walks through it step by step.

The point is not that Kubernetes is bad. The point is that the isolation, persistence, and state model an agent needs is well understood, and you can either build and operate that model yourself or hand the operating to a managed platform. If you are still deciding how much infrastructure to own, the first-agent deploy guide is a good next step, and the official Kubernetes documentation remains the best reference for the primitives themselves.

Key Takeaways

Running AI agents on Kubernetes comes down to respecting that agents are stateful. Use namespaces, NetworkPolicy, and resource limits to fence each agent in. Use PersistentVolumeClaims and StatefulSets so the working directory survives every restart. Design around long-running tasks with grace periods, disruption budgets, and checkpointing. And be honest about the operational tax: the primitives are powerful, but someone has to run them. Whether that someone is you or a managed platform is the real decision.