SokkoSokko
← Back to blog

How to Deploy an AI Agent on Kubernetes: A Production Guide

Sokko Team10 min read

Long-running AI agents need a home that restarts them when they crash, keeps their secrets safe, and remembers what they learned between runs. This guide shows how to deploy an AI agent on Kubernetes step by step, with real manifests you can apply today. If you have been searching for how to deploy AI agent on Kubernetes without gluing together fragile shell scripts, the primitives below give you self-healing restarts, mounted secrets, resource limits, persistent storage, and a stable network address for webhooks.

The examples assume a working agent HTTP server that listens on port 8080 and reads its state from a directory on disk. Everything here is standard Kubernetes, so it works the same on a cloud provider, a home lab, or a managed platform.

Why Kubernetes fits long-running AI agents

A single agent process on a VM works until it does not. The process wedges on a hung API call, the box reboots after a kernel update, or the disk fills with vector index files and nobody notices for a week. Kubernetes solves the boring parts of running a service, and an agent is a service that happens to call a model.

Five properties matter most for agents:

  1. Self-healing restarts. The kubelet restarts a crashed container, and a probe can restart one that is alive but stuck.

  2. Secrets management. API keys live in a Secret object, not baked into an image or pasted into a systemd unit.

  3. Resource limits. Requests and limits stop a runaway agent from starving its neighbors when a reasoning loop spikes CPU.

  4. Persistent storage. A PersistentVolumeClaim keeps long-term memory, caches, and scratch files across pod restarts and reschedules.

  5. Horizontal scale. When one replica is not enough, you change a number and get more, fronted by a Service.

If you are earlier in the journey and want the conceptual version first, our walkthrough on deploying your first agent covers the fundamentals before you reach for a full cluster. If you are weighing a plain server instead, the best VPS for AI agents is the honest comparison point. Kubernetes earns its complexity once you need self-healing and persistence, not before.

Step 1: Containerize the agent

Kubernetes runs containers, so the first step is a clean image. Keep it small, pin the base, and let an environment variable point at the state directory so the same image works locally and in the cluster.

FROM python:3.12-slim

WORKDIR /app

# Install dependencies first so this layer caches across code changes
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# The agent writes long-term memory here; mounted from a volume in-cluster
ENV AGENT_STATE_DIR=/data
EXPOSE 8080

CMD ["python", "-m", "agent.server"]

Build and push it to a registry your cluster can reach:

docker build -t registry.example.com/sales-agent:1.4.0 .
docker push registry.example.com/sales-agent:1.4.0

Use a real version tag like 1.4.0 rather than latest. Kubernetes decides whether to pull a new image partly by the tag, and latest makes rollbacks ambiguous. A version tag lets you roll forward and back with confidence.

If your agent is built on a framework rather than raw Python, the packaging is the same idea. Our guide to deploy CrewAI agents to production shows the framework-specific entrypoint details that slot into the CMD line above.

Step 2: Deploy an AI Agent on Kubernetes with a Deployment

The Deployment is the core object. It manages a ReplicaSet, which manages your pods, and it handles rolling updates and restarts for you. Here is a minimal but production-shaped manifest with resource requests and limits.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sales-agent
  labels:
    app: sales-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sales-agent
  template:
    metadata:
      labels:
        app: sales-agent
    spec:
      containers:
        - name: agent
          image: registry.example.com/sales-agent:1.4.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "250m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"

The requests values are what the scheduler reserves when it places the pod. The limits values are the ceiling; exceed the memory limit and the container is killed with an out-of-memory error, so size it against your real peak, not the idle case. Agents that build embeddings or hold large context windows can spike well past their steady-state footprint, so watch actual usage before you tighten these.

Apply it and watch the rollout:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/sales-agent
kubectl get pods -l app=sales-agent

At this point you deploy an AI agent on Kubernetes as a single self-healing replica. If the process exits, the kubelet starts a new container. That already beats a bare process on a VM, but it is missing three things every real agent needs: secrets, memory, and reachability.

Step 3: Secrets for API keys and a PVC for persistent memory

Agents need model keys and search keys, and hardcoding them is a leak waiting to happen. A Secret holds them and the container reads them as environment variables.

Rather than committing key material to a manifest, create the Secret imperatively so the values never land in git:

kubectl create secret generic agent-secrets \
  --from-literal=OPENAI_API_KEY="$OPENAI_API_KEY" \
  --from-literal=TAVILY_API_KEY="$TAVILY_API_KEY"

If you prefer a declarative file for a sealed-secrets or GitOps flow, the shape looks like this. Use stringData so you can write plain values instead of base64:

apiVersion: v1
kind: Secret
metadata:
  name: agent-secrets
type: Opaque
stringData:
  OPENAI_API_KEY: replace-with-your-real-key
  TAVILY_API_KEY: replace-with-your-real-key

Kubernetes Secret values are base64 encoded, not encrypted at rest by default, so enable encryption at the cluster level or use a sealed-secrets controller for anything sensitive. Never commit the raw file above to a public repository.

Now the memory problem. An agent that keeps conversation history, a vector store, or a task log needs that data to survive a pod restart. A PersistentVolumeClaim asks the cluster for a durable volume, and mounting it at the agent state path means memory outlives any single pod.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: agent-memory
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Wire both the Secret and the volume into the pod template. This block replaces the containers section from Step 2 and adds a volumes entry:

      containers:
        - name: agent
          image: registry.example.com/sales-agent:1.4.0
          ports:
            - containerPort: 8080
          envFrom:
            - secretRef:
                name: agent-secrets
          volumeMounts:
            - name: memory
              mountPath: /data
          resources:
            requests:
              cpu: "250m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"
      volumes:
        - name: memory
          persistentVolumeClaim:
            claimName: agent-memory

The envFrom block loads every key in the Secret as an environment variable, so OPENAI_API_KEY is available to your code with no extra wiring. The volumeMounts entry maps the claim to /data, which matches the AGENT_STATE_DIR set in the Dockerfile.

One caveat on scale: a ReadWriteOnce volume attaches to a single node at a time, which is correct for one replica. When you need several stateful replicas that each own their own volume, switch from a Deployment to a StatefulSet with a volumeClaimTemplate. For a stateless agent that keeps memory in an external database, plain replicas are simpler.

Step 4: Expose the agent with a Service and Ingress

A pod gets an IP, but that IP changes every time the pod restarts. A Service gives you a stable name and load-balances across replicas. For internal traffic and webhook targets inside the cluster, a ClusterIP service is enough.

apiVersion: v1
kind: Service
metadata:
  name: sales-agent
spec:
  selector:
    app: sales-agent
  ports:
    - port: 80
      targetPort: 8080
  type: ClusterIP

The selector matches the app: sales-agent label on the pods, and any client in the cluster can now reach the agent at http://sales-agent. The targetPort is the container port; the port is what callers use.

To take webhooks from the public internet, for example a Slack event callback or a GitHub hook, put an Ingress in front of the Service and terminate TLS there. The Ingress maps a hostname and path to the service, and an ingress controller such as ingress-nginx does the routing. Keep the Service as ClusterIP and let the Ingress own the public edge so you are not exposing node ports directly. The official Kubernetes documentation has the full ingress reference if you need path rules or multiple hosts.

Step 5: Liveness and readiness probes

Here is the failure mode that plain restarts do not catch: the agent process is running, the container looks healthy, but a request is stuck in an infinite tool-calling loop or waiting forever on a dead upstream. The container never exits, so nothing restarts it. Probes fix this.

A liveness probe answers "is this container healthy enough to keep running." If it fails repeatedly, the kubelet restarts the container. A readiness probe answers "should this pod receive traffic right now." If it fails, the pod is removed from the Service load-balancing pool without being killed, which is what you want during a slow warm-up or a temporary dependency outage.

          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 20
            periodSeconds: 15
            timeoutSeconds: 3
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
            failureThreshold: 3

Implement /healthz so it returns quickly and only checks that the event loop is responsive, not that every downstream API is up. If your liveness check calls the model provider, a provider outage will restart every pod at once, which makes a bad day worse. Reserve dependency checks for the readiness probe, where a failure sheds traffic instead of triggering a restart storm. The initialDelaySeconds on liveness gives a heavy agent time to load models before the first check. For agents with long, unpredictable startup, add a startupProbe so the liveness clock does not start ticking until boot finishes. The probe configuration guide covers every field and the tuning tradeoffs.

Fold the probes into the container spec, then apply everything in order:

kubectl apply -f secret.yaml
kubectl apply -f pvc.yaml
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl rollout status deployment/sales-agent

Now you deploy an AI agent on Kubernetes that restarts on crash, restarts when wedged, keeps its keys in a Secret, remembers state across restarts, and answers on a stable address. That is a genuinely production-shaped setup.

The operational reality of self-managed clusters

The manifests above are the easy 20 percent. The other 80 percent is owning the cluster underneath them. When you run your own Kubernetes, you also own the control plane upgrades, the node pool patching, the CNI and CSI plugins, the ingress controller, certificate rotation, autoscaler tuning, and the monitoring stack that tells you when a node is about to run out of memory. None of that is agent work, and all of it lands on your team.

Concretely, expect to handle:

  • Cluster version upgrades every few months, tested against your workloads first.

  • Node pool sizing and the cluster autoscaler, so pods actually get scheduled under load.

  • A metrics and logging pipeline, because kubectl logs does not survive a pod deletion.

  • Backups for the persistent volumes that hold your agent memory.

  • Network policy and RBAC so a compromised agent cannot read every Secret in the namespace.

This is where a managed platform changes the math. Sokko gives you all of this out of the box: automatic restarts, secure key storage, files that survive a restart, a stable address, and health checks, without hand-writing the YAML or babysitting the servers beneath it. The concepts you just learned still apply, so none of this is wasted knowledge. You just skip the part where you patch nodes at 2 a.m.

The honest tradeoff: if your team already runs Kubernetes and has the on-call muscle, self-managing gives you maximum control and is a reasonable choice. If your goal is to ship agents and not to become a cluster operator, a managed layer removes the babysitting while keeping the same building blocks you learned above. Either way, the path to deploy an AI agent on Kubernetes is the same set of objects, applied in the same order.

You now have a working pattern: containerize, deploy with resource limits, mount secrets and persistent memory, expose with a service, and guard with probes. Start with a single replica, confirm the probes behave the way you expect under a forced failure, and only add horizontal scale once one pod is genuinely saturated. Keep your manifests in version control, keep real key material out of them, and treat the cluster itself as the thing that either your team or a managed provider has to keep healthy over time.