What is the ai agent infrastructure stack?
An AI agent that works on your laptop is a script. An AI agent that works in production is a system. The ai agent infrastructure stack is the set of layers that turns the first into the second: the compute it runs on, the process that keeps it alive, the storage that remembers what it did, the secrets it needs, the network it reaches out through, the models it calls, the sandbox it runs code in, the telemetry that tells you what happened, and the access rules that decide who can touch it.
Skip a layer and you usually find out at the worst time. No process supervision means your agent dies at 3 a.m. and nobody notices until a customer does. No egress control means a prompt-injected agent quietly ships your database to an attacker. This guide walks the whole stack from the bottom up, explains what each layer does, what breaks without it, and names a real tool you can reach for at each one.
Here is the stack, bottom to top:
Compute: where the agent process executes.
Runtime and process supervision: what keeps it running.
State and memory: what it remembers between runs.
Secrets management: how it holds credentials.
Networking and egress control: what it can reach.
Model access: how it calls LLMs.
Tooling and sandboxing: how it runs code safely.
Observability: how you see what it did.
Access control: who is allowed to operate it.
The bottom layers: compute, runtime, and state
Compute: VMs, containers, and Kubernetes
Everything sits on compute. At the small end that is a single virtual machine, say a $6/month VPS with 1 vCPU and 1 GB of RAM. That is enough to run one lightweight agent. The moment you want isolation between agents, repeatable deploys, and the ability to run many of them at once, you package the agent as a container with Docker and schedule those containers with an orchestrator.
Kubernetes is the standard orchestrator for this. It decides which machine runs which container, restarts containers that fail, and lets you declare resources like cpu: 500m and memory: 1Gi per agent. If your agents need GPUs (for self-hosted models or heavy embedding work), Kubernetes handles GPU scheduling through the NVIDIA device plugin, so a pod that requests nvidia.com/gpu: 1 lands on a node that actually has one. You can read how the scheduler assigns work in the official Kubernetes documentation.
What breaks without it: with no orchestration you hand-place agents on machines, over-provision to feel safe, and rebuild by hand when a node dies. Bin-packing ten agents onto three nodes by hand is a bad weekend.
Runtime and process supervision
An agent is often a long-lived process: it waits for work, runs a task that takes minutes or hours, then waits again. Something has to keep that process alive. If the agent crashes on a bad tool call, something must restart it. If the machine reboots, something must bring it back.
On a single VM this is systemd or a process manager like supervisord. On Kubernetes it is the kubelet plus a liveness probe: you declare a health check, and if the agent stops responding the platform restarts the pod. Surviving a crash and a reboot without a human in the loop is most of what running an agent 24/7 actually means.
What breaks without it: the agent exits on the first unhandled exception and stays down. You find out from a user, not from a graph.
State and memory
By default a container is ephemeral. Kill it and everything it wrote is gone. Agents need three kinds of state that must outlive any single container:
A durable workspace: files the agent creates, clones, or edits. On Kubernetes this is a PersistentVolume mounted into the pod so the data survives a restart or a reschedule.
A database for structured state: task queues, run history, results. PostgreSQL is the boring, correct default.
A vector store for semantic memory and retrieval: embeddings the agent writes and searches. pgvector, Qdrant, or Weaviate all fit here.
What breaks without it: every restart is amnesia. The agent re-clones the repo, forgets the last ten runs, and repeats work you already paid for. Planning state is the step people skip when taking an agent from local to the cloud, and it is the one that bites hardest.
The security layers: secrets, networking, and access control
Secrets management
Agents hold real credentials: model API keys, database passwords, GitHub tokens, cloud keys. These cannot live in the image or in a committed .env file. A secrets layer stores them encrypted and injects them at runtime.
HashiCorp Vault is the full-featured option, with dynamic short-lived credentials and audit logs. If you are already on Kubernetes and want something lighter, Bitnami Sealed Secrets lets you commit an encrypted secret to git that only the cluster can decrypt. Either way the agent reads the secret from an env var or a mounted file, never from source control.
What breaks without it: a leaked repo or a chatty log line exposes your model API key, and you are the one paying for someone else's traffic.
Networking and egress control
Most people firewall inbound traffic and forget outbound. For agents, outbound is the dangerous direction. An agent that browses the web and runs tools is a prime target for prompt injection, and a compromised agent with open egress can exfiltrate whatever it can read.
Egress control means the agent can reach the endpoints it needs (your model provider, your APIs) and nothing else. On Kubernetes a NetworkPolicy expresses this as allow-list rules per pod. You also want inbound TLS so the agent's terminal or API is not open to the world.
What breaks without it: a single injected instruction turns your helpful agent into a data pump with a clear route out.
Access control
Once more than one person can operate agents, you need to say who can do what. Role-based access control (RBAC) is the mechanism: this engineer can deploy and read logs, that one can only read, the billing owner can see spend. In a multi-tenant setup, RBAC plus namespace isolation is what stops team A from reading team B's agent memory.
Kubernetes ships RBAC as first-class objects (Role, RoleBinding, ServiceAccount). What breaks without it: every operator is effectively root, one fat-fingered command wipes another team's workload, and you cannot answer the question "who deployed this?" during an incident.
The agent-specific layers: model access and sandboxing
Model access
Agents are useless without a model to think with, and this layer governs how the agent reaches one. There are two broad options:
Hosted APIs: Anthropic, OpenAI, and others. Fast to start, pay per token, subject to rate limits you must handle with retries and backoff.
Self-hosted models: run an open-weight model with vLLM or Ollama on your own GPUs. More control and no per-token fee, but you own the GPU bill and the ops work.
Most teams mix both. Whichever you pick, this layer is where you centralize API keys, enforce per-agent rate limits and budgets, and add a fallback when a provider returns a 429. Choosing between hosted and self-hosted is really the self-hosted versus managed question applied to the model itself.
What breaks without it: one runaway agent burns your whole rate limit, every other agent starts failing, and there is no budget ceiling to catch it.
Tooling and sandboxing
The thing that makes an agent an agent is that it acts: it runs shell commands, executes generated code, edits files. That is also the thing that can wreck your environment. Sandboxing gives the agent a real place to run code that cannot reach the host or other tenants.
In practice that is a container with a locked-down profile: dropped Linux capabilities, a read-only root filesystem where possible, and a strict egress policy (see the networking layer). gVisor adds a stronger syscall boundary if you run fully untrusted code. The point is that a stray rm -rf inside the sandbox costs you a pod, not your cluster.
What breaks without it: the agent runs generated code with host access, and one bad command takes down far more than itself.
Observability: knowing what your agent did
An agent that runs unobserved is a liability. When it does something surprising, and it will, you need to reconstruct what happened. Three signals, the classic pillars:
Logs: structured records of each step and tool call.
Metrics: token spend, task latency, error rate, restarts. Prometheus scrapes these and Grafana graphs them.
Traces: the causal chain of a single task across model calls and tools. OpenTelemetry is the vendor-neutral standard, and for LLM-specific traces (prompts, completions, cost per step) tools like Langfuse are built for it.
What breaks without it: an agent quietly loops on a failing tool for six hours, spends $400 in tokens, and the first signal you get is the invoice.
The full ai agent infrastructure stack in one table
Here is every layer, its job, and a concrete tool you can reach for:
| Layer | Job | Example tools |
|---|---|---|
| Compute | Run the agent process | Docker, Kubernetes, a VPS |
| Runtime / supervision | Keep it alive, restart on crash | systemd, supervisord, kubelet probes |
| State / memory | Remember across restarts | PersistentVolume, PostgreSQL, pgvector |
| Secrets | Hold credentials safely | HashiCorp Vault, Sealed Secrets |
| Networking / egress | Control what it can reach | NetworkPolicy, TLS ingress |
| Model access | Call LLMs with limits | vLLM, Ollama, hosted APIs |
| Tooling / sandbox | Run code without damage | gVisor, hardened containers |
| Observability | See what it did | Prometheus, Grafana, OpenTelemetry |
| Access control | Decide who operates it | Kubernetes RBAC, namespaces |
Read the stack top to bottom and you have a checklist. Most half-built agent deployments have the top (a model and some code) and skip the middle (state, secrets, egress, observability), which is exactly where production incidents come from.
How Sokko bundles the ai agent infrastructure stack
Assembling all nine layers yourself is a real project. You can absolutely do it: pick a cloud, learn Kubernetes, wire up Vault, write NetworkPolicies, stand up Prometheus, and configure RBAC. Plenty of teams do, and if you want that level of control, the self-hosted versus managed tradeoff is worth reading in full.
Sokko is the other path. It's secure, always-on infrastructure tuned for long-running agents, managed for you, and it ships the whole ai agent infrastructure stack as one thing: your agent stays alive, its files and memory persist, credentials are stored securely, access is private by default, and you get logs and sensible limits, already wired together for you. You create an instance from the dashboard (a npm i -g sokko CLI is there too if you prefer), pick a runtime (OpenClaw for a cross-channel gateway assistant, Hermes for long-running research, Paperclip for multi-agent orchestration, OpenSRE for site reliability), and each agent gets its own private URL. The layers are still all there. You just are not the one holding them together.
The stack does not change based on who assembles it. Compute, runtime, state, secrets, networking, model access, sandboxing, observability, and access control are the layers every production agent needs. The only real question is whether you build and operate them yourself or let a platform run them so you can spend your time on the agent instead.