How to Scale AI Agents Without Breaking Production

What breaks first when scaling AI agents

Scaling AI agents sounds like a throughput problem and turns into a blast-radius problem. One agent that occasionally spikes to 4 GB of memory is a rounding error. A hundred of them sharing a node, each holding a browser session and a model client open, is an outage waiting for a trigger. This article covers what actually fails when you go from one agent to many, and the concurrency, isolation, cost, and observability controls that keep production intact.

The failure modes show up in a predictable order:

API rate limits. The model provider throttles you long before your servers sweat. This is usually the first wall.
Memory pressure. Agents that hold context, tool state, or headless browsers leak or balloon, and the kernel OOM-killer starts reaping processes.
Noisy neighbors. One runaway agent starves the others on the same node.
Cost surprises. Token spend grows superlinearly when agents retry, loop, or fan out sub-tasks.
Blind spots. With one agent you read the logs. With a hundred you cannot, and you have no idea which one is stuck.

If you have not yet run a single agent in production, start with host an AI agent on a server and come back once one instance is stable. Everything below assumes you already have one working and now need ten or a hundred.

The mental model: agents are stateful, bursty, and I/O-bound

A typical web request is short, stateless, and CPU-bound. An AI agent is the opposite. It can run for minutes or hours, it holds conversation and tool state the whole time, and most of its wall-clock time is spent waiting on a model API or a shell command. That combination breaks the usual autoscaling intuitions. You are not scaling CPU, you are scaling concurrent long-lived sessions, each with an unpredictable memory footprint.

Concurrency and rate limits

The first ceiling you hit when scaling AI agents is almost never compute. It is the model provider's rate limit. Anthropic and OpenAI both meter requests per minute (RPM) and tokens per minute (TPM), and those limits are per-organization, not per-agent. Ten agents sharing one API key share one budget.

Do the arithmetic before you scale. Suppose your account allows 400,000 input TPM and each agent step consumes roughly 8,000 tokens of context. That is about 50 agent steps per minute across your whole fleet, regardless of how many pods you run. Launching 100 agents against that limit does not give you 100x throughput. It gives you 100 agents taking turns and a lot of HTTP 429 responses.

A few concrete tactics:

Queue, do not stampede. Put a work queue in front of the fleet and let agents pull tasks. A pull model naturally caps concurrency at the number of running workers instead of letting every task start at once.
Back off on 429. Honor the Retry-After header. Exponential backoff with jitter prevents the thundering-herd retry that turns one rate-limit event into a sustained one.
Split keys by workload. Give the batch fleet a different API key and quota than the interactive one, so a nightly refactor job cannot starve a user-facing agent.
Cap per-agent concurrency. Most agent frameworks will happily open several tool calls in parallel. Setting a per-agent limit of 2 to 3 keeps a single agent from consuming the whole org quota.

Tune the token budget, not just the pod count

Adding pods does nothing if the bottleneck is TPM. The lever that actually helps is per-request token discipline: trim system prompts, summarize long histories instead of resending them, and cap max_tokens on the output side. A fleet that averages 4,000 tokens per step instead of 8,000 doubles its effective throughput against the same limit without a single extra pod.

Isolation and noisy neighbors

Once concurrency is under control, the next thing that breaks when scaling AI agents is isolation. Agents run untrusted-ish code by design: they execute shell commands, install packages, and spawn subprocesses. On a shared node, one agent that forks a compile job or leaks memory degrades every other agent next to it.

Kubernetes gives you the primitives to contain this, but only if you actually set them. The single most important control is per-pod resource requests and limits. Requests reserve capacity so the scheduler spreads pods sanely; limits cap the damage a single pod can do. The Kubernetes docs on managing resources for containers spell out the semantics, but the short version is: set both, and set the memory limit equal to the request so the pod gets the Guaranteed QoS class and is last in line for OOM eviction.

Here is a baseline Deployment plus a HorizontalPodAutoscaler for an agent workload:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: openclaw-agent
spec:
  replicas: 3
  selector:
    matchLabels:
      app: openclaw-agent
  template:
    metadata:
      labels:
        app: openclaw-agent
    spec:
      containers:
        - name: agent
          image: ghcr.io/openclaw/openclaw:latest
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "1"
              memory: "1Gi"
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: openclaw-agent
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: openclaw-agent
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

apiVersion: apps/v1
kind: Deployment
metadata:
  name: openclaw-agent
spec:
  replicas: 3
  selector:
    matchLabels:
      app: openclaw-agent
  template:
    metadata:
      labels:
        app: openclaw-agent
    spec:
      containers:
        - name: agent
          image: ghcr.io/openclaw/openclaw:latest
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "1"
              memory: "1Gi"
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: openclaw-agent
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: openclaw-agent
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

A note on the HPA: CPU utilization is a weak signal for I/O-bound agents that spend their time waiting on the model API, so a pod can be "busy" at 10% CPU. In practice you often scale on a custom or external metric instead: queue depth, or requests-in-flight. The HorizontalPodAutoscaler docs cover the type: External and type: Pods metric sources you will want for that. The stabilizationWindowSeconds of 300 above matters too: agents are long-lived, so scaling down aggressively kills work in progress. Give it a wide window.

Isolation beyond resource limits

Resource limits stop the memory and CPU blast radius. They do not stop the security one. An agent that runs arbitrary commands should not be able to reach another tenant's data or the cluster control plane. The controls worth setting:

Namespaces per tenant or team, with ResourceQuota and LimitRange so no single namespace can exhaust the cluster.
NetworkPolicies that default-deny egress and allow only the destinations an agent needs. An agent rarely needs to talk to your internal databases.
A restricted PodSecurity level so agents cannot run privileged, mount the host filesystem, or escalate.
Separate node pools for untrusted workloads if you are running agents from multiple customers.

This is exactly the layer that gets tedious to build and maintain yourself. Sokko runs each tenant in its own isolated, private space with per-agent resource limits already enforced, so a runaway agent hits its own ceiling instead of the neighbor's. You create an instance in the dashboard, or with one CLI command, and the resource limits that come with your plan travel with it:

sokko instances create --runtime openclaw --name outreach-bot \
  --byok --provider anthropic --api-key sk-ant-...

sokko instances create --runtime openclaw --name outreach-bot \
  --byok --provider anthropic --api-key sk-ant-...

The size (vCPU, RAM, storage) is set by your plan rather than per-instance, so you change tiers in the dashboard instead of passing a size in the request.

The point is not that you cannot build per-tenant isolation on raw Kubernetes. You can. It is that the noisy-neighbor blast radius is the thing most teams under-invest in until an incident forces the issue.

Cost at scale

Token spend is where scaling AI agents surprises finance. Cost does not grow linearly with agent count, because busier fleets retry more, hold longer contexts, and spawn sub-agents. Here is a realistic shape of how the numbers move as you scale a coding-agent workload. Treat the dollar figures as order-of-magnitude, using roughly $3 per million input tokens and $15 per million output tokens at mid-2026 frontier pricing.

Fleet size	Peak memory (aggregate)	Tokens/hour (approx)	Model cost/hour	Blast radius of one bad agent
1 agent	1-2 GB	~1.5M	~$8	Its own task only
10 agents	10-20 GB	~15M	~$80	One shared node, one API key's quota
100 agents	100-200 GB	~150M	~$800	Whole cluster, org-wide rate limit, the monthly bill

Two things fall out of that table. First, the model bill dwarfs the infrastructure bill: at 100 agents you might spend $800/hour on tokens and a fraction of that on compute. Optimizing pod density saves pennies while a smarter prompt saves dollars. Second, the blast-radius column is why isolation and cost controls are the same conversation. A single agent stuck in a retry loop at 100x scale is a four-figure hourly mistake.

Practical cost controls:

Set a hard token budget per task and kill agents that blow past it. A loop detector that trips at, say, 50 steps without progress prevents the classic infinite-retry bill.
Right-size the model. Not every step needs the frontier model. Route cheap classification or routing steps to a smaller model and reserve the expensive one for reasoning.
Cache aggressively. Prompt caching on repeated system prompts and tool definitions can cut input-token cost by more than half for agents that share a large fixed preamble.
Scale to zero when idle. Long-lived agent pods that sit waiting still hold reserved memory. If your workload is bursty, prefer a model that spins pods up per task and tears them down after.

Observability when you run many agents

With one agent you tail the logs. When scaling AI agents past ten, kubectl logs stops being a strategy. You need structured telemetry that answers three questions fast: which agents are running, which are stuck, and which are burning money.

The signals worth collecting from day one:

Per-agent token counters (input, output, cached) tagged with the task ID. This is your cost attribution and your loop detector.
Step latency and step count. A rising step count with no output change is the signature of a stuck agent.
Tool-call error rates. A spike in failed shell commands or 429s tells you whether the problem is the agent or the environment.
Pod lifecycle events. OOM kills, restarts, and evictions, ideally alerting before they cascade.

Emit these as structured logs and Prometheus metrics. A minimal metric set looks like this in your app code:

from prometheus_client import Counter, Histogram

agent_tokens = Counter(
    "agent_tokens_total",
    "Tokens consumed",
    ["agent", "task", "direction"],
)
agent_steps = Histogram(
    "agent_step_seconds",
    "Wall-clock per agent step",
    ["agent"],
)

def record_step(agent, task, in_tok, out_tok, seconds):
    agent_tokens.labels(agent, task, "input").inc(in_tok)
    agent_tokens.labels(agent, task, "output").inc(out_tok)
    agent_steps.labels(agent).observe(seconds)

from prometheus_client import Counter, Histogram

agent_tokens = Counter(
    "agent_tokens_total",
    "Tokens consumed",
    ["agent", "task", "direction"],
)
agent_steps = Histogram(
    "agent_step_seconds",
    "Wall-clock per agent step",
    ["agent"],
)

def record_step(agent, task, in_tok, out_tok, seconds):
    agent_tokens.labels(agent, task, "input").inc(in_tok)
    agent_tokens.labels(agent, task, "output").inc(out_tok)
    agent_steps.labels(agent).observe(seconds)

From there, a single Grafana panel of tokens-per-task over time will catch the runaway agent before the invoice does. Alert on two thresholds: absolute cost per task, and step count without a state change.

Correlate identity, not just pods

At scale you also need to know who launched what. When an agent misbehaves, "which pod" is less useful than "which user, which project, which credential." That is where access control and secret scoping intersect with observability. If every agent runs with the same god-mode API key you cannot attribute anything, and you cannot revoke one agent without revoking all of them. Read managing secrets for AI agents for the scoping and rotation patterns, and RBAC and team management for tying agent actions back to a real person in an audit log.

The bottom line

Scaling AI agents is less about raw horsepower and more about containing four things that all grow with the fleet: rate-limit contention, memory pressure, cost, and blindness. The playbook is consistent:

Queue work and respect provider rate limits before adding pods.
Set resource requests and limits on every agent, and default-deny network egress.
Watch the token bill more closely than the compute bill, because it is bigger.
Instrument per-agent tokens and steps so a stuck agent is visible in seconds.

You can assemble all of this on raw Kubernetes, and plenty of teams do. The tradeoff is that per-tenant isolation, per-instance limits, egress rules, and audit-friendly identity are exactly the parts that are fiddly to get right and painful to get wrong. Sokko exists to make those the default rather than a project. Whichever path you pick, decide your isolation and cost ceilings before you scale the fleet, not after the first four-figure hour.