What Is Context Engineering for AI Agents?
Context engineering for AI agents is the practice of deciding what information lands inside a model's context window on any given turn, and what stays out. A large language model does not remember your last conversation. It only sees the tokens you send in the current request. Everything the agent appears to "know" during a session, the system prompt, the running chat history, retrieved documents, tool outputs, and scratchpad notes, has to be assembled by your code before the API call and packed into a single request. Getting that assembly right is the difference between an agent that stays on task across 40 tool calls and one that loses the thread after 5.
This is why context engineering is not the same discipline as prompt engineering. Prompt engineering asks: how do I phrase one instruction so the model responds well? Context engineering asks a harder question: given a fixed token budget and a growing pile of history, retrieved facts, and tool results, which subset do I include right now, in what order, at what level of compression? Prompt wording is a small part of that. The bigger part is data plumbing, memory, and retrieval logic that runs on every turn.
If you have ever watched an agent behave brilliantly for ten minutes and then start hallucinating or repeating itself, you were watching a context management failure, not a model that got dumber. The model is stateless. Your context pipeline is what carries state forward, and it is the part you actually control.
The Context Window as a Finite Token Budget
Every model ships with a hard ceiling on how many tokens fit in one request. Claude models expose a 200k-token context window, and some configurations reach 1M tokens. Many GPT-class and open-weight models sit at 128k tokens. Those numbers sound enormous until you run a real agent. A single large source file can be 8k tokens. A verbose API response might be 15k tokens of JSON you mostly do not need. Ten turns of back-and-forth with tool calls can quietly consume 60k tokens before the agent has done anything hard.
Two costs push back against filling that window. The first is money. If a provider charges roughly $3 per million input tokens, and your agent sends 150k tokens of context on every one of 30 turns in a session, that single session bills 4.5M input tokens, about $13.50, before you count output. Multiply by thousands of users and sloppy context becomes a real line item. The second cost is quality. Models attend less reliably to material buried in the middle of a very long prompt, an effect often called "lost in the middle." A window that is technically 200k tokens does not mean 200k tokens of equally usable attention.
So the working mental model is a budget, not a bucket. You have, say, 180k usable tokens after reserving room for the model's reply. Every item you add competes for that space. Good context engineering for AI agents is continuous triage: keep what earns its place, drop or compress the rest.
Retrieval: Pulling In Only What the Turn Needs
Retrieval is how you avoid stuffing the entire knowledge base into every request. Instead of pasting a 300-page manual, you embed it once, store the vectors, and on each turn fetch only the handful of chunks relevant to the current question. This is the retrieval-augmented generation (RAG) pattern, and it is the backbone of most production agents.
A few common storage options:
pgvector: a PostgreSQL extension that adds a vector column and similarity search to a database you may already run. It keeps your embeddings next to your relational data, which simplifies operations. See the project at github.com/pgvector/pgvector.
Mem0: a memory layer built specifically for agents, handling extraction and retrieval of durable facts about a user. See github.com/mem0ai/mem0.
Zep: a memory server that summarizes conversation history and exposes fast retrieval, aimed at long-running assistants.
Dedicated vector databases such as Qdrant, Weaviate, or Milvus when you need billions of vectors and standalone scaling.
The tradeoff with retrieval is precision. Fetch too few chunks and the agent misses a fact it needed. Fetch too many and you both waste budget and dilute attention. Most teams tune a top-k value (often between 4 and 12 chunks), add a similarity threshold so weak matches are dropped, and sometimes rerank results with a smaller model before they enter context. Retrieval quality, not raw window size, is usually what separates a helpful agent from a confidently wrong one. If you want the deeper background on how memory stores like these fit together, our piece on the types of AI agent memory breaks down the categories.
Summarization and Compaction of History
Retrieval handles external knowledge. Compaction handles the conversation itself. Left alone, chat history grows without limit, and eventually a long session either overflows the window or spends most of its budget re-reading old turns.
The standard fix is a rolling summary. Keep the most recent turns verbatim, because recency matters most, and periodically compress older turns into a compact summary that preserves decisions, facts, and open tasks while discarding filler. A practical policy:
Keep the last 6 to 10 message pairs in full.
When history crosses a threshold (for example 40k tokens), summarize everything older than the recent window into 500 to 1,500 tokens.
Store the raw, unsummarized transcript somewhere durable so nothing is truly lost and you can re-summarize later if needed.
Compaction is lossy by design, and that is the honest tradeoff. A summary that is too aggressive drops a detail the user mentioned an hour ago, and the agent looks forgetful. A summary that is too gentle barely saves any tokens. Teams usually tune this empirically, and many keep the full transcript on disk so a later turn can retrieve a specific old exchange verbatim when it matters.
Tool-Output Management and a Token-Budget Sketch
Tool outputs are the most underestimated context hog. An agent that queries a database, calls a search API, or reads a file can pull back thousands of tokens of raw output, most of which is noise. Dumping that verbatim into context is how budgets evaporate. The discipline is to truncate, filter, or summarize tool results before they enter history: keep the fields the agent asked for, drop pagination metadata and null columns, and cap any single result at a sane size.
Here is a compact sketch of how a budget-aware assembler might prioritize and evict, in Python:
# Token budget assembler: fill from highest priority down, evict the rest.
BUDGET = 180_000 # usable input tokens, reply reserved separately
def count_tokens(text: str) -> int:
# Use the provider's real tokenizer in production.
return len(text) // 4 # rough 4-chars-per-token estimate
# Each block: (priority, label, text). Lower priority number = keep first.
blocks = [
(0, "system_prompt", system_prompt),
(1, "recent_turns", recent_turns_text),
(2, "retrieved_docs", top_k_chunks_text),
(3, "history_summary", rolling_summary_text),
(4, "tool_output", truncate(latest_tool_output, max_tokens=4_000)),
]
context, used = [], 0
for priority, label, text in sorted(blocks, key=lambda b: b[0]):
cost = count_tokens(text)
if used + cost <= BUDGET:
context.append(text)
used += cost
else:
# Evict: log what was dropped so behavior is debuggable.
print(f"dropped {label}: {cost} tokens, budget exhausted")
prompt = "\n\n".join(context)# Token budget assembler: fill from highest priority down, evict the rest.
BUDGET = 180_000 # usable input tokens, reply reserved separately
def count_tokens(text: str) -> int:
# Use the provider's real tokenizer in production.
return len(text) // 4 # rough 4-chars-per-token estimate
# Each block: (priority, label, text). Lower priority number = keep first.
blocks = [
(0, "system_prompt", system_prompt),
(1, "recent_turns", recent_turns_text),
(2, "retrieved_docs", top_k_chunks_text),
(3, "history_summary", rolling_summary_text),
(4, "tool_output", truncate(latest_tool_output, max_tokens=4_000)),
]
context, used = [], 0
for priority, label, text in sorted(blocks, key=lambda b: b[0]):
cost = count_tokens(text)
if used + cost <= BUDGET:
context.append(text)
used += cost
else:
# Evict: log what was dropped so behavior is debuggable.
print(f"dropped {label}: {cost} tokens, budget exhausted")
prompt = "\n\n".join(context)The point is not the exact code. It is that context assembly should be an explicit, inspectable step with priorities and a hard ceiling, not an accident of appending strings.
Eviction and Prioritization Strategies
Once you accept a fixed budget, you need a rule for what to drop when the budget is exceeded. A few strategies that work in practice:
Recency: keep the newest turns, evict the oldest. Simple and usually right for conversational agents.
Relevance: score each candidate block against the current query and evict the least relevant. This is retrieval applied to your own history, not just external docs.
Pinning: mark certain items as never-evict, such as the system prompt, the user's stated goal, and hard constraints like "the deployment target is production."
Tiered storage: hot facts live in context, warm facts live in a vector store one retrieval away, cold facts live in durable storage. Most memory belongs in the warm and cold tiers, not the window.
There is real tension here. Aggressive eviction saves tokens and money but risks dropping something the user will reference later. Conservative eviction keeps the agent well-informed but expensive and slower. There is no universal setting. The right balance depends on session length, how much users refer back to earlier turns, and your cost tolerance.
One structural requirement underlies all of this: your memory artifacts, the vector index, the rolling summaries, the raw transcripts, have to survive between runs. If the agent's storage is wiped every time its process restarts, none of these strategies help, because there is nothing to retrieve. That failure mode is common enough that we wrote a whole guide on why an AI agent forgets everything between sessions and how to give it durable state.
This is where the runtime matters. Sokko runs agents on secure infrastructure we manage, with durable per-agent storage that survives restarts, so the pgvector index, cached embeddings, and summary files an agent builds up are still there after a redeploy or a crash. If you want the general picture of the infrastructure behind always-on agents, we cover it in running AI agents on Kubernetes.
Putting Context Engineering Into Practice
Strong context engineering for AI agents comes down to a repeatable loop you run on every turn. Assemble the system prompt and pinned constraints first. Add the recent verbatim turns. Retrieve the top-k relevant chunks from your vector store and the rolling summary of older history. Compress or truncate any fresh tool output before it lands. Then check the total against your budget and evict from the bottom of the priority list until you fit.
Start measuring three things and the rest follows: tokens per turn, retrieval hit rate (did the fetched chunks actually contain the answer), and cost per session. When tokens per turn creep up, tighten tool-output truncation. When retrieval misses, tune chunk size and top-k. When cost spikes, compact history sooner. For the model-side details on window sizes and pricing that drive these decisions, the official documentation for your provider and the Kubernetes docs for the runtime side are worth keeping open. Context engineering is not a one-time setup. It is the ongoing work of keeping a stateless model well-fed, on budget, and on task.