Agent memory, explained for CIOs

PlatformAI engineer, Zürich28 October 20266 min read

Every vendor deck now has a slide called "memory". Few of them say what is actually stored, where, and under whose permissions. Here is the version we give CIOs before an architecture review.

Three kinds of memory, three different problems

Working memory is the context window: what the model can see while it handles one task. It is ephemeral, it is large, and it is where prompt-injection attacks land. The residency question here is simple: the working memory is sent to the model provider on every call. If that provider is outside the EU, so is your customer's data, for a few seconds, unless the gateway redacts it first.

Episodic memory is what the agent remembers across sessions about a specific customer, case or employee. "This customer prefers e-mail." "We already offered a payment plan in March." This is personal data by any definition. It needs a legal basis, a retention period and a way to delete it when someone exercises their rights. It belongs in your database, in your region, under row-level permissions that mirror the ones your people already have.

Semantic memory is organisational knowledge: policies, product documentation, procedures, past resolutions. It is retrieved, not learned. The risk is not residency but staleness and leakage: an agent that reads a document the asking employee is not allowed to see has just created an access-control incident. Retrieval must run under the identity of the person in the loop, not under a service account.

What memory costs

KindWhere it livesCost driverGovernance question
WorkingModel provider, per callTokensRedaction and zero retention
EpisodicYour databaseStorage, deletion workflowsLegal basis, retention, erasure
SemanticYour document and vector storesIndexing and re-indexingPermission-aware retrieval

Working memory dominates the bill. A long context full of retrieved documents, sent to a frontier model on every turn, costs more than most teams expect. Routing simple turns to a smaller model and capping the retrieved context are the two levers that matter.

Questions to ask any vendor

  1. Which of the three kinds do you implement, and where does each one physically live?
  2. Can episodic memory be deleted per data subject within the GDPR deadline, and can you prove it?
  3. Does retrieval run under the end user's permissions or under a shared identity?
  4. Is anything stored by the model provider after the call? Show me the contract clause.
  5. How do I see what an agent remembers about a given customer, and correct it?
Memory is the part of an agent that outlives the conversation. Treat it like a database, because it is one.

The platform answers these with a short list: episodic memory in your Postgres, semantic retrieval through the identity provider, working memory through the gateway with redaction, and nothing retained by model vendors. That is not sophisticated. It is just written down.

PlatformEUDACHEnterprise AI
AI engineer, Zürichunus · Applied AI for the European enterprise

Talk to the people who wrote this.