Every vendor deck now has a slide called "memory". Few of them say what is actually stored, where, and under whose permissions. Here is the version we give CIOs before an architecture review.
Three kinds of memory, three different problems
Working memory is the context window: what the model can see while it handles one task. It is ephemeral, it is large, and it is where prompt-injection attacks land. The residency question here is simple: the working memory is sent to the model provider on every call. If that provider is outside the EU, so is your customer's data, for a few seconds, unless the gateway redacts it first.
Episodic memory is what the agent remembers across sessions about a specific customer, case or employee. "This customer prefers e-mail." "We already offered a payment plan in March." This is personal data by any definition. It needs a legal basis, a retention period and a way to delete it when someone exercises their rights. It belongs in your database, in your region, under row-level permissions that mirror the ones your people already have.
Semantic memory is organisational knowledge: policies, product documentation, procedures, past resolutions. It is retrieved, not learned. The risk is not residency but staleness and leakage: an agent that reads a document the asking employee is not allowed to see has just created an access-control incident. Retrieval must run under the identity of the person in the loop, not under a service account.
What memory costs
| Kind | Where it lives | Cost driver | Governance question |
|---|---|---|---|
| Working | Model provider, per call | Tokens | Redaction and zero retention |
| Episodic | Your database | Storage, deletion workflows | Legal basis, retention, erasure |
| Semantic | Your document and vector stores | Indexing and re-indexing | Permission-aware retrieval |
Working memory dominates the bill. A long context full of retrieved documents, sent to a frontier model on every turn, costs more than most teams expect. Routing simple turns to a smaller model and capping the retrieved context are the two levers that matter.
Questions to ask any vendor
- Which of the three kinds do you implement, and where does each one physically live?
- Can episodic memory be deleted per data subject within the GDPR deadline, and can you prove it?
- Does retrieval run under the end user's permissions or under a shared identity?
- Is anything stored by the model provider after the call? Show me the contract clause.
- How do I see what an agent remembers about a given customer, and correct it?
Memory is the part of an agent that outlives the conversation. Treat it like a database, because it is one.
The platform answers these with a short list: episodic memory in your Postgres, semantic retrieval through the identity provider, working memory through the gateway with redaction, and nothing retained by model vendors. That is not sophisticated. It is just written down.