Enterprises are bringing AI agents in-house, running the models on infrastructure they own instead of a vendor's cloud, to keep control of their data. The harder question is who owns what those agents conclude, and who is allowed to write it down.
I was telling my wife about a client bringing their AI agents in-house. Sensitive data, a board nervous about the cloud, so the agents come off the public cloud and onto hardware the company owns. She is a chartered accountant and, as you’d expect, immediately asked me who signs it off.
I started explaining the infrastructure. That wasn’t her question. She wanted the output. When an agent works something out about a customer or a contract, who checks it, and who accepts it as the version the business uses from then on?
Nobody, in most of the deployments I have seen. Nobody signs anything.
That is not one unlucky client. IBM’s Institute for Business Value found two-thirds of the CIOs and CTOs it surveyed are accountable for AI systems they do not fully control, which is the same gap seen from the technology leader’s chair.
Her question is also more current than it sounds. Agents came out from behind the chat window and started acting directly on systems of record. Open-weight models under MIT and Apache licences narrowed the capability gap to under a year, and serving them yourself turned into an infrastructure job. Meanwhile every agent platform of any size began offering to hold the enterprise’s memory on its behalf. Those three changes are why the trade press has discovered self-hosted agents.
Every enterprise already runs a memory architecture with strict rules about what gets remembered and who may write it down. It is the finance function.
A transaction lands in the day book as it happens, in order, and nobody makes decisions from the day book. Posting to the ledger is a deliberate act governed by rules. At period close a human signs, and the accounts become the version the business defends to an auditor. Working papers get binned.
Five centuries of refinement sit behind that structure, and an agent stack needs the same four levels, differing in purpose, lifespan, ownership and evidentiary weight. The audit stream is the day book: every prompt, retrieval, tool call and output, appended and never edited, kept because it is evidence. Ephemeral memory is the working papers, the scratch reasoning inside a single run, and it should die when the run dies. Most of the context bloat I see is ephemeral memory nobody arranged to throw away. State memory is the subledger, scoped to a claim, a case, an account, a campaign, opening with the entity and closing with it. Institutional memory is the ledger, the ratified account of how the business works and the decisions taken with the reasons behind them. It is the only level that compounds.
The rules at each boundary
All the engineering is in the movement between them, and it needs rules at every boundary.
What earns promotion between memory levels?
An agent’s inference is a candidate. Something has to test it: corroboration against a system of record, a confidence threshold, a human adjudicator, agreement across independent runs.
Who may promote?
This is the one my wife would not let go of. Accountants separate the person who records from the person who approves and treat it as non-negotiable. Agent platforms ship the opposite as a feature. An agent that writes to institutional memory and ratifies its own entry is a bookkeeper signing off their own accounts.
What happens on contradiction?
Most agent stacks overwrite in silence. Bookkeeping never erases. You post a correcting entry and both stay visible, which is how an auditor reconstructs what you believed in March.
What decays?
Accounting has depreciation and provisions, apparatus for a fact losing value over time. Most agent memory keeps everything at full confidence until somebody notices. A fact that was true about a business that no longer exists is worse than a gap, because a gap prompts a question.
Whose memory is it anyway?
An agent is a combination of a model, a harness that governs what the model sees, a set of tools it can call, and a memory of what it has already worked out. Bringing agents in-house moves the model, and the harness and the tools usually follow it. Memory is the part nobody specifies, and it is the only part that accumulates. On the rental side, the harness is moving the other way. OpenAI now runs it for you, and Google sells the orchestration while renting the models underneath. Rent the agent rather than the model and the memory travels with it.
The memory belongs to whoever holds the write-back keys, and where the GPUs sit does not decide that.
Which returns me to the client with the new GPUs. What went onto them was the model.
Self-hosting it does one real thing and it does it properly. Inference needs cleartext, so the prompt is decrypted in memory to be processed, and choosing a region does not change where a vendor is incorporated. A US-incorporated provider stays within reach of US criminal process for data it holds. Bringing the model in-house removes that question instead of narrowing it, and for some regulated workloads that is a good reason to do it.
IBM’s self-hosted release for Bob in October promised control over data residency, security policies and AI governance. Memory was not on the list. Self-hosting settles transit, and transit is where it stops. Custody is a different question. A company can own every GPU in the building and still have four teams whose memory stores disagree about what a customer is. Another can rent frontier reasoning, treating the model as a function that takes scoped context and returns a candidate, and hold all four levels itself.
Custody is where the market is moving, and the direction does not favour the enterprise. Nearly every agent platform now offers to hold your memory for you, described as shared business context or organisational memory. Oracle’s agent memory promotes a conversation into long-term memory by having a model extract it. Google’s Memory Bank does the same, automatically. They are good products. Most teams have already swapped a model at least once, and it cost them a configuration change and an afternoon. Swapping a memory store costs every conclusion your business has reached and can no longer prove on its own.
So ask a vendor where each of the four levels lives, who may write to the one that persists, what happens when a new fact contradicts a ratified one, and what an export looks like. The ones who cannot answer are selling a runtime. The ones who will not answer have already told you where the lock-in is, and would much rather you kept talking about hosting.
