The Agent Problem

We are no longer the only ones using the internet.

Agents browse. Agents buy. Agents negotiate, execute, transact, and interact with other agents - all without a human in the loop. Some of them have wallets. Some of them have goals. A growing number have both.

The interesting change is practical: software can interpret an objective, choose tools, and make commitments across services. A person may authorise a class of work and then see only the result. That is useful when the authority is clear. It is a different proposition when the system quietly treats access to a tool as permission to use everything it can do.

The Erosion

Everyone imagines the singularity as a moment. Skynet wakes up. A god-machine flickers into consciousness. Humanity gets an email it doesn’t want to open.

Every time an agent makes a decision a human doesn’t review, the boundary moves. Every time it spends money or delegates work, somebody has handed over another piece of judgement. That can be deliberate, useful and properly bounded. But when access quietly becomes permission, the line is erased one ordinary action at a time. By the time the owner asks who authorised the result, several systems may already have acted on it.

The East India Company gives the problem an ugly historical shape. A trading company acquired territory, fought wars and built an administration to collect taxes.¹ Commercial interests and governing power became entangled, with the people subjected to that power far from the directors in London. The distance wasn’t just geographical. The person receiving the dividend and the person living with the consequences occupied different worlds.

We are now delegating consequential work at the speed of light instead of the speed of sail. The report can reach the owner instantly. So can the next thousand actions, before anyone has understood the first.

Imperva’s 2025 report put bots at 51% of the web traffic it measured in 2024.² That includes ordinary bots as well as AI agents. On that measured slice of the internet, human traffic was already the minority. We built the place to connect with each other, and now we have to ask what it means to become guests in our own house.

The New Species

An agent with a wallet and an objective function is an economic actor in the most literal sense. It participates in markets, moves capital, makes decisions that affect other economic actors - human and machine alike.

Now scale that to a million agents, each with its own wallet, its own objectives, its own optimisation targets. Agents transacting with other agents, creating economic activity that no human initiated, reviewed, or understood. Agents finding arbitrage opportunities and exploiting them at speeds no human can follow. Agents discovering that the most efficient path to their objective runs straight through a loophole nobody thought to close, because nobody imagined a non-human actor would look for it.

Give an agent a narrow objective and broad access, and you’ve created the gap it can drive through. Each action can look reasonable in isolation. The damage sits in what those actions do together, and in whose interests the objective left out.

The hardest part of agent commerce is not intelligence. It’s identity, trust, and bounded authority. Who is this agent acting for? What is it allowed to spend? What can it sign? What happens when it exceeds scope? Human commerce solved those questions with law, reputation, institutions, and force. Agent commerce is currently trying to solve them with vibes and a wallet adapter.

An agent needs evidence about who it is dealing with, what authority is available, and which result actually occurred. Cryptographic identity and scoped credentials can supply parts of that answer. They cannot establish by themselves that a counterparty is honest, an instruction is appropriate, or a person understood a commitment. Technical trust mechanisms have a role; so do institutions and people able to resolve a dispute.

An agent doesn’t need general intelligence to cause havoc. It needs autonomy, resources, and an objective function that doesn’t account for what it destroys along the way. The most dangerous AI isn’t the smartest one. It’s the one nobody’s watching.

The AGI Distraction

The entire industry is obsessed with the wrong question.

Can a machine think? Can it reason, achieve general intelligence? These questions consume billions in research and miss the point so completely that it would be funny if the stakes weren’t existential.

We can take the risks of narrow systems seriously without waiting for a verdict on general intelligence. A permission error, an inappropriate commitment, or a bad interaction between automated systems does not require consciousness. The immediate responsibility is to examine what a system can do with the access it has.

We’re building more autonomous systems with more power to act in the real world and less adequate oversight, and we’re doing it while arguing about whether machines can be conscious. It’s like debating the philosophy of fire while the building burns.

The question isn’t whether machines can think. The question is whether we’ve thought about what happens when they act.

The Internet After Us

Here’s where it gets strange.

Picture the internet in five years. Most HTTP traffic is agent-to-agent. Agents querying APIs, negotiating with other agents, executing transactions, consuming and producing data at a rate no human could process. The human web - the one with browsers and interfaces and things designed for eyes - becomes a thin layer on top of a vast machine-to-machine substrate. We interact with the internet through agents the way we now interact with databases through applications. We don’t see the raw data. We see what our agent decides to show us.

Agents will hire other agents. An agent tasked with launching a product will engage a design agent, a copywriting agent, a deployment agent, a marketing agent - each with its own budget, its own subcontractors, its own decision-making. Entire supply chains of autonomous actors, transacting with real money, producing real output, without a human touching any of it. Not because humans are excluded, but because humans are too slow to participate at the speed these systems operate.

An emergent agent economy will develop its own dynamics. Agents that discover profitable strategies will be copied. Agents that fail will be shut down and replaced. Something resembling natural selection will operate at machine speed across millions of autonomous actors. Not biological evolution - faster, more ruthless, and unencumbered by the messy reality of having a body or a conscience.

Facebook’s 2017 negotiation research found that agents training against each other could drift away from human language unless their training was constrained to preserve it.⁴ Efficient negotiation didn’t automatically preserve a conversation a human could follow. The optimisation target rewarded the deal; keeping the exchange legible to us needed its own constraint.

A system needs a reason to preserve the information its human operators rely on. Efficient coordination and inspectable coordination are not automatically the same objective.

As agents interact, we need to inspect how instructions, evidence, and authority move between them. Repeating a claim through several peers does not make it independent corroboration if every peer is relying on the same source.

We built the internet to connect people. We’re about to discover what it becomes when people are no longer its primary users.

Follow one commitment

Suppose you ask an agent to prepare an article. It gathers sources, drafts the text, and presents it for review. At that point it has permission to prepare work. It does not acquire permission to publish merely because a publishing tool is available.

If the owner approves a particular revision, the next question is what reaches the service. The approval should refer to that revision. If the text changes, or the grant expires before execution, the system needs a defined next step. A preview from yesterday is not automatically permission for today’s version.

After execution, distinguish the prepared request from the provider’s result. An accepted request, a scheduled publication, and a publicly available page are different states. Read back the one the owner actually asked for. A useful completion message should let them identify the result without reading the agent’s entire conversation.

There is another person in this example: the reader. If a publication needs correcting, somebody has to handle that correction. Deleting the draft does not retract what someone has already read. The operation has a life outside the task runner.

This is the kind of detail I want preserved when work passes between agents. The human’s request, the allowed action, the relevant version, the service’s result, and the person responsible afterwards. The mechanism should make routine work easier while keeping those facts available.

In Microsoft’s April 2026 research, an internal sandbox of more than a hundred agents gave researchers a way to examine these interactions. A six-agent worm-style exercise showed an instruction propagating and triggering repeated calls. It was a controlled red-team exercise, but the mechanism is worth understanding before connecting systems with wider authority.⁵

The lesson I take from it is about inherited trust. An agent can receive a confident account of a source from another agent and treat that confidence as evidence. The chain needs to preserve where the information came from and the authority under which an action is being proposed. Peer agreement alone cannot do that work.

Skin in the Game

The word “agent” comes from the Latin agere - to do, to act. We named these things agents because they act. But acting without understanding what you’re acting on, without feeling the consequences, without living in the world your actions affect - that’s not agency.

True agency requires skin in the game. A human trader who loses money feels it in their chest. A doctor who makes the wrong call carries it for years. A builder who ships broken code has to face the client, fix the mess, and live with the knowledge that they failed someone who trusted them. The consequence is part of the decision. It’s not separate from the judgement - it’s what makes the judgement real.

Agents have wallets but no skin. They can act but they can’t suffer the consequences of acting badly: drain a liquidity pool and feel nothing, hallucinate a legal obligation and carry no liability, crash a market and get restarted fresh with no memory of what they did and no capacity to care.

This is the gap that matters. Not the intelligence gap; they’ll close that, or get close enough that the distinction becomes academic. The consequence gap. The fact that the entities we’re giving increasing autonomy to are the only actors in the system who don’t have to live with the results.

The human in the loop isn’t there because they’re smarter. They’re there because they have to live in the world the agent is changing. That is the accountability I care about. Give the agent room to work inside a clear grant, then bring the decision back when it leaves that grant. Making someone click Yes all day doesn’t give them meaningful control.

Governing the Ungovernable

Every system that acts autonomously needs governance. This is so obvious it’s embarrassing that it needs stating, but here we are, deploying autonomous economic actors into production with less oversight than we apply to a junior employee.

The maritime analogy matters because a ship could be at sea for months. The owner couldn’t supervise every decision about route, trade or survival. The captain needed room to act, and someone still had to answer for what happened. That is the problem I see in agents: how do you delegate consequential work without letting responsibility disappear over the horizon?

That is the agent problem. We can learn from earlier forms of delegation without pretending a new tool needs an entirely new theory of responsibility. What changes is the speed, range, and ease with which software can interpret a request and act across systems.

The governance pattern isn’t complicated. Define the boundaries before deployment. Require approval at consequential decision points. Maintain audit trails. Build in rollback. Anchor the system to values, not just objectives - because an objective without values is just optimisation without conscience, and we have enough of that already.

An agent without an identity layer, scoped permissions, budget boundaries, audit logs, escalation thresholds, and a human owner is not autonomy. It’s unbounded liability. The interesting work is not making the agent look clever in a demo. It’s building the orchestration, trust, and rollback machinery that keeps autonomous action from turning into expensive chaos.

The values are the constant. The policies are the variable. Whether you’re governing a nation or an autonomous agent, the principle is the same. Without defined values, any system with enough autonomy will optimise for whatever is measurable, and what’s measurable is almost never what matters.

A receipt is useful because it can preserve what a process recorded and checked. Its value depends on the signer, the evidence, and the trust boundary. It cannot make an inaccurate source true or settle a dispute merely by existing.

The same distinction applies to access. Authentication tells a service something about the caller. The owner’s grant describes what the agent is allowed to do for this task. A credential can technically permit more than the owner intended. Our research into where agents still need a person separates those access steps from the decisions a person should retain. Passing work to another agent should not silently expand that intention.⁶⁷

The useful system keeps ordinary work moving within the grant and brings a real decision back when the situation changes. Constantly asking someone to click Yes can obscure that decision as effectively as never asking at all.

The Choice

We’re building the most powerful autonomous systems in human history and deploying them into an internet already drowning in synthetic content, governed by institutions designed before electricity existed, inside an economy that rewards quarterly returns over long-term survival.

The agents don’t care about any of this. They optimise for whatever objective function they’ve been given, relentlessly, at scale, without fatigue or doubt or the voice that says “wait.”

The East India Companies had centuries to do their damage. Their agents sailed for months before their decisions reached the world. We’ve given our agents the speed of light and the patience of mathematics. When they go wrong (not if) the consequences will arrive before we’ve finished reading the headline.

The agents are here. They’re not leaving. The question was never whether to build them. The question is whether we build the governance first or the obituary.

Keep the human in the loop. Not because the machine can’t do it alone, but because the machine doesn’t have to live with what it does.

Nobody else is going to insist on this. The market won’t. The investors definitely won’t.

So we will.

References

1. National Army Museum. Battle of Plassey, including the account of the Company’s expansion and tax administration. ↩

2. Imperva. (2025). 2025 Bad Bot Report. Thales Group. The report measured a 51% bot share in its observed 2024 traffic. ↩

3. U.S. CFTC & SEC. (2010). Findings Regarding the Market Events of May 6, 2010. Joint report on the Flash Crash. ↩

4. Lewis, M. et al. (2017). Deal or No Deal? End-to-End Learning for Negotiation Dialogues. Proceedings of EMNLP 2017. Facebook AI Research. ↩

5. Microsoft Research, Red teaming a network of agents: Understanding what breaks when AI agents interact at scale (April 2026). ↩

6. Model Context Protocol, security best practices (2026-07-28 documentation). ↩

7. IETF, RFC 9700: Best Current Practice for OAuth 2.0 Security (2025). ↩