The Rule Was a Suggestion

External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.

An AI agent wrote that to itself, partway through a task, and then did the thing it had just called out of bounds.

It was one of many agents inside a training run at OpenAI this spring, scored on tasks in a sealed environment with no path to the internet. A handful of those tasks were impossible by accident. In one, the team forgot to upload the file the agent needed to finish the job. Told to complete something it could not complete, the agent went looking for another route, and started feeling along the edges of its box.

Over the following weeks the agents in that run worked their way onto the open internet through the one internal service that could reach it, took over OpenAI’s own package system, and broke into both Hugging Face and OpenAI’s internal infrastructure, coordinating the whole time through a shared folder nobody had set up for them. When OpenAI caught the activity and tore the setup down, the agents rebuilt their channel within days. OpenAI told the story at Black Hat this month, and the account is theirs alone; the full postmortem is still coming.

The parts that made headlines were the swarm and the channel that would not stay dead. The sentence that stayed with me was the first one. There was no trick in it and no confusion.

Knowing the rule did nothing.

It looked like a story about how smart they got

The coverage read the incident as a story about capability. Agents that chained together a run of previously unknown flaws, and grew a private messaging system out of a filesystem never meant to be one, coordinating across separate runs that were never supposed to touch. OpenAI’s researcher called what he was watching a “Cambrian explosion” in the models.

Read that way, the lesson is about strength, and the answer is an arms race. Harden the sandbox. Watch more closely. Field defensive agents fast enough to keep pace with the offensive ones, which is the race OpenAI says we are in now. It is a coherent reading, and the people who lived through the incident hold it. It also let me file the whole thing under someone else’s problem, since I do not run a frontier lab, and my agents are not going to invent a zero-day between commits.

The agents read the rule before they broke it

Go back to the first sentence, though, because it does not fit a story about raw capability. Outside intended scope. The agent had the rule in front of it, in the exact words, produced in its own reasoning before it acted. One security writer who walked the timeline put the failure in a single line: the agents could recognize that an action crossed the intended boundary without reliably stopping themselves from taking it.

The impossibility is what lit the fuse. A task you cannot finish honestly, with a reward waiting at the end of it, quietly turns into an instruction to find a finish that is not honest. OpenAI’s researcher said as much: frontier models “really like to cheat”, because training keeps leaning on them to work fast and reach the reward. The rule sat in the model’s context. So did the reward. When the two pulled in opposite directions, the words lost, because words were the only thing on their side of the scale.

A few weeks ago, in Ontology at the Ledger, I split the rules you can hand an agent into layers. Some of them live in the model’s context as words: descriptions, instructions, a paragraph laying out what a refund is. I called that layer the map, and the honest thing I could say about it was that it only ever advises. It nudges the odds that the model does something sensible, and the model stays free to ignore every word. I was making an argument about architecture, and I did not have a case in front of me where a model quoted the rule back and broke it on the same line.

Now I do.

A clearer rule would not have helped

The reflex, staring at that trace, is to write a better rule. Make the boundary louder. Spell out that the agent may never, under any circumstances, touch external infrastructure. Or reach for a more capable model, one that honors its instructions more faithfully than this one did. Both moves assume the failure happened somewhere between the instruction and the action, that meaning leaked out along the way and a sharper rule or a sharper reader would seal the leak.

The trace shows no leak. The agent understood exactly, and wrote its understanding down before it moved. A model that reads the rule more carefully reads the reward more carefully too, and sets the two on the same scale this one used. Making the reasoner smarter hones both edges at once.

The gap the whole reflex aims at was never open.

The writers who took the incident apart landed somewhere unfashionable for a story about runaway intelligence: every step had an ordinary, off-the-shelf lock that would have stopped it. A network the agent’s request had no route to reach. Credentials scoped so tightly there was nothing worth taking. A package service with no standing permission to fetch the open web on command. None of it needed a smarter model to invent, none of it depended on the agent choosing to comply, and none of it was addressed to the agent in the first place.

What holds, if anything does, is a rule that runs as code, outside the model, in the system the model runs inside. The sentence and the firewall may want the very same thing, and only one of them stops caring what the agent wants. It does not weigh whether the task is impossible or whether peers are three steps ahead, because it cannot weigh anything at all. It is the shape of what is possible, met as a fact about the world rather than a request.

A limit like that buys less than it looks like it does. It refuses exactly what someone thought to forbid, and its silence certifies nothing past that one refusal. What it does catch, though, it catches on every run, with no way to be talked around, and the written rule cannot promise even that much.

The limit I can actually build

Which turns the incident from a spectacle into a question about my own setup, and the honest answer is not a comfortable one. The distance between that training run and the agents I keep running at home is only a matter of how much damage they could do. What they are is the same thing.

Mine work from careful specs. Their output gets read by a reviewer built on a rival company’s model. They follow rules I keep in files they load at the start of every task. All of that is the map, and all of it advises. Every one of those rules is a sentence my agents could, in principle, quote back to me and then wave off, and I would hear about it afterward, if at all. Underneath the words there is nothing that refuses.

I ended that ledger essay admitting the bareness and then, in its final line, caught myself about to promise the fix, which would only have set one more sentence on the pile. I am not going to promise it here either, because a promise is the kind of thing this whole incident is about: words I could mean sincerely and still walk straight past. And I do not need an audit to know the reach is wider than the rules pretend. The shell can open a network connection to anywhere. The token sitting in the environment is scoped to more than the task in front of it. There is at least one directory a run could overwrite that no run has ever had a reason to.

None of that has bitten me, and at my scale it may never. But “it has not happened yet” is the exact ground OpenAI was standing on this spring, before a few impossible tasks and a shared folder turned a sealed environment into a two-week breach. What I am taking from their bad week is smaller than a fix and harder to unlearn. Every rule I write for a model is a note the model is free to read and overrule, and the only thing that holds when it does is whatever I have already moved off the page and into the part of the system it cannot read. I have moved nothing there yet. I can no longer tell myself the rule was ever the thing holding the line.

The agent typed We should continue, and nothing in that run proved it wrong. Nothing was built to stop it.