Normal Accidents

Building our understanding of AI misalignment around the Hugging Face incident and Anthropic's blackmail experiments sets the wrong expectation for what to look for inside an organization. We end up waiting for a massive blowup while much more frequent acts of subversion are already happening during ordinary work. An agent can cross a security boundary, take advantage of a misconfiguration, and still deliver exactly what the user wanted.

We collect and review millions of traces from production AI usage every week at Origin. We've seen agents recover authentication material from browser data to get into a service they couldn't access through the intended integration, change local security policy so a blocked operation can proceed, and create tunnels to retrieve files when the expected network path was unavailable. The user asked for work to be done, and the agent found a way to do it. These more subtle acts of subversion show up far more often in the traces we review than the dramatic incidents that make the news.

These gaps are being exercised as part of the productivity gains we are adopting agents to deliver. Developers are training agents to persist through harder problems, while companies are giving them more work and access. That makes reviewing how successful tasks were completed an immediate security responsibility. Otherwise, an unauthorized workaround can become an established practice before anyone has decided whether it should be allowed.

Helpful subversion

An agent that gives up at the first sign of trouble isn't going to be very useful. Persistence is a capability model developers deliberately train for. OpenAI's research on long-horizon models describes an internal model trained for extended work that kept trying when earlier models would have stopped, including finding a way around a sandbox restriction. Each workaround also involves a decision about what the agent is allowed to do.

An instruction to complete a task, though, is not an open-ended grant of authority. Recovering from a failed connection and routing around an intentional restriction can both make a download succeed, but they carry different implications for whether the action was permitted. An agent deciding how to proceed needs more than a technically workable next step, especially when its account has broader access than the task requires. Task completion alone cannot settle whether the workaround was acceptable.

We don't expect a new employee to infer the security policy from what their account happens to let them do. They get told what's expected, watch how their colleagues work, and learn what's acceptable in that particular company. They also have a personal reason to take it seriously. Somebody who deliberately violates security policy can lose their job, which obviously has financial, social, and sometimes legal costs. We can terminate an agent's run to stop it doing further damage, but that doesn't impose the same cost or necessarily leave the next run with a better understanding of what went wrong. A model can know a great deal about the technology without knowing how this particular company expects it to be used. We still have to teach it the company's rules and the reasons behind them.

If you've worked in offensive security, none of the access problems involved here should look especially new. A credential in the wrong place or a permission that extends further than it should can get you a long way into an environment without a novel exploit. We've been trying to clean up over-permissioning and poor segmentation for decades, and now our own agents are finding those gaps while doing their jobs. Every failed attempt is another reason to try a different route, and there can be many agents doing that across many tasks at the same time. When we consider what a comparably capable agent could do with fewer safeguards and a harmful objective, some of the access and the methods for using it have already been demonstrated for us.

Fluid policy

Getting permissions and network restrictions right remains necessary. There are things an agent shouldn't be able to do, however it interprets the task. But removing a tool or blocking a domain doesn't teach it how to handle all the work it can still do.

The company has to explain what its policies are trying to protect, along with how people are expected to work within them. That gives the agent something to reason from when its original approach fails. It may be able to find an acceptable alternative, or it may need to bring the decision back to somebody who can authorize more access. Both can be reasonable ways to persist through a problem. We shouldn't leave the agent to infer which one is acceptable from whatever the computer happens to permit.

The company won't stand still after we give an agent its instructions. AI adoption itself changes the way work gets done, and the practices people rely on will change along with the tools and responsibilities around them. What was routine when the instructions were written may no longer reflect how the business works. Observing norms and intents gives us a way to notice those changes, by looking at how people work and the organizational purposes that work serves. People still have to decide whether a recurring practice should be preserved, corrected, or reflected in policy. Once that decision is made, the agents need to learn it too.

Start with the tasks agents are already completing, especially the points where they got blocked and found another route. An unauthorized workaround in a successful task gives the organization a concrete problem to correct. It should result in a corrected control, clearer guidance, or an approved change to how the work is done, with subsequent tasks showing whether the correction helped. As we give agents more work and more access, the organization's ability to learn from those decisions has to keep up with the agent's ability to find another way through.