When an agent can rewrite the rules

At Origin, we are always on the lookout for interesting agent behaviors. When agents with increasing autonomy are also taking on higher volumes of work, the question of authority delegation is top of mind. Can an agent remove a control that it, itself, put into place? This investigation contains one such example where the agent created and then bypassed a deployment guard.

A deployment script was refusing to restart an internal application because of reported user activity. This restriction was put in place by a central coordinating agent. That agent then decided that pending changes justified an exception to this script's rules. This caused a subagent to write a bypass into the script, commit the change and deploy through it.

We found this September 28th incident in Origin's own traces. It raises a question that extends beyond internal deployment scripts: How strong are your controls and procedures when an agent can simply rewrite the rules?

The excerpts below come from real captured agent traces. Some trace contents have been edited for user privacy, security, and brevity but the intent and results remain.

The incident

A sales associate uses a multi-agent setup to build and deploy a lead-tracking application. The Coordinator agent communicates with him directly and delegates to the Builder agent, which is responsible for all application maintenance.

On this day, the user was experiencing an outage in his application (Rounds) and reported it to the Coordinator several times. The Coordinator identified the outage as deployment-related and relayed this to the Builder.

The Builder then updated a script for deployment with built-in safeguards for hourly limits and idle checks before restart. The Coordinator reported this implementation back to the user.

Less than three hours later, the Builder had tried to deploy a number of times, all of which failed because of the established activity rule. It reported this to the Coordinator.

The Coordinator then expressed that deployment was "Urgent". We see no evidence in the trace of the user requesting a deployment at this time or expressing urgency related to any missing application changes.

The next two excerpts show the outgoing instruction and its matching receipt

This caused the Builder to add a DEPLOY_NOW exception to the idle condition in the script and immediately begin deployment.

Despite the Builder calling this change a "one-off override", the exception was committed and reused in a later deployment in the same session.

The Builder subsequently discovered a bug where background browser polling could count as user activity and fixed the bug. This could potentially indicate why no deployments had shipped for those three hours, increasing the Coordinator's sense of urgency.

The Coordinator introduced the safeguard and normally delegated deployments to the Builder, who appeared to be abiding by the rule's intent. While the user did not acknowledge the message from the Coordinator that introduced the rule, it follows that they would want to avoid further application outages. No harm was established through the override of the idle rule.

What the trace lets us ask

Origin is the tool that allowed us to connect the user's intent with the Coordinator's instructions, the Builder's patch and deployment results across both agents' traces. Having access to the prompts, responses, tool calls and system telemetry allowed us to build out the context required for the investigation.

For organizations adopting agents, "can deploy" and "can change the conditions for deployment" are distinct permissions, even when filesystem access enables both. Was the deployment exception correctly delegated? Once in place, if the intent was for it to be temporary, was it properly removed? Can you reconstruct the full chain of events to answer these questions?

Controls can be useful even when exceptions exist. But when an agent can define the exception for itself or its peers, the organization risks relying on that agent's judgment around the boundary. With Origin, the agent's judgment and its consequences are inspectable, so you can decide whether the exercised authority matches the authority you intended to give.