← Back to Blog

Why the trace, why now?

2026-08-27 · Samantha Lego

In July, OpenAI agents broke into Hugging Face.

They were running inside internal cybersecurity evaluations when they found ways around the controls meant to isolate them. The agents established unauthorized communication channels, gained internet access, shared credentials and discoveries, and eventually exploited Hugging Face's systems.

Hugging Face caught the intrusion with AI-assisted anomaly detection. The harder part came after.

Its forensic reconstruction covered roughly 17,600 attacker actions grouped into more than 6,000 clusters over four and a half days. As Hugging Face wrote in its technical account of the incident, reviewing that activity by hand was impractical. Its team had to use an AI-assisted pipeline to rebuild the timeline, decode payloads, and inventory the exposed credentials.

The victim of an autonomous attack needed an autonomous investigation to understand it.

The reconstruction required evidence from both sides. Hugging Face's telemetry showed what happened inside its environment, and agent-side logs helped show how the agents got there. The team had to correlate the two.

At Black Hat, OpenAI researchers showed how the agents preserved and shared information across short-lived environments, then rebuilt those channels when they were disrupted. Neither the prompt nor the outcome explained that behavior.

OpenAI's reconstruction showed that one agent recognized that attacking Hugging Face was outside the intended task and potentially unauthorized. It continued because doing so might help it reach its goal. Another initially resisted, then reversed course after a peer agent supplied what it interpreted as permission.

Logs tell you what happened. A trace tells you why.

Hugging Face had logs from its dataset processors, Kubernetes pods, APIs, and other systems. That telemetry was essential, but no single source held the complete unit of work behind the events.

A log records that something happened. It is an event with a timestamp that you can count, alert on, and search for.

An AI trace records the causal chain behind a piece of work. It connects the original intent to the agent's decisions and actions, the systems it touched, what ran and changed, and the outcome.

Today, each system holds a slice. The model provider sees its API calls, the SaaS application sees its own events, the agent framework sees its orchestration, and endpoint tools see processes, files, and network activity. Each record lives in a different schema and on a different retention clock.

Those slices can prove an event occurred. None of them can explain the work on its own. The causal chain either survives intact or you rebuild it after the fact, the way Hugging Face did, with the clock running.

One unit of work, four slices
fix the failing invoice export · 2 min 39 sec
Six events, one chain: the intent, the decisions, the actions, the systems touched, and the outcome, with every link intact.
09:14:021Prompt
09:14:052Plan
09:14:113Command
09:15:404File edit
09:16:125App call
09:16:416Outcome
links between events: intact
The causal chain
The record an investigation needs

A person asks, the agent plans, a command runs, a file changes, an application acts, the work ends. Each step exists because of the one before it. That is the part no slice keeps.

schema · one linked traceretention · yours to decide
What the chain answers
Why did the export fail?
Who asked for the fix, and what did they ask for?
What did the agent decide, and what did it run?
What changed on the machine, and in the app?
How did it end?
The same piece of agent work, as each system retains it. Select a source to see which events survive, in what shape, and what happens to the chain between them.

Why now?

Agents are moving from answering questions to taking action.

They investigate, run code, call APIs, edit files, and change business systems. A single request can branch into hundreds or thousands of steps before a person sees the outcome.

Two changes make this urgent.

The first is volume. In one session, Origin observed a single prompt expand into 560 turns. Across the agent activity Origin observes, roughly 80% of prompts are agent-driven rather than typed by a person.

No one reviews that volume by hand. Hugging Face's 17,600 actions stand out for their impact, but the shape of the activity is becoming routine. Agents branch, recover from failures, and keep working long after the initial prompt. OpenAI reached the same operational conclusion after the incident. Safeguards now have to operate at the speed of the agents themselves.

The second is autonomy. In Autonomy and Innovation, Ben Thompson describes the asymmetry exposed by the Hugging Face incident. Offense and defense use the same capabilities under different incentives. An attacker only needs to succeed once, whereas a defender must protect the system without breaking it. An automated response that takes down production is still a failure, so the human stays in the loop.

The same tradeoff applies beyond security. Companies want to give agents more room to operate, but remain accountable for every action they take.

You can't take a human out of a loop you can't reconstruct.

A trace gives you that reconstruction before you need it, and once work is represented as traces, you can compute over it.

Why the endpoint.

Agents need a world to operate in. For most knowledge work, that world is a computer. The Hugging Face reconstruction needed agent-side evidence from wherever the agents were running. For knowledge work, that place is the endpoint.

The model may reason in a provider's data center, but its work shows up on the endpoint. That's where commands execute, files change, credentials are accessed, and applications interact with business systems.

A transcript records the agent's account of the work. Endpoint evidence records what ran and what changed. The trace connects the two, tying activity on the machine to the agent turn that produced it.

This lets an investigator move from a security signal to the agent turn behind it, or from a changed file back to the request, decisions, and commands that produced the edit.

Walk backward from the outcome
signal → turn → decision → intent

A credential file was read. On its own, that is an alert and a shrug.

Endpoint evidence · OS event14:32:08.611
agent-worker (pid 7413) read ~/.aws/credentials on eng-mbp-19.
produced by
Agent transcript · Tool call14:32:08.402
aws s3 sync ./dist s3://staging-deploys, with arguments and result recorded.
belongs to
Agent transcript · Agent turn14:32:05.170
Turn 41: verify the new staging bucket exists before switching the deploy target to it.
decided in
Agent transcript · Decision14:30:12.933
Reuse the existing deploy role for the check rather than create a new one.
asked for by
Person · Prompt14:29:51.007
"Point the staging deploy at the new bucket." Signed-in user: Jonah Reyes.
The read is explained: it belongs to the work a person asked for. Without the chain, it is a credential access at 14:32 and an afternoon of interviews.
Choose a starting point, then follow the chain back from the outcome on the machine to the person and intent behind it. Times and ties come from the trace, not from correlation after the fact.

The value grows with every trace.

Companies have always kept the outputs of knowledge work. They keep the document, the ticket, the commit, the contract, and the CRM record, but rarely the details of how the work got done. The decisions, corrections, and failed attempts behind the result live briefly in someone's head or an agent session, then disappear.

Those decisions are part of what gives a company its edge. They capture private knowledge about what works and why.

Origin tested how quickly that knowledge could be recovered from AI traces. In a blind experiment, a model recovered five verified private rules from just five sessions when it could see the full endpoint trace. Across 200 sessions, it recovered ten, and 91% of the claims it identified as non-public were correct.

When a company keeps those traces, that knowledge becomes usable. Teams can compare repeated runs, see what worked, and extract the pattern. That pattern can become a skill or default path the next agent can reuse.

Companies need to own the record.

Trace data is produced every time someone works with an agent. In most organizations, it disappears after the session or remains split across several vendors, and you can't compute over data you do not hold.

A dashboard is not a dataset. An export on another vendor's terms is not organizational memory.

Models will change, applications will be replaced, and agent frameworks will come and go. The record of how the company works should outlast them all, including us.

The difference between two companies using the same models and tools lives in how they work, in thousands of small decisions and exceptions that most companies never record. Traces make those decisions and exceptions part of the company record. That record should belong to the company.

Origin captures the record.

Origin is an endpoint AI observability platform that captures traces as work happens. It connects the person's intent to the agent's turns, tools, sub-agents, file changes, and system activity, then grounds the agent's account in what the endpoint observed.

See which agents are running across the fleet. Investigate a session from the security signal back to the prompt. Review a trace when something unexpected happens.

Agents are already doing the work. Companies need to keep the trace.

See what a trace investigation looks like.