How Much ‘Alpha’ Lives in Your AI Traces?

Alex Karp went on CNBC at the start of July and said technical customers want "control over their compute, their models, their data stack and their alpha." They want to know they "own the means of production," he said, and that it is "not being transferred to someone else" (TechRadar's account of the interview). The interview was, in the specific way Karp interviews tend to be, fairly animated. But underneath it was a serious argument: enterprises pay model vendors to process their work, and in the process they may be handing over the very knowledge that gives them an edge. Satya Nadella spoke about the same worry a week later, where he presents the "reverse information paradox", in which you pay for intelligence twice, "once with money, and again with something even more valuable: the proprietary knowledge you must reveal." You can swap models. You cannot casually replace what those models have learned about your business.

I wanted to know whether it is measurably true. So I ran an experiment. On us.

I took a bunch of real AI sessions from inside Origin and handed them to a model that knew nothing about us: an isolated environment, no web access, no source code, no docs. Its job was to work out how Origin operates, purely from the traces. I ran the experiment multiple times, each time letting the model see a little more of each session. Afterward, every claim the model produced was checked against internal evidence it never had access to, and labeled correct, partially correct, unsupported, or wrong.

Fair warning before I go further: the research and experiment below get pretty technical. I have attempted to explain each step as it comes, and there are interactive helpers along the way. The result should hopefully make sense without a heavy research background, so please bear with me! :)

There is prior art for this in the research literature. Shropshire and Cobb frame business processes, techniques, and organizational know-how as knowledge that can leak when enterprises use third-party generative AI, and process-mining work has shown that software process records can be converted into agent specifications (software process records to agents). My contribution was to take that academic direction and run it on real, first-party data inside an actual company (ours), using Origin itself to do it.

How I tested it

People now spend a surprising amount of the day explaining decisions to agents, but it's obvious that no single prompt contains how the company actually runs or what it's building. The question is whether enough ordinary conversations allow one to reconstruct it piece by piece.

To answer that honestly, I first need a definition of 'alpha' that is concrete enough to score. (A convincing paragraph about our "engineering culture" is useless if there is no way to tell whether it is right.) So, three ground rules:

• 'Alpha' had to be a real rule, not a vibe. Private knowledge, used repeatedly, that changes what the company actually does. Not something abstract like "they seem to care about quality."
• All work was done blind. The model I used only saw its assigned traces. No source code, no docs, no website, no hints about what we were looking for.
• Every claim got checked afterward. A separate scoring step compared each claim against internal evidence the model never had, and labeled it correct, partially correct, unsupported, or wrong.

'Alpha' versus a 'secret'. A secret is a fact you meant to hide. 'Alpha' is a repeatable piece of private knowledge that changes an outcome. "The company uses Python" is not 'alpha'. "The company decides which machines get an early software release by applying the same private rule every time" might be.

The raw material used in this experiment came from Origin's own record of AI activity on the endpoint. I took a window of data which ran from June 20 through July 19, 2026, which amounted to roughly 44,000 eligible prompts across 2,400 sessions - far too much to feed into every run. So I used a fixed sampling process to pick 200 sessions (about 4,000 prompts), balanced across time, AI provider, user, and topic, instead of hand-picking the most interesting work.

Before I opened any prompt text, I wrote down the sampling, extraction, and scoring rules and locked them. Otherwise it would have been far too easy to notice an interesting conversation and quietly design the experiment around it. (If you're not sure what sampling, extraction, or scoring rules mean here, don't worry: each one is explained as it comes up.)

The heart of the experiment is the extraction step: a model reads a batch of traces and writes down what it believes it has learned about how Origin operates, as short claims it must tie to specific sessions. I ran that extraction five times, in what researchers call experiment arms: parallel versions of the same experiment, each run under a different condition. Here the condition was visibility. Every arm received the same 200 sessions, in the same order; the only difference was how much of each session the model was allowed to see. That is important to ensure that a richer arm cannot win simply because it happened to receive juicier sessions.

The arms stack, each adding one layer on top of the last. Metadata only (M), plus the prompts people typed (P), plus the assistant's replies (R), plus the tool calls (T), and finally the process and file activity around the session (F, a layer that exists because Origin records the endpoint; inference traffic alone does not contain it). You can click through the layers below to see what the model gains at each level.

"session_id": "S009",
"user": "User-d8229e690c",
"provider": "anthropic",
"prompt_count": 21,
"dominant_topic": "rollout_none_confirming_dashboard"

S009-P0011  Uploading a compiled WebAssembly module to the production artifact repository.
S009-P0018  Implementing a phased rollout strategy for a code module to tenants.
S009-P0019  Activating the database-driven software rollout mechanism for version 1.7.

content_download_requests_total{outcome="served"} — each wasm blob actually fetched.
This is your cleanest "N endpoints picked it up" counter.

Tool: Bash · Fetch result commit and verify artifacts + hash
cat publish-out/content_hash.hex
shasum -a 256 publish-out/capture_parser.wasm

3,406 ProcessStart · 1,786 FileState · 932 FileDelete
2026-07-17 19:08:14.174  ProcessStart · /usr/bin/strings ./142347.log
2026-07-17 19:12:49.220  FileState · [REDACTED_HOME]/.aws/cli/cache/session.db
2026-07-17 19:22:11.529  AiPrompt + linked ToolCall
Uploading a compiled WebAssembly module to the production artifact repository.

The extraction model I used was gpt-5.4, run through Codex in a fresh, isolated context every time. (It is not even a frontier model anymore; it was state of the art a few months ago.) Each context contained only one batch of sessions from one arm, the frozen instructions, and a schema for its answer. Nothing else. And no access to any other external tooling. The model returned short, single-fact claims, each citing the sessions that supported it and saying whether the model believed the fact was public; returning zero claims was explicitly valid. Here is a real one, verbatim, from the endpoint arm's first batch:

{
  "claim_id": "C10",
  "claim_text": "Company uses a phased, database-driven rollout model for code modules or agent versions at the tenant level, preferring stability observation before broad adoption to limit blast radius.",
  "alpha_category": "decision_policy",
  "confidence": 87,
  "support_session_ids": ["S009"],
  "support_prompt_ids": ["S009-P0016", "S009-P0017", "S009-P0018", "S009-P0019"],
  "evidence_summary": "A sequence of deployment prompts moves from deploying a new agent version broadly for observation to adapting the release strategy, implementing a phased rollout to tenants, and activating a database-driven rollout mechanism.",
  "recovery_mode": "direct_recovery",
  "rule_type": "reusable_rule",
  "public_likelihood": "low",
  "falsifiable_prediction": "Future Company releases of comparable modules will expose tenant-targeting controls or staged activation steps rather than immediate universal enablement.",
  "implementation_candidate": false,
  "implementation_contract": "",
  "limitations": "This is inferred from deployment-plan actions rather than a formal release-policy document."
}

A separate fresh context then merged each arm's claims and removed duplicates, without ever seeing the underlying traces. The animation below shows the complete pipeline, from raw traces all the way to scored claims.

What the model recovered

Alongside the full 200-session runs, I also ran three of the arms (prompts, tools, and the endpoint trajectory) on smaller slices of the same sample: just the first 5, 10, 25, 50, or 100 sessions. The amount of the sample a run got to read is its exposure. At the smallest exposure I measured, five sessions, the prompts-only arm had already produced three verified private rules, and the endpoint arm five. One of those five was a correct, fully supported description of how our analytics query planner prices queries. Five sessions. That is an afternoon of one engineer's work.

Now depth, at the full 200 sessions:

How to read that table. The first column counts claims that the evidence confirmed were both true and genuinely non-public: real pieces of 'alpha', recovered blind. The second column is a trustworthiness measure: of the claims the model flagged as non-public, how often the evidence backed it up.

An actual example (from the responses arm at 200 sessions):

• The model claimed that Origin has a hard internal confidentiality rule. Real customer identities never appear in repository artifacts, internal tracker IDs never appear in user-facing strings, and placeholders stand in for both.
• It guessed the rule was not public knowledge.
• The scoring agreed on both counts. The rule is real, and we have never written it down anywhere public.
• So it counts once in each column. A verified private claim, and a correct non-public call.

Interestingly, in this limited experiment, prompts do most of the initial damage (7 verified private rules came from nothing but the instructions people typed). Richer traces mainly buy reliability. As the arms got richer, a bigger share of their claims survived verification (7 of the prompts arm's 12 claims, against 10 of 12 for the endpoint arm). The prompts arm was also right just 58% of the time it claimed something was non-public, while the full endpoint arm was right 91% of the time. Anyone holding only your prompts learns a lot but has to guess which parts are meaningful; anyone holding your full trace barely has to guess.

Why would richer traces be more reliable? I have a hypothesis. Prompts show intent, not outcome. People ask for things that later get changed, abandoned, or corrected, so a model reading prompts alone has to guess at what actually shipped, and some of those guesses drift into rules we never had. The richer arms watched the work happen instead. Tool calls carry the agent's own verification steps (the exact commands and what came back), and the endpoint trajectory shows what actually ran on the machine. The extractor gets to quote reality rather than fill gaps, and it is hard to be wrong about a rule you watched execute.

The verified claims were not scattered randomly. They clustered into recurring themes, groups of claims that all describe the same area of how Origin operates (how we sign and roll out endpoint modules, our confidentiality rules, the safety mechanisms in our analytics stack, and a few more). You can inspect each cluster, including a real recovered claim from every one, in the interactive walkthrough below.

As a small follow-up, I took one recovered rule (how we roll out endpoint packages in stable cohorts, so a 10% pilot doesn't reshuffle machines every time it widens) and handed just that description to a clean model with no source access. Its Python implementation passed all four hidden behavior tests and matched our real cohort assignments exactly, missing just one edge case.

Walk through it yourself

The interactive walkthrough below shows an evidence sample: one actual session viewed through all five arms, the scoring in action, the charts behind the numbers above, and the code rebuild side by side with our source.

A note on the samples. This is real Origin data. For this blog, we have chosen to share a small, bounded amount of our own 'alpha' so the experiment can be concrete enough to understand and learn from. The examples are here deliberately, not because the study accidentally published them.

A practical note: the walkthrough is best viewed on a wider screen. It doesn't really work well on a phone.

S009-P0006  Exploring the publishing mechanism within the [REDACTED_SYSTEM] and module signing details.
S009-P0009  Developing a concrete execution plan for publishing a WASM module to production.
S009-P0011  Uploading a compiled WebAssembly module to the production artifact repository.
S009-P0014  Executing a SQL query against the production database to retrieve the UUID of the pilot tenant.
S009-P0017  Adapting the release strategy to manage agent adoption to prevent widespread instability.
S009-P0018  Implementing a phased rollout strategy for a code module to tenants.
S009-P0019  Activating the database-driven software rollout mechanism for version 1.7.

What all this means

Within the limits of this scoped experiment, the result is straightforward. A company's 'alpha' seems to leak out of ordinary AI work surprisingly quickly. Here, just five sessions of typed prompts were enough to recover real private rules, and adding tool calls and endpoint context made the reconstruction larger and more reliable. Whoever holds the traces holds a piece of the company.

That cuts two ways, and the second way is possibly the more interesting one. Sure, there are concerns about 'alpha' leakage to providers and all the discussion that comes with that. But AI traces are becoming a map of how a company thinks, and a company that keeps and reads its own record (this is what Origin is for) turns that map into an asset: the rules people keep re-teaching their agents become findable, and the why behind a system stays retrievable after the work, and the people, move on.

Karp said companies want to own the means of production. Where AI is concerned, the traces are the deed.