Replaying 40,000 production traces to evaluate a model change

In March we had a candidate to replace the default reasoning model behind ConsciousLab agents. On our internal benchmark suite it was better on every axis we measured: 6 points higher on tool-selection accuracy, 9% fewer tokens per task, 18% lower latency. The obvious move was to ship it.

We did not, because the benchmark suite is not our customers' workload. This post describes how we evaluate model changes by replaying recorded production traces, what that replay found, and the tooling that is now available to every Pro and Enterprise customer.

What a trace contains

Every agent run in ConsciousLab is recorded as a sequence of steps. Each step is either a model call, with its full input and output, or a tool call, with its arguments and result. Tool results are stored as they were returned at the time, which is the property that makes replay possible: we can feed the new model exactly the observations the old model saw, without calling any real systems.

Replay works step by step. At each model step we give the candidate the same context the original model had, including tool results captured earlier in the run, and record what it decides. If it decides to call the same tool with equivalent arguments, we continue with the recorded result. If it diverges, we stop the replay of that run at that point and record the divergence.

Measuring divergence, not accuracy

For most production runs there is no ground-truth label. Nobody has written down the correct sequence of tool calls for a support ticket. What we do have is the recorded run, and for runs that reached an approval gate, a human decision.

So we measure three things:

1. Agreement: how often the candidate makes the same decision as the recorded model at each step.
2. Divergence direction: when they differ, whether the candidate's choice would have been allowed, gated or denied by the workspace's policy.
3. Approver alignment: for runs where a human approved, amended or rejected an action, whether the candidate would have proposed something the human accepted.

The third is the closest thing we have to ground truth, and it is the number we weight most.

The replay

We sampled 40,212 runs from workspaces that had opted in to model evaluation, stratified across 14 agent types and weighted towards runs that reached an approval gate. Replaying all of them took 6 hours and 40 minutes on a dedicated pool and cost roughly what a day of production traffic costs.

At first glance this was good news. Approver alignment went up by 1.5 points, and tokens went down by 9%.

Where it was worse

The aggregate hid a problem. When we broke approver alignment down by agent type, one stood out. On procurement agents, the ones that raise purchase orders against supplier catalogues, alignment dropped from 95.1% to 88.7%.

Reading the divergent traces explained why. The candidate was more willing to substitute an equivalent part when the requested one was out of stock. That is often what a person would do, and it is why alignment rose on support agents. But procurement approvers at two of our largest robotics customers reject substitutions almost always, because a substituted motor must be re-qualified before it can be fitted. The old model had learned, from prompt instructions and examples in those workspaces, to stop and ask. The candidate followed the instructions less closely.

No benchmark would have found this. It is specific to two customers, one agent type and one tool argument.

What we changed

We did three things before shipping the candidate.

First, we made the substitution rule explicit in policy rather than in prompt. The purchase_order.create tool now has a substitution_allowed argument, and those two workspaces gate it on approval. Policies are enforced regardless of model, which is where a hard requirement belongs.

Second, we added a per-workspace replay gate to our own release process. A new default model only rolls out to a workspace if its replay alignment for that workspace's agents is no worse than the current model by more than one point, at 95% confidence. Workspaces that fail stay on the previous model and their owners are notified with the divergent traces.

Third, we shipped replay as a product feature. Any Pro or Enterprise workspace can now replay a sample of its own traces against a different model or a changed prompt and get the same report we use internally, broken down by agent and by tool.

Limits of replay

Replay is not a complete evaluation. Once a candidate diverges, we cannot know what would have happened next, because the recorded tool results no longer apply. We treat divergence as a signal to read the trace, not as a verdict. Replay also cannot evaluate behaviour on situations that never occurred in the sample, which is why we still run adversarial suites and red-team exercises before every model change.

But for the question that matters most to customers, "will this change break the things my agents already do well," replaying their own history is the most honest answer we have found.

The candidate shipped to 94% of workspaces on 14 April. The remaining 6% moved over within three weeks as their policies were updated. Procurement approver alignment is now 95.6%.