Regression cases

A case is a production trace that must never recur. You do not write it; you promote it — from a failing trace, or from a failure cluster — and from then on every CI gate for that agent replays it.

What promoting captures

1. The input — the user request exactly as it arrived, plus a digest of it so the same input is never promoted twice and a CI run can be matched to its case without any wiring.
2. A fixture bundle — every tool call and model call the run made, in order, with arguments, outputs and error status, stored in object storage. This is what makes hermetic replay possible: in CI the agent’s code runs for real, but its tools and model calls are served from the recording.
3. The reference trajectory — the ordered list of steps the failing run took.
4. A fail-to-pass contract — the assertions a candidate run must satisfy: no run error; the tools the agent was asked for and must actually execute; whether the tool sequence must match; whether a tool error is tolerated when the agent handles it gracefully.

Then the contract is validated against the source trace: the failing run must fail its own case. A case the original failure would pass is not a regression test, and Tracely refuses to create it.

Statuses

A case

• Assertions — the contract, one line each, with the last verdict.
• Reference trajectory — the source run’s steps.
• Replay — paste the trace id of a candidate run (a fixed version of the agent, run in staging or CI) and Tracely evaluates it against the contract right here: FAIL while the bug is present, PASS once it is fixed. Every replay is kept as history, so a case carries its own red → green story.
• Source trace — the production run it came from, one click away.