Hermetic replay

This is the seam that makes “the recorded run is the test” real: the same agent code runs live in production and deterministically offline in CI — no API keys, no cost, no flakiness.

The idea

Wrap each external call in call_tool / call_llm instead of calling it directly:

import tracely_sdk as tracely
 
def run(user_input: str) -> str:
    with tracely.agent("support-agent"):
        order = tracely.call_tool("get_order", lambda: get_order(order_id),
                                  args={"order_id": order_id})
        answer = tracely.call_llm("gpt-4o", lambda: chat(messages),
                                  input=messages, usage=(812, 96))
        return answer

• In production (no fixtures active): call_tool / call_llm invoke your fn, record its output (and any error) on the span, and return it — a normal traced run.
• In CI replay: tracely replay loads the regression case’s recorded fixture bundle and activates it; call_tool / call_llm then serve the recorded outputs (in order, or matched by args) and never call your fn.

No rewrite needed: @observe tools and auto-instrumented LLM calls replay too

The call_tool/call_llm seam is the fully-portable path, but most agents don’t need a rewrite:

• Tools — a function decorated @tracely.observe(as_type="tool") is transparently hermetic: under fixtures it serves the recorded output (or re-raises the recorded failure as tracely.ToolError) instead of running. Decorate your tools and the tool side replays as-is.
• LLM calls — inside a fixtures() block Tracely class-patches OpenAI chat.completions and Anthropic messages, so code that calls the SDK directly (under instrument="auto", or via the wrap_openai / wrap_anthropic drop-ins) is served the recorded completion — reconstructed into a provider-shaped response object — and never hits the network. Other providers’ direct calls fall back to live in replay; use call_llm there.

Activating fixtures manually

tracely replay does this for you, but the primitive is public:

with tracely.fixtures(bundle):     # bundle = the case's recorded tool/LLM outputs
    run(case_input)                # call_tool/call_llm now serve recorded values
# outside the block, calls are live again

fixture(kind, name) peeks the next recorded output for a tool/model without consuming it.

Why this matters

• Deterministic — the agent sees the exact tool/LLM outputs the production trace saw.
• Offline & free — no live keys or spend in CI; lambda: … is never called under fixtures.
• Faithful — recorded errors replay as ToolError, so error-handling logic is gated, not bypassed.

The reference agent in sdk/examples/weather_agent.py wires this up (run, run_broken, run_handles, …) and is what tracely replay --entrypoint weather_agent:run executes.

Replay is single-turn and hermetic: it re-runs promoted regression cases through your own code against recorded fixtures. To gate multi-turn behaviour against a live service — in any language, with no agent code in CI — see Scenarios. Most teams run both.

Next: the CI gate CLI.

Comparing tools? Langfuse alternatives, compared honestly covers how hermetic replay differs from dataset-and-live-calls CI, including where the other approach wins.

Comparing tools? Langfuse alternatives, compared honestly covers how hermetic replay differs from dataset-and-live-calls CI, including where the other approach wins.