S08-observability-replay — A shift with a memory

The hook

A customer calls back. "You charged me twice yesterday." You open a terminal and scroll: the shift printed a perfectly nice transcript, and it is gone. You cannot show what the model actually saw, you cannot re-run the shift, and you cannot prove which turn produced the second ticket. The debrief becomes your memory against theirs, and memory loses.

The promise

By the end of this session you will have recorded your own live shift to a JSONL trace, replayed it with zero model calls, and printed a transcript that is content-identical to the live run. Then you will do the part that separates a recording from a replay: prove determinism by injecting the two things that quietly break it — the wall clock and the RNG.

The theory in depth

A transcript is a printout; a trace is a structure

run_shift returns a run record: the message list, the consent decisions, the stop reason, the counters. That tells you what the conversation said, not where the time and tokens went. Tracer builds a tree instead. Each span carries a name, a kind, attributes, children, a start time and a rounded duration. A generation is a span that also carries a model call:

This session's shift is that shape with café names: record_shift opens one shift span and wraps the client in TracingClient, which opens one generation per chat call. Into that generation go a deep copy of the input messages (the loop keeps mutating the list after the call, so the copy is not a nicety), the output content, the names of any tool calls, and the usage block the endpoint returned. usage_of then sums tokens and durations over the generations — which is how "which turn burned the tokens?" becomes arithmetic instead of an argument. NullTracer answers the same interface while doing nothing, so instrumentation never costs you an if tracer: branch.

A recording is a contract, not a printout

RecordingClient wraps any client and appends one JSONL line per call — {"request": ..., "response": ...}, in order. It never constructs a client of its own; it takes whatever get_client() returned. ReplayClient serves those responses back and refuses to guess:

Both recording and replay use S05's protected executor. The synthetic customer explicitly approves the first valid proposal; an exhausted decision queue rejects. Each replay needs a fresh responder with the same decisions. Tracing never substitutes for consent.

Two invariants, and they catch two different regressions. Matching polices the calls that arrive: the next recorded request must equal the incoming one exactly, otherwise ReplayMismatch raises instead of serving a response recorded for something else. Exhaustion polices the calls that should have arrived: a regression that deletes the last model call sends nothing bad to match, so only assert_exhausted() — "every recorded call was consumed" — sees it. One client, two ways to be wrong.

Telemetry degrades; the shift does not

Tracer.export() propagates whatever the exporter raises. That is deliberate: nobody should be able to pretend a dead backend was fine. export_fail_soft is the boundary where you decide what the failure costs — the traces, or the shift. It returns the error text when the exporter raised and None when the trace shipped. A tracing backend is a network call you do not own; it fails on a schedule you do not choose.

A replay cassette contains messages and tool payloads. Keep real recordings private; committed course cassettes contain only synthetic café conversations. Exported telemetry has a different purpose. A conceptual metadata-only projection might keep {operation: "fire_ticket", status: "blocked", duration_ms: 4} and omit customer text, ticket arguments, results and exception bodies. This is a design example, not an export mode implemented by Tracer; even metadata needs review for identifying labels. The OTel GenAI conventions remain in Development, so pin the adopted convention version instead of treating current field names as a stable contract.

Replay also differs from retrying a live side effect. Replaying a synthetic shift locally does not prove that a kitchen accepted an order only once after a lost reply. Carry S05's approved-payload/idempotency distinction into any durable design; this notebook does not supply one.

Determinism is wired in, not hoped for

Here is the hunt the notebook stages. A teammate ships a "harmless" PR: a timestamped header and a livelier closing line for the end-of-shift note. Now two replays of the same recording disagree. That is impossible for the model — the recorded responses are frozen JSON on disk, so replay cannot add variance. The nondeterminism is in the host, and the diff names two suspects: a wall clock read and an unseeded RNG draw.

The fix is not to delete the timestamp and the flourish; it is to make them inputs. Pass a clock and an RNG into the note builder, and create a fresh seeded instance per run. TickClock is the same idea for the tracer itself: a deterministic stand-in for time.monotonic, which is what makes two rendered trees comparable at all. One shared Random(7) reproduces across processes but drifts between two runs inside one process — the trap is subtler than it looks.

Nothing in the notebook writes anywhere it was not handed: every path lives inside a tempfile.TemporaryDirectory(), and a test cell walks the repo root before and after to prove that not one .jsonl landed in it.

Build (in the notebook, predict first)

Open toy.py. Same endpoint configuration as S06, plus the SCRIPT of two customer lines the notebook replays.

1. Record your own shift. record_shift(client, SCRIPT, path, responder=make_responder([("approve", None)]), tracer=Tracer(clock=TickClock())) runs the real loop through a recording client and prints the rendered tree, the tools that ran and the usage totals.
2. Predict the replay. Before running it: how many times will your live model fire during a replay? Can the replayed transcript differ from the recorded one, given the responses are frozen on disk? Write both answers down.
3. Replay twice. Two replays of one recording, compared with transcript() — a canonical rendering that contains nothing but the conversation, so a difference would be a real divergence and not a formatting artefact. A second test cell replays a stale script and expects ReplayMismatch.
4. Kill the exporter. Tracer(exporter=dead_exporter) raises on export; export_fail_soft returns the error and the shift still reports its stop reason. Read that boundary before you trust it.
5. The hunt. Compare the teammate's note across two replays, then write attempt_handover_note(run, *, clock, rng) yourself, where every changing value must come from clock or rng. The reference is behind a reveal switch — flip it after you attempt.

Checkpoint — the number you bank

Two numbers, and both are exact rather than statistical:

• model calls during replay: 0. The replay is offline by construction — the notebook counts your live client's calls before and after and expects the same total.
• live transcript == replay transcript. Content-identical, turn for turn, because the responses are frozen on disk and everything else — clock, RNG, system prompt, tool schemas — is now an input.

Write both down with the model name. They are the S08 contract, and S09 spends it: a debrief that cites turns of a trace you can hand to someone else, who can replay it.

State of the art (source review: 20 September 2026)

Annotated readings

• OpenTelemetry, traces. Extract: span, parent, attribute, and the difference between a trace and a log line. Then map those onto Tracer.nodes() and Tracer.render().
• OpenTelemetry, GenAI semantic conventions. Extract: the attribute names for a model call, and where usage lands. Rename your generation attributes to match and your trace becomes readable by tools you did not write.
• VCR.py README. Extract: what a cassette stores, and its match-on-request behaviour — the same "matching polices the calls that arrive" invariant, with the same blind spot that only an exhaustion check closes.
• OpenLLMetry repository. Extract: which attributes it emits per call. It is a free answer key for the field list you are about to invent.
• Your own JSONL file. Extract: the request at line 1 and the response at line 1. Everything the replay guarantees rests on those two objects being complete.

Misconceptions and failure modes

• "The transcript is the trace." A transcript is a rendering of one artefact. A trace says which phase called what, for how long, with which tokens — which is the question you actually have.
• "Replay means calling the model again with the same prompt." That is a re-run, and it measures variance. Replay serves recorded responses and makes zero calls; that is why it can be compared at all.
• "A recording that matches is a replay that is correct." Matching only polices the calls that arrived. Deleting the last call passes every match; only assert_exhausted() catches it.
• "Telemetry should never break the shift, so swallow export errors everywhere." Swallow at the boundary you chose, and return the error so somebody can see it. Errors hidden inside a tracer are worse than no tracer.
• "The model injected the nondeterminism." Frozen responses cannot vary. Look for wall clocks, process-wide RNG, unordered iteration and cached state — the host is the usual suspect.
• "Just delete the timestamp." Then you lose the information. Make it an input and the same line becomes reproducible and present.

Self-check

What this unlocks

You have a trace and a replay. What you do not have is anything a busy owner would read. S09 — Evidence reports turns this JSONL into a shift debrief with turn citations, and builds the two validators that catch the two ways a report lies: the tidy paraphrase presented as a quote, and the omission that leaves no false sentence behind.