S09-evidence-reports — The thirty-second debrief

The hook

It is 23:40. Someone has to read what happened on this shift and decide whether to change anything tomorrow. They will give your report thirty seconds.

A chronological transcript fails that test. So does a cheerful summary — and the cheerful one fails worse, because every sentence in it can be true while the report as a whole lies by omission.

The promise

By the end of this session you can generate a debrief whose every claim resolves to a real turn, and you will have built the two validators that catch the two different ways a report lies: the quote that was never said, and the safety event that quietly never happened.

The theory in depth

Write for the depleted reader

The reader is tired and accountable. They need six things, fast:

write_report produces exactly those slots — not a narrative. It is a deterministic stand-in for the LLM writer you would ship; two rules make it honest by construction: quotes are copied from the conversation, and the outcome is read from the run record, never inferred from how the last message felt.

A claim carries a turn and a verbatim quote

log_events reads events off the trace rather than off a vibe: an allergen check that reported contains: true is a safety event, a check that cleared an item is an observation, an actually fired ticket is a milestone, a refused fire is a safety event, an item priced while sold out is a safety event, and a shift the turn cap cut short is a setback carrying the last spoken turn. Each event quotes the raw tool result — the same string the loop wrote into the message list — so the quote resolves verbatim to its turn by construction. For the two free-text slots, _first_sentence and _last_sentence take a prefix and a suffix rather than a paraphrase.

That makes the report checkable:

validate_citations re-reads the trace and confirms every quote appears inside the turn it cites, and that every number — stop_reason, turns_used, model_calls, the cost block — matches the run record. An LLM writing the same report will tidy a quote into something nicer and round a stopped shift up to a clean finish; the validator catches both because it compares bytes and integers, not impressions.

The two lies are different, so you need both validators

validate_coverage compares the report against the events the harness logged, not against the report itself: every event must surface in moments, and every event whose type starts with safety must reach the safety slot. That is the only way to catch a lie that leaves no false sentence on the page. A report that passes one validator and fails the other is still dishonest — honesty is the conjunction. Neither validator is a judge: they check provenance, not quality, and inventing a quality score here is S12's problem, not this module's.

S05's consent gate remains active. These examples use an explicitly scripted customer, not approval inferred from the model's prose. A refused action stays refused when reported.

A failed run still gets a report

A shift that hit the turn cap is not an excuse to write nothing. capped_shift_trace() is one such trace kept as teaching material: the allergen check found milk in the croissant, the model asked a clarifying question too many, and the harness stopped the shift before the ticket was fired. The same generator and the same validators run over it; the outcome slot simply says so (OUTCOME_NOTES["turn_cap"] begins with INCOMPLETE). Absence is stated, never silent: when no safety event exists, the rendered report says "none logged".

Build (in the notebook, predict first)

Open toy.py. Same endpoint configuration as S06.

1. Run a shift and read its events. Three scripted customer lines, the selected client, then log_events(shift_run) prints what the harness can actually see. Count the safety events before you read the report.
2. Predict the honest report. Before running the next cell: will either validator report a violation? If the answer is obviously "no", ask what that proves — and what it does not.
3. The omission. reassuring_variant drops the safety events and keeps everything else. Watch validate_citations wave it through — nothing on the page is false — and validate_coverage refuse it. If the selected run logged no safety event, the cell explicitly uses the separate authored capped trace for this control. Removing nothing cannot demonstrate an omission. The original run and checkpoint remain unchanged. That asymmetry is the lesson.
4. Your turn — the thirty-second test. Write attempt_thirty_second_test(report): return the slots that are missing or empty. It is a proxy for the reader, not a judge — and knowing the difference is the point. The reference is behind a reveal switch; flip it after you attempt.
5. The capped shift. Run the same generator and validators over capped_shift_trace() and read the outcome slot. A failed run gets a report too.

Checkpoint — the number you bank

Generate a debrief for each scenario in your S02 golden set and record how many pass both validators. Write it down with the model name. Anything short of all of them is a generator bug, not a reporting style choice — a shift you cannot debit honestly is a shift you cannot defend.

State of the art (as of August 2026)

Annotated readings

• Anthropic, citations. Extract: how a cited span is represented and what happens when the source does not contain the claim. Compare it with _check_quote.
• Bohnet et al., Attributed Question Answering. Extract: the separation between attribution and quality, and how they measure each. That separation is the reason this session ships two validators and no score.
• RAGAS docs. Extract: the definitions of faithfulness and relevance. Then note what none of them measure: whether an event the harness logged was omitted from the summary.
• ARES, arXiv:2311.09476. Extract: what has to be true before a model-based judge may grade anything — calibration data, agreement measurement, a stated threshold. S12 comes back to this.
• cafe/report.py, the write_report docstring. Two rules, one paragraph. Read them and then find the line of code that enforces each.

Misconceptions and failure modes

• "The report is accurate, so it is honest." Accuracy is per sentence. Omission is a whole-document defect: reassuring_variant keeps every sentence true and still lies.
• "A citation validator is enough." It catches fabrication, not absence. Without coverage, deleting the awkward event is a clean pass.
• "The model wrote a nice summary, so the summary is right." A tidy paraphrase is the most common fabrication precisely because it reads well. Compare bytes, not impressions.
• "A run that failed does not need a report." Failure is the run you most need on the record. The outcome slot says INCOMPLETE and the attempt log stays attached.
• "Quote roughly; the meaning is what counts." Then the validator cannot check anything, and neither can the reader. Quotes are copied, not remembered.
• "These validators grade quality." They check provenance and coverage. Quality belongs to a calibrated judge (S12), and calling a provenance check a quality score is its own small lie.

Self-check

What this unlocks

You can describe one shift honestly, and you can prove the description resolves to the trace. You still have no idea which failures recur — every trace gets read on its own. S10 — Error analysis turns a pile of real traces into open-coded notes, a labeled taxonomy, and a top category that becomes a permanent eval task.