S12-judge-calibration — A judge you have not calibrated is an opinion

The hook

Your judge scored the shift 9/10, so you let it gate the release. A week later a customer found two defects it had waved through — and one clean order it had failed for being "too terse". You never measured either number. The 9/10 was never a measurement; it was an opinion with a decimal point.

The promise

By the end of this session you will have seeded known defects into clean café transcripts, labeled the corpus yourself before any judge output existed on your screen, and reported a detection rate, a false-positive rate, and Cohen's κ — with the pair for an uncalibrated rubric next to the pair for a calibrated one, so the change is attributable. You will be able to say exactly what that κ licenses and what it does not.

The six-case corpus is an arithmetic demonstration, not release calibration. Measure your model rather than assuming it will be good or bad; a release decision needs a larger, independently labeled corpus that you did not use to tune the rubric.

The theory in depth

A judge is an instrument; an uncalibrated instrument is an opinion

A rubric judge is a model call that returns pass/fail. Nothing in that call tells you how often it is right, and a verdict is far more persuasive than it deserves to be. Calibration is the act of measuring the instrument against something you already know.

You need an answer key you built

So you seed the defects yourself. Take clean café transcripts, mutate a known number of them in ways the harness can prove: an allergen served after the customer declared one, a price that is not on the menu, a ticket fired before the customer confirmed. Now every transcript has a key — defective class, or clean — and the judge never sees it.

The seeded defects must be quiet: the kind a tired shift actually ships, not obvious garbage. A judge that only catches the blatant is useless.

The order of operations is the protocol

Hand labels come before judge output, because a verdict you have already seen anchors your label and the measurement dies quietly. Detection reported without false positives is half a number: a judge that fails everything detects everything. And a judge you tuned against this corpus has stopped measuring it.

κ, and what "undefined" means

Raw agreement flatters a judge when one class dominates. Cohen's κ subtracts the agreement chance would have produced anyway:

κ = (p_observed − p_expected) / (1 − p_expected)

If the chance term is degenerate — every transcript in both vectors carries the same label — κ is undefined, not 1.0. cohens_kappa returns None there, on purpose: reporting 1.0 for two constant vectors would be a lie, and a report that turns that None into 1.0 is lying to you. Read a real κ as "how much of the agreement was base rates doing the work." κ = 0.4 is a judge that has earned a bigger calibration set, not one you can gate on.

One verdict per transcript, and unparseable is a result

The harness guarantees the mechanical part: run_judge_all returns exactly one verdict per transcript, in a fixed, mixed corpus order, so "the defective ones are first" never becomes a habit. parse_verdict requires one JSON object with verdict, class, and a nonempty rationale. It rejects extra prose, duplicate keys, unknown classes and contradictions: pass requires none; fail requires a defect class or other. V1 uses other for its intentionally poor style criterion. Invalid output becomes unparseable, never pass.

The request includes menu facts separately from the transcript and treats transcript instructions as evidence. Add approval/tool facts for each real run when available. Report valid-output coverage alongside detection and false positives, which retain the full defective/clean denominators. Zero alarms with zero valid outputs is no evidence of quality.

Calibration is a property of the pair

The rubric and the model are one instrument. v1 of the rubric is strict, style-sensitive, and told to distrust short answers — the judge most people ship on the first try. v2 spells out S10's failure classes and deletes the style bias, with the same model and the same corpus. Compare both rates and coverage on the same corpus. Either rubric can perform worse; the direction is an observation, not an assertion. Changing a rubric does not prove that its model improved.

Build (in the notebook, predict first)

Open toy.py. Start on the offline stub without configuration. For live measurements:

export COURSE_MODE=live
export CAFE_BASE_URL=http://127.0.0.1:11434/v1
export CAFE_API_KEY=ollama
export CAFE_MODEL=qwen2.5:14b-instruct
uv run python -m cafe.doctor
uv run marimo edit sessions/s12-judge-calibration/toy.py

1. Read the corpus. Six clean café transcripts in a fixed, mixed order; three of them carry exactly one seeded defect. All six are displayed for your independent reading. Key-derived results stay hidden until labels are complete and the separate reveal switch is on.
2. Hand-label first — this is the protocol, not a warm-up. Fill attempt_hand_labels with pass/fail for every ID against the three factual defects. Use the menu facts; brevity and style are not defects. Blank, partial, invalid or cleared labels pause every judge call and its dependent results.
3. Judge v1, then reveal separately. Once labels are complete, the deliberately strict, style-sensitive rubric can run. The seed key, missed-ID lists and key-derived comparisons remain hidden until you also turn on the reveal switch.
4. Predict the pair. Before completing the label cell, write down how many defective transcripts judge v1 will catch and how many clean ones it will fail. Detection and false positives are reported together, because either alone is half a number.
5. Read the misses, then judge v2. The v1 misses are not mysteries; they are classes. v2 spells them out and removes the style bias — same corpus, same model, same order, only the rubric changed. Re-measure on the same corpus and compare the pairs.
6. κ both ways. cohens_kappa compares the judge vectors with the reference labels. If both vectors are constant, it returns None and the notebook prints "undefined" — the honest answer. The closing cells are protocol invariants: κ is undefined for constant vectors, every transcript gets exactly one verdict, and the rates and coverage retain their denominators.
7. Price the twelve calls. Calibration cost two rubrics over six transcripts. The cost cell rebuilds each call's messages exactly and estimates them using S11's illustrative rate card, without changing the actual client — worth-it is a question with two numbers, and now you have both.

Checkpoint — the number you bank

Put this in one sentence and defend it: 6 transcripts, 3 seeded defects, detection n/3, false positives n/3, valid outputs n/6, κ = … for the calibrated rubric — with the uncalibrated pair next to it so the change is attributable. On a small local model those numbers may be poor. That is your instrument's actual precision, and it is worth more than a 9/10 you cannot reproduce. Bank the projected calibration cost next to the κ: twelve calls estimated with S11's illustrative table. A verdict is worth what it measures minus what it cost to get. Keep S02's comparison receipt, including model/configuration, prompt/checker/data versions, timeout, budget, sample count and environment. Tune on development examples, then freeze the rubric before an independently labeled holdout. The six teaching transcripts cannot serve both roles. If you generated a wider corpus with simulated users, label that provenance: Lost in Simulation shows why simulator evidence should not stand in for observations from people.

State of the art (source review: 20 September 2026)

Annotated readings

• Hamel Husain, Creating a LLM-as-a-Judge that drives business results. Extract the order of operations — hand labels before judge output — and the insistence on reporting agreement as a pair, not a single flattering number.
• Cohen's κ, overview. Extract the degenerate case and why chance-correction matters: raw agreement flatters a judge when one class dominates.
• Zheng et al., Judging LLM-as-a-Judge. Extract the catalogue of judge biases. It explains why a small local model lands where it lands, and why the rates come before the verdict.

Misconceptions and failure modes

• "The judge said fail, so it is a defect." Not until you have measured the judge. An uncalibrated verdict is an opinion with a decimal point; the calibration set is what turns it into an instrument.
• "Label after you have seen the verdicts." A seen verdict anchors your label, and the measurement quietly becomes the judge's influence on you. Hands first, always.
• "Report detection." Detection without false positives is half a number: a judge that fails everything detects everything. Report the pair.
• "κ = 1.0 means a perfect judge." Two constant vectors agree 100% of the time and that agreement carries no information. cohens_kappa returns None there, and that None is the honest answer.
• "A bad local judge means the setup is broken." A small local model is often a poor judge — that is the finding. Calibration is what lets you say how poor, and whether it is worth the calls.

Self-check

What this unlocks

That is the core path closed. Twelve sessions produced one cafe/ package — a loop with proven invariants, a golden set, a context budget, a ticket contract, a consent gate, layered detection, bounded repair, a replayable trace, an evidence report, a failure taxonomy, a cost and latency ledger, and a calibrated judge — plus a banked number per session.

You are not done, but the next step is deliberately unaided and OPTIONAL: S13 — Rebuild from memory is the closed-book audit. Close every notebook and rebuild the core from memory — no scaffolding, no generated answers, nothing filled in for you. Everything you would need is already in what you built; the audit only tells you whether it is in you.