
Why a Model Trained by an AI Judge Stopped Saying Rome
Why a Model Trained by an AI Judge Stopped Saying Rome Source: Blind Spots Travel, Knowledge Stays Home: What RL Inherits from an LLM Judge Paper was published on November 18, 2026 This episode was AI-generated on November 20, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. A Llama model knew the capital of the Roman Empire thirty-two times out of thirty-two. After a short round of training with an AI judge as grader, it refused to answer — and blocking it from saying 'I don't know' brought Rome right back. A new preprint traces the whole collapse to eleven rejected answers, and argues that what transfers from an AI grader isn't its knowledge but its blind spots. Key Takeaways Why a grader's knowledge about individual answers mostly stays on the training set — worth about one corrected answer per hundred new questions How a judge's mild skepticism (accepting 13–45% of correct former-state capitals) became a near-total refusal rule in the student (89–98%) The hand-edited causal test: rejecting 4 of 750 training answers pushed category refusal to 85%; rejecting all 12 pushed it to 97% Why the damage is invisible on a dashboard — judges' students refused 25.5 known answers per 100 vs 26.5 for random rejection, the same total, redistributed Why a five-judge panel from different labs doesn't fix it: judges agreed on rejections 76–88% of the time and the jury still removed 85–95% of the category The label-free alternative — a student grading itself by 32-sample majority vote — and the one student profile where it breaks down 00:00 — The model that forgot Rome on purpose The cold open: a Llama model trained with an AI judge refuses the capital of the Roman Empire, but produces it instantly when 'I don't know' is blocked. 01:18 — Why anyone lets an AI grade an AI Finn explains why reinforcement learning needs a grader, why most tasks have no answer key, and the unspoken hope that the judge's knowledge rubs off on the student. 02:04 — The quota that forces good answers out The experimental setup — 814 runs, swappable graders, and a half-point reward for abstaining — plus the arithmetic showing a strict judge must reject correct answers. 04:09 — Does the grader's knowledge actually transfer? A patternless control grader shows single mistakes get memorized but don't generalize, and a 32-sample self-consistency baseline shows trained students barely beat it. 06:25 — A lean in the judge, a rule in the student The former-state capitals probe reveals judges are skeptical by degree while students turn that skepticism into near-total refusal. 07:33 — Eleven examples did all of this The hand-edited verdict experiment establishes causation, scaling refusal from 31% to 97% by flipping twelve grades — and it holds at a gentler, answer-key-matched setting. 08:55 — Withheld, not lost — with one condition Blocking abstention recovers ~95% of refused answers in Llama and ~90% in Qwen, but only under a training penalty that keeps the model near its original self — and one dataset shows genuine loss. 10:11 — Same grocery bill, vegetables gone Aggregate refusal rates look identical to random rejection, and a five-judge majority vote fails because judges from different labs share the same doubts. 11:15 — Where the dramatic numbers stop holding The steelman: the biggest effects come from the strictest setting, the gentler setting is unstable run to run, the category was found in logs first, and open-ended tasks are untested. 12:10 — What if the student grades itself? A label-free self-vote grader matches the real answer key, never collapses, and preserves the category — except against an overconfident student, where the agreement threshold has to be tuned. Recommended Reading Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — The canonical study of LLM judges and their systematic biases — useful context for why the episode's strict graders reject 'Rome' and why five judges from different labs share the same doubts. Scaling Laws for Reward Model Overoptimization — Quantifies what happens when you optimize hard against an imperfect proxy reward, the general phenomenon behind this episode's finding that a judge's small lean becomes a student's absolute rule. Self-Consistency Improves Chain of Thought Reasoning in Language Models — The majority-vote-over-samples idea that the paper repurposes as a label-free grader — the method that matched the real answer key while preserving the lost category. Language Models (Mostly) Know What They Know — Background on whether a model's own sampled agreement tracks correctness, which is exactly the assumption the episode's 32-sample abstention baseline and self-vote grader rest on.
- Transcript