The problem #

Chapter 08’s harness scored outputs with exact-match and rule-based checks: does the output contain a substring, does it match a regex, does a field equal an expected value. That works when correctness is objective. Most agent outputs aren’t. “Is this summary faithful to the source” and “did the final answer actually address the question” have no fixed string to compare against, and the phrasing varies on every run, so a rule-based scorer either rejects valid answers or lets bad ones through. You still want an automated, repeatable score for every case in the dataset, not a person reading transcripts by hand.

The concept #

The fix is a second LLM call whose only job is grading. Give it three things: the original input, the output being graded, and explicit criteria (a short rubric describing what “passes” means for this case). It returns a structured verdict, not another paragraph of prose: a pass/fail plus a reason.

graph LR
    I[Input] --> G[Generator]
    G --> O[Output]
    I --> J[Judge]
    O --> J
    C[Criteria] --> J
    J --> V[Verdict]

Two paths branch off the same input. One produces the output. The other grades it, and takes the output, the input, and the criteria as its own separate set of inputs, nothing else. That branching is the entire design, and it’s worth taking apart properly: why the judge has to be its own call, what specific failure modes that separation is defending against, and where it still falls short.

Why the judge has to be a structurally distinct call #

The tempting shortcut is to let the model that produced the output also grade it, in the same turn: “and now, rate your own answer.” That fails for a mechanical reason, not a motivational one. A model grading its own just-generated output still has that output’s reasoning trace in context, the same chain of token-level decisions that led to the answer. Asking it to judge that answer is asking it to evaluate its own conclusion using the same reasoning that reached the conclusion in the first place, so it re-finds the path it already committed to and rubber-stamps it. This is the same structural problem [[Generator-Evaluator Pattern]] names directly: agents are systematically poor at evaluating their own free-form work, and the fix in that pattern is a standalone evaluator, tuned to be skeptical, that never shares the generator’s reasoning context.

A separate call, with its own system prompt and no memory of why the output was written that way, can only see the artifact, not the intent behind it. It has nothing to rubber-stamp; it has to grade against the stated criteria because the criteria are all it was given. This is also chapter 07’s structured output guardrail, aimed at a grading decision instead of a tool call: with_structured_output(Verdict) forces the response into passed: bool and reason: str, so the model can’t dodge into “it’s mostly fine, though there are some considerations” and call that a verdict.

Self-preference and self-enhancement bias #

Structural separation alone doesn’t fully solve the next problem: self-preference bias, sometimes framed more broadly as self-enhancement bias, is the tendency of a judge to rate an output more favorably when that output came from the same model, or the same model family, as the judge itself. This isn’t a personality quirk, it’s a byproduct of how the judge scores fluency and coherence. A model’s sense of what “good” text looks like is shaped by its own training distribution: phrasing conventions, the specific way it structures an explanation, the alignment-tuning artifacts baked into how it hedges or concludes. When it’s handed an output that matches those patterns, because the same family produced it, that output reads as more fluent and more “obviously correct” to the judge, even graded on criteria that say nothing about style. Research on this has found GPT-4o scoring its own outputs several points higher than equivalent human or other-model text on the same rubric, with the effect persisting even when authorship is hidden from the judge, and it tends to scale with model size and the amount of post-training the judge model has had. Part of the mechanism is self-recognition: larger, more capable models are measurably better at implicitly recognizing their own outputs’ stylistic fingerprint, and that recognition capacity correlates with how strong the self-preference effect is.

This is exactly why structural separation reduces the bias without eliminating it. Moving the judge into its own call strips away the generator’s specific reasoning trace, so the judge can no longer just replay the chain of thought that produced the answer. But if the judge and the generator share the same underlying weights, or even just the same training lineage, they still share the same priors about what fluent, well-formed text looks like. The call is separate; the taste is not. That’s the mechanical reason a materially different model family as judge is worth the extra setup, and why a same-family judge’s pass rate should be read as optimistic rather than neutral.

Position bias in pairwise judging #

A related but distinct failure shows up once you move from single-output grading to pairwise comparison, where the judge sees two candidate outputs side by side and picks the better one. Position bias is the tendency to favor whichever answer sits in a particular slot, first or second, independent of content. A large-scale study evaluating this across a dozen LLM judges and over 100,000 pairwise comparisons found the effect is systematic, not sampling noise: swapping which answer appears first, with content held constant, measurably changes the verdict for many judges, and the bias is stable across repeated runs rather than washing out. The practical mitigation is mechanical: run the same pairwise comparison twice with the two candidates’ positions swapped, and only trust a verdict where both orderings agree.

Run-to-run variance #

The single-output judge has its own reliability problem, independent of both biases above: ask it to grade the identical input and output twice, and it will not always return the same verdict. This is decoding stochasticity showing up in a structured field that looks deceptively authoritative. Test-retest studies on judge consistency put the same-verdict rate above 95% at temperature 0, dropping to roughly 70% at temperature 1, so the sampling temperature you pick for the judge model directly trades off against how reproducible its verdicts are. Pinning the judge to temperature 0 buys consistency, but not for free: some of that same research finds a fully deterministic judge can drift further from human judgment than one with a small amount of temperature, so consistency and accuracy aren’t the same axis and pushing on one can cost you the other. What does help reliably is aggregation: running the judge multiple times per case and taking a majority verdict measurably improves agreement with human evaluators over trusting any single run.

Put together, the three failure modes sit at different layers. Self-preference and self-enhancement bias come from what the judge and generator have in common (shared training priors). Position bias comes from how a pairwise prompt is structured, regardless of which model is judging. Run-to-run variance comes from decoding itself, and would exist even with a perfectly unbiased judge. Structural separation, the distinct call with its own context, addresses the first cleanly and the second not at all, which is why each needs its own mitigation rather than one fix covering all three.

The judge call in code #

The mechanism above is exactly what the code does, line for line. Verdict is the structured response from chapter 07’s pattern: a Pydantic model with passed: bool and reason: str, so the judge can’t return anything that isn’t already parsed and typed. JUDGE_PROMPT is the judge’s entire context: a system message that explicitly tells it to grade against the stated criteria rather than its own opinion (the instruction-level defense against the self-preference mechanism described above), and a human message that hands over exactly the three inputs from the diagram: input, output, and criteria, nothing about how the output was produced.

 1from pydantic import BaseModel, Field
 2from langchain_core.prompts import ChatPromptTemplate
 3
 4class Verdict(BaseModel):
 5    passed: bool = Field(description="Whether the output meets the stated criteria")
 6    reason: str = Field(description="One or two sentences justifying the verdict")
 7
 8JUDGE_PROMPT = ChatPromptTemplate.from_messages([
 9    ("system", "You are a strict evaluator. Judge only against the criteria given, "
10               "not your own opinion of what a good answer looks like."),
11    ("human", "Input:\n{input}\n\nOutput to grade:\n{output}\n\nCriteria:\n{criteria}\n\n"
12              "Does the output satisfy the criteria? Return a verdict."),
13])
14
15def make_judge(judge_model):
16    structured_model = judge_model.with_structured_output(Verdict)
17    chain = JUDGE_PROMPT | structured_model
18
19    def judge(input: str, output: str, criteria: str) -> Verdict:
20        return chain.invoke({"input": input, "output": output, "criteria": criteria})
21
22    return judge
23
24# Wired into the chapter 08 harness shape:
25def run_eval(cases, agent, judge):
26    results = []
27    for case in cases:
28        output = agent(case["input"])
29        verdict = judge(case["input"], output, case["criteria"])
30        results.append({"case": case["id"], "passed": verdict.passed, "reason": verdict.reason})
31    pass_rate = sum(r["passed"] for r in results) / len(results)
32    return pass_rate, results

make_judge takes judge_model as a parameter rather than reusing whatever model instance produced output, so the separation from the mechanism section is enforced by the function signature, not just by convention: pass it a different model and there’s no code path back to the generator’s reasoning. Inside run_eval, agent and judge are two separate calls per case, in that order: the generator runs first and produces output, then the judge runs against case["input"], that same output, and case["criteria"], matching the diagram’s arrows exactly. judge(...) returns a typed Verdict, so verdict.passed is a real boolean and verdict.reason is a real string, not a paragraph you’d need to parse for “yes” or “no.” The rest of run_eval is unchanged from chapter 08’s shape: a loop, a tally, a pass rate.

Trade-offs: when a judge earns its call #

None of this is free. Every case in the dataset now costs two model calls instead of one, the generator’s and the judge’s, so the harness’s latency and API spend both roughly double, and that’s before running the judge multiple times per case to deal with run-to-run variance. On a small eval set that’s negligible; on a dataset large enough to catch real regressions, run repeatedly in CI, it’s a real line item, and it’s worth sizing before wiring a judge into every case by default.

That cost is also the reason a judge isn’t the default scorer, it’s the scorer for when a simpler one can’t do the job. If the case has an objective check, an exact string, a required substring, a value within a numeric tolerance, chapter 08’s rule-based scorers are cheaper, faster, and perfectly deterministic; there’s no bias to mitigate because there’s no judgment being made. A judge earns its call specifically where correctness is open-ended: faithfulness to a source, whether an answer actually addresses the question, tone or completeness, anything where the space of acceptable phrasings is too large to enumerate as a rule.

Given that a judge is going in, the mitigations from the mechanism section aren’t optional extras, they’re the difference between a useful signal and an optimistic one. Use a materially different model family for the judge than for the model under test, so self-preference bias has less shared prior to lean on. Run the judge more than once per case and take a majority verdict rather than trusting a single sample, which is the concrete answer to run-to-run variance. And calibrate the rubric itself: [[Grading Criteria for Subjective Quality]] describes decomposing a vague “is this good” into named dimensions with explicit descriptions of what passing and failing look like for each one, which reduces the room a judge has to fall back on its own priors when the criteria are vague. None of these make the judge an oracle. Together they turn a noisy instrument into one whose noise you’ve actually measured and bounded.

Summary #

An LLM judge exists because chapter 08’s rule-based scorers run out of road the moment correctness stops being a fixed string to match. Structuring the judge as its own call, with its own prompt, its own model instantiation, and a schema-constrained verdict, is what makes it usable at all: it’s the same separation principle as [[Generator-Evaluator Pattern]], applied through chapter 07’s structured output guardrail. But that separation is a mitigation, not a cure. Self-preference bias survives because judge and generator can share training priors even across separate calls; position bias survives in pairwise setups because it’s a property of prompt structure, not model identity; run-to-run variance survives because it’s decoding stochasticity, present even in an unbiased judge. Treat every pass rate a judge produces as a measurement with error bars, not a fact, mitigated by model-family diversity, repeated sampling, and a calibrated rubric, but never fully removed. Chapter 08 gave this judge a harness to plug into; chapter 10 covers tracing agents with OpenTelemetry, for when an aggregate pass rate says something broke and you need to see, inside one specific run, exactly where.