The problem #

Chapter 07 closed on a gap: a validated JobSearchResponse proves the agent’s output has the right shape, not that it’s true. The usual way people check truth is by running the agent a few times and reading the output, which works until it doesn’t scale, and it never catches a regression. Tweak a prompt, swap a model, bump a dependency, and the only way to know if anything broke is to remember what “good” looked like last time and eyeball it again. That doesn’t survive more than a handful of changes. What’s missing isn’t more manual testing, it’s something repeatable that hands back a number.

The concept #

Strip away the color, and every eval harness, from a forty-line script to a hosted platform, decomposes into the same four pieces.

The anatomy of an eval #

A dataset. A fixed, versioned list of cases. Each case pairs an input with something to check the output against: an expected value, a required substring, a numeric target, a rubric, whatever fits the task. “Fixed” matters as much as “list”: if the cases themselves change between two runs, a change in pass rate stops meaning anything, because there’s no way to tell whether the agent regressed or the test did.

A target. The thing under evaluation, wrapped so the harness can call it the same way for every case: hand it a case’s input, get back an output. It doesn’t matter whether that’s a single model.invoke() call, a create_agent run that loops through several tool calls, or a longer chain, the target function’s whole job is to hide all of that behind one call with a consistent signature.

An evaluator (a scorer). A function, one per case or one shared across cases, that takes the target’s output, usually alongside the case’s expectation, and returns a verdict: pass or fail, or a score. This is the piece that actually encodes what “correct” means for a given case, and it’s also the piece people underinvest in most, more on that below.

Aggregation. Tally the individual verdicts into something a human can compare across runs: “7/10 passed,” a mean score, a breakdown by category. Aggregation is what turns forty isolated pass/fail judgments into one answer to the only question that matters after a change: did this help or hurt.

graph LR
    A[Dataset: cases] --> B[Target: agent under test]
    B --> C[Evaluator: scorer per case]
    C --> D[Aggregate: pass rate]

Dataset feeds target, target’s output feeds evaluator, evaluator’s verdicts feed aggregation. Every eval harness is a variation on this pipeline; the differences are in how much infrastructure sits around each box.

Picking a scorer: exact match, rule-based, model-graded #

The evaluator step splits into three broad families, and the difference between them isn’t stylistic, it’s about how much of “correct” can be captured mechanically before real judgment is required.

Exact match compares the output to a single expected string, after normalizing case and whitespace. It’s the cheapest scorer to write and the least forgiving: it only works when the task genuinely has one right answer, a capital city, a status code, a parsed number. Reach for exact match anywhere the space of correct answers isn’t a single string, and it will fail acceptable outputs at the same rate it catches real bugs, which is the fastest way to make a team stop trusting the harness.

Rule-based (or heuristic) scoring replaces the single string with a check: a required substring, a set of terms that must all appear, a regex, a numeric value within a tolerance. It’s still deterministic code, no model call involved, but it tolerates the phrasing variance exact match can’t: “Dublin,” “Dublin, Ireland,” and “The capital is Dublin” all satisfy a substring check for “dublin” even though none of them match each other exactly. Rule-based scoring is the right default whenever correctness reduces to a fact, a range, or a required element, even when the surrounding sentence varies.

Model-graded scoring hands the judgment to a second LLM call, for cases where neither of the above can express what “correct” means: correctness is subjective (is this summary faithful to the tone of the source), or the space of acceptable outputs is too open-ended to enumerate as a rule. That trade costs determinism, the same input can grade differently across two runs of the judge, in exchange for being able to score something no regex could. Chapter 09 covers building that judge call properly; the three cases below don’t need it, because “capital of Ireland” and “12% of 340” are exactly the kind of cases rule-based and exact-match scoring were built for.

Picking the right family per case matters more than picking a framework, and a single dataset is allowed to mix all three: a job-search agent might need exact match on a returned currency code, a substring check on a required disclaimer, and a model-graded check on whether the summary actually answers the question that was asked.

What a platform formalizes #

This is the same shape LangSmith’s evaluate() formalizes at scale: a dataset object holding versioned examples, a target function (the application or model being evaluated), one or more evaluator functions, code-based or LLM-as-judge, attached to that run, and an experiment: LangSmith’s record of one full pass over the dataset, tying every output and score back to a specific application version so two experiments can be diffed against each other later.

A minimal hand-rolled version deliberately skips the layer built around that loop: no dataset versioning beyond “it’s a list in a file,” no stored experiments, no UI for comparing two runs side by side, no automatic diffing against a previous run, just the dataset, the loop, and a printed number. What you get in exchange for skipping it is a script readable top to bottom in thirty seconds, runnable without a network call to anything but the model under test. What you give up is everything a platform buys once a dataset outgrows a Python list or more than one person needs to see the results, which the trade-offs section below gets into.

Minimal example #

Three scorers below cover the three families just described, and the dataset picks the right one per case instead of forcing every case through the same check:

 1def exact_match(output: str, expected: str) -> bool:
 2    return output.strip().lower() == expected.strip().lower()
 3
 4
 5def contains_all(output: str, expected: str) -> bool:
 6    # expected is a comma-separated list of terms that must all appear
 7    terms = [term.strip().lower() for term in expected.split(",")]
 8    return all(term in output.lower() for term in terms)
 9
10
11def numeric_close(output: str, expected: str, tolerance: float = 0.5) -> bool:
12    try:
13        return abs(float(output) - float(expected)) <= tolerance
14    except ValueError:
15        return False
16
17
18CASES = [
19    {"input": "What's the capital of Ireland?", "expected": "Dublin", "scorer": exact_match},
20    {"input": "Name two EU countries that use the euro", "expected": "ireland, spain", "scorer": contains_all},
21    {"input": "What's 12% of 340?", "expected": "40.8", "scorer": numeric_close},
22]
23
24
25def run_agent(user_input: str) -> str:
26    """Stand-in for a real agent.invoke() call. Swap this for the
27    agent under test, keeping the same string-in, string-out contract."""
28    ...  # e.g. agent.invoke({"messages": [HumanMessage(user_input)]})["messages"][-1].content
29
30
31def run_eval(cases: list[dict], agent) -> None:
32    passed = 0
33    for case in cases:
34        output = agent(case["input"])
35        ok = case["scorer"](output, case["expected"])
36        passed += ok
37        print(f"{'PASS' if ok else 'FAIL'}: {case['input']!r}")
38    print(f"\n{passed}/{len(cases)} passed")
39
40
41if __name__ == "__main__":
42    run_eval(CASES, run_agent)

CASES is the dataset: three cases, each pairing an input with an expected value and the scorer that actually fits it. exact_match handles the capital-city case, where there’s exactly one acceptable string. contains_all is the rule-based scorer for the EU-countries case: it doesn’t care about sentence structure, only that both required terms show up somewhere in the output. numeric_close is a rule-based scorer of a different shape again, a tolerance band instead of a substring, for the case where the model does arithmetic and returns a number instead of a phrase. None of the three needs a model-graded scorer, because none of them has a genuinely open-ended answer space.

run_agent is the target: a stand-in with the same string-in, string-out contract a real agent.invoke() call would have, so swapping in the agent under test is a one-line change. run_eval is the runner and the aggregator in one loop: it calls the target once per case, hands the output and that case’s own scorer to get a verdict, prints a line per case, and tallies the pass count into the one number, passed/len(cases), that a second run can be compared against.

Nothing here talks to LangSmith, a database, or a dashboard. Change the prompt, swap gpt-5-mini for another model, add a fourth case, rerun the file, compare the printed ratio to the last one.

Trade-offs and design rationale #

A script like this earns its place before reaching for a platform, but it’s worth being precise about exactly what it buys and what it quietly defers.

Hand-rolled versus adopting a platform. The build cost above is close to zero: it’s the four pieces from the anatomy section and nothing else, and it runs anywhere Python runs. What it defers is everything that turns into a real cost once a dataset grows past a handful of cases or more than one person needs the result: version history for the dataset itself (did case four’s expected value get edited last week, or has the agent actually regressed), a stored history of past experiments to diff against, a UI a non-engineer can read, multi-run aggregation across repeated judge calls. None of that is hard to build from scratch, but building it is exactly the work a platform like LangSmith has already done, and re-deriving it inside a one-off script spends effort on infrastructure instead of on cases. The honest reason to start hand-rolled isn’t that platforms are bad, it’s that a three-case script answers “is this harness idea worth the investment” in an afternoon, and a platform is easy to adopt once that’s proven and the dataset has outgrown a Python list.

What a small, easy dataset doesn’t catch. Three cases the agent has always nailed prove nothing about the fourth case that actually changed behavior. A dataset that only contains the inputs someone thought of while writing the harness tends to cluster around the happy path: well-formed questions, common phrasings, the kind of input the agent was obviously built to handle. It’s the cases nobody thought to add, an ambiguous question, an input format the tool call chokes on, a phrasing that slips past a rule-based scorer’s exact wording, that a prompt change or a model swap is most likely to break. A dataset that never grows past the first pass stops teaching you anything new about the agent from the second run onward.

Why pass-rate alone hides regressions. “7/10 passed” and “7/10 passed” look identical printed side by side even when the two runs failed on entirely different cases. A prompt change that fixes three previously-broken cases while breaking three different ones nets to the same ratio as no change at all, and the harness will report zero net movement on a version that’s actually worse for a whole category of input. The fix isn’t a better single number, it’s keeping the per-case results, not just the aggregate, logged somewhere alongside a date and a commit hash, so “did that change help” gets answered by which specific cases flipped, not just by whether the ratio moved.

Summary #

A dataset, a target, an evaluator, and an aggregate are the entire shape of an eval, whether it’s a forty-line script or a LangSmith experiment: a platform adds versioning, storage, and comparison around those same four pieces, it doesn’t replace them. Chapter 07 gave the agent’s output a validated shape; this harness is what checks that a validly-shaped output is also a correct one, and exact-match and rule-based scorers are enough for that as long as correctness reduces to a string, a substring, or a number within tolerance. The moment a case’s correctness turns subjective, or its space of acceptable answers gets too open to enumerate as a rule, that’s the case for a model-graded scorer, a second LLM call whose only job is grading, in place of writing a stricter regex. That’s chapter 09.