The problem #

Chapter 09 closed with a gap: the eval harness can tell you a pass rate dropped from 92% to 78%, but not which of the failing cases went wrong for the same reason, or where in the agent’s steps it happened. When one specific run misbehaves, calls the wrong tool, hangs for ten seconds, or returns the wrong final answer, print statements and the final output alone don’t show you the sequence that led there. You need to see, for that one run, every model call and every tool call in order, with their inputs, outputs, and timing. That’s per-run visibility, and it’s a different problem from aggregate scoring.

The concept #

OpenTelemetry represents a single run as a tree of spans. A span is a timed unit of work: a name, a start and end time, a set of key/value attributes, and a status. Spans nest, and that nesting is the whole point: a top-level span for the agent or graph invocation is the root, and every model call and tool call inside it becomes a child span, in the order they actually ran. Read the tree and you’re reading the run’s timeline, down to which node called which tool, what arguments went in, what came back, and how long each step took, all without adding a single print statement.

flowchart TD
    A["invoke_agent<br/>trace_id 7b3e...a1, span_id 01"] --> B["chat gpt-4o-mini<br/>span_id 02, parent 01"]
    A --> C["execute_tool get_weather<br/>span_id 03, parent 01"]
    A --> D["chat gpt-4o-mini follow-up<br/>span_id 04, parent 01"]

That picture is simple to read, but the mechanism underneath it is worth being precise about, because “span” and “trace” get used loosely.

Trace ID, span ID, and how the tree actually forms #

Every span carries a SpanContext: a trace ID (128 bits, shared by every span in the run), a span ID (64 bits, unique to that one span), and a small set of trace flags. A trace isn’t a container object anyone creates; it’s just the set of every span that happens to share a trace ID. The tree shape comes from one more field: each child span records its parent’s span ID. invoke_agent’s span has no parent, so it’s the root; the chat gpt-4o-mini span inside it records invoke_agent’s span ID as its own parent, and so does execute_tool get_weather. That’s the entire nesting mechanism: no separate tree data structure anywhere, just IDs pointing at IDs, reassembled into a tree when a backend renders the trace.

call_model and call_tool, in the code below, never pass a parent span to each other explicitly, and they don’t need to. start_as_current_span pushes the new span onto a context variable that tracks “the current span” for whatever’s executing right now. Open a span inside another open span’s with block and the SDK reads that current span from context and stitches in the parent ID automatically. (The same SpanContext is what gets serialized into a traceparent header when a trace crosses a network call into another service, which is how a trace stays intact across process boundaries. A single-process agent like the one here never needs that, but it’s the same mechanism, just scaled up.)

What the GenAI semantic conventions actually standardize #

A span with a name and a timestamp is useful on its own, but it doesn’t tell a backend what kind of work happened, which is the problem the gen_ai.* semantic conventions solve. They standardize two things: the attribute keys that go on a span, and the span name pattern built from them.

A model call span sets gen_ai.operation.name to chat (or text_completion, generate_content), plus gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, and token counts under gen_ai.usage.input_tokens / gen_ai.usage.output_tokens. A tool call span sets gen_ai.operation.name to execute_tool, plus gen_ai.tool.name and gen_ai.tool.call.id, identifying which tool ran and which call it was. The span name follows from the operation: chat gpt-4o-mini, execute_tool get_weather.

Why standardize this instead of letting every team pick its own keys (“model” versus “modelName” versus “llm.model”)? Because the payoff lands on the backend side, not the producing code. A tracing UI that wants to show tokens per call, or group spans by model, doesn’t have to special-case every team’s naming scheme: it reads gen_ai.usage.output_tokens and gen_ai.request.model, and that works whether the span came from this chapter’s manual code, LangChain’s auto-instrumentation, or an unrelated team’s service calling a different model entirely. As of 2026 these conventions are still marked experimental in the OpenTelemetry spec (attribute names can still shift between spec versions), but they’re stable enough that the major tracing backends already render them the same way.

Getting a trace: automatic, decorator, or manual #

Underneath any of them is the same idea: a nested tree of timed, attributed steps, one per model call or tool call, in the order they actually ran. Two lower-code alternatives exist to the manual approach this chapter builds. If the traced code is already built from LangChain or LangGraph constructs, tracing turns on with an environment variable (LANGSMITH_TRACING=true) and zero call-site changes, via LangChain’s callback manager, or its OTel-flavored equivalent opentelemetry-instrumentation-genai-langchain; either way, only LangChain-native calls are visible, so a bare ollama.chat() call like chapter 2’s raw loop never gets seen. For arbitrary Python outside LangChain, the langsmith SDK’s @traceable decorator gets a comparable trace with one decorator per function, inferring nesting from the call stack instead of an explicit context variable, at the cost of a LangSmith-shaped trace.

This chapter builds the third option, manual OTel spans, because it’s the only one of the three that isn’t tied to a single backend, and because it maps directly onto the trace ID, span ID, and gen_ai.* mechanics explained above. Real codebases often mix tiers rather than picking one: a LangChain-native agent gets the automatic tier for free, a raw loop next to it gets @traceable, and a step neither covers (a custom retrieval call, a boundary you want attributed a specific way) gets a manual span layered in.

Exporters: what actually ships a span off the process #

Creating and populating a span, with start_as_current_span and set_attribute, only builds it in memory. Getting it anywhere useful is a separate step. When a span ends, it’s handed to a SpanProcessor; a BatchSpanProcessor (the one used in anything beyond a demo) buffers finished spans and flushes them periodically, or once a batch size is reached, then hands the batch to an exporter. The exporter serializes that batch, commonly as OTLP, and ships it over the network to a backend or collector: Jaeger, Tempo, Honeycomb, LangSmith, whatever’s configured. With no exporter configured, every span in the tree above still gets created, attributed, and closed correctly; it just never leaves the process, and is gone the moment that process exits.

This is a different, complementary layer to chapter 08 and 09’s evals. Evals run many cases and give you a number: something is wrong, at roughly this rate. Tracing gives you the one run, with every step visible: here, specifically, is where it went wrong. You need both. A pass rate tells you to go look. A trace tells you what you’ll find.

Minimal example #

You can instrument spans manually with the OpenTelemetry SDK, which is the clearest way to see how the conventions and the propagation mechanism above actually map onto code:

 1from opentelemetry import trace
 2
 3tracer = trace.get_tracer("agent.tracing")
 4
 5def call_model(client, messages, model="gpt-4o-mini"):
 6    with tracer.start_as_current_span(
 7        f"chat {model}",
 8        attributes={
 9            "gen_ai.operation.name": "chat",
10            "gen_ai.provider.name": "openai",
11            "gen_ai.request.model": model,
12        },
13    ) as span:
14        response = client.chat.completions.create(model=model, messages=messages)
15        span.set_attribute("gen_ai.response.model", response.model)
16        span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
17        span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens)
18        return response
19
20def call_tool(tool_name, tool_call_id, fn, **kwargs):
21    with tracer.start_as_current_span(
22        f"execute_tool {tool_name}",
23        attributes={
24            "gen_ai.operation.name": "execute_tool",
25            "gen_ai.tool.name": tool_name,
26            "gen_ai.tool.call.id": tool_call_id,
27        },
28    ) as span:
29        result = fn(**kwargs)
30        span.set_attribute("gen_ai.tool.call.result", str(result)[:500])
31        return result

call_model and call_tool never take a parent-span argument, for the reason described above: whichever span is current when either function runs becomes the parent automatically, through context, not through an explicit handoff. Every attribute set on either span, gen_ai.operation.name, gen_ai.provider.name, gen_ai.usage.output_tokens, is a key from the semantic conventions, not an invented one, which is what lets a backend that has never seen this code render it correctly. Notice the truncation on the last line of call_tool: str(result)[:500] caps how much of a tool’s output gets attached to the span, a small, concrete instance of the verbosity-versus-cost trade-off discussed below.

Wrap the whole agent step in a parent span (invoke_agent, or the graph run) and every call_model and call_tool invocation inside it becomes a child, so the exported trace looks like the tree shown earlier: one invoke_agent span, containing a chat gpt-4o-mini span, then an execute_tool get_weather span, then another chat span for the follow-up. Open that trace in any OTel-compatible backend and you’re looking at exactly what the agent did, in order.

For a real LangGraph app you’d rarely write this by hand: this is the manual tier, chosen deliberately when you need a custom attribute or a span boundary the automatic and decorator tiers won’t give you, both of which describe the same shape of run at a different point on the code-versus-control line.

Trade-offs #

Instrumenting only the top-level call costs almost nothing, one span around agent.run(...), and buys almost nothing back: it tells you the run took 4.2 seconds and succeeded or failed, which is barely better than a print statement. All the diagnostic value sits in the child spans, and getting it costs real verbosity, a with block and a handful of attributes at every call site that matters, plus the runtime cost of creating, populating, and exporting each one. That’s the actual trade-off, not a mistake to avoid but a dial to set: a flat trace is cheap and tells you almost nothing; a fully instrumented one costs setup and per-call overhead and tells you exactly where a run went wrong. Attribute size is part of that same cost once you’re populating spans: it’s easy to attach a full prompt or a full tool response to every span “just in case,” and the str(result)[:500] truncation in the code above is a deliberate cap on that, because an exporter shipping full payloads on every call is a real cost in both network and, depending on the backend, storage.

Manual and auto-instrumentation sit on the same cost curve from a different angle. Manual instrumentation costs code at every call site but buys exact control over what’s on each span and where boundaries fall, which matters when a step in your pipeline (a retrieval call, a re-ranking pass) isn’t a model or tool call at all and wouldn’t get a span from any instrumentation package by default. Auto-instrumentation trades that control for zero code, but only at the boundaries its authors chose, so most production setups end up mixing both rather than picking one.

There’s a second axis besides code-versus-control: where the trace is allowed to go. The zero-code LangSmith tier and @traceable both point at LangSmith specifically, while OTel spans, hand-written or auto-instrumented, are portable to any OTel-compatible backend: Jaeger, Tempo, Honeycomb, or LangSmith too, via its own OTel import path. If LangSmith is already the only place anyone looks at traces, @traceable is close to free and there’s little reason to reach for manual spans; that changes the moment a second backend, or a service that doesn’t speak LangSmith, enters the picture.

Whether any of this is worth the setup cost, an SDK, an exporter, a backend to look at the result, depends on how much structure there is to lose. A single call (one prompt in, one response out) has nothing a log line with a latency and a token count doesn’t already capture; standing up tracing for that is pure overhead. Tracing earns its cost once a run has real shape: several tool calls, a branch that sometimes takes a different path, a retry, a subagent handing off to another subagent, anything where the order and nesting of steps is itself part of the question, not just the final answer. Below that complexity threshold, logging isn’t a lesser tool, it’s the right-sized one.

Summary #

Chapters 08 and 09 gave you an aggregate signal: a pass rate, or an LLM-judged verdict, that tells you something is wrong and roughly how often, across a whole dataset of runs. Tracing is the layer underneath that: a span tree, keyed by a shared trace ID and stitched together by parent span IDs, showing every model call and tool call in one specific run, in order, with what went in, what came back, and how long each step took. One layer tells you to go look; the other tells you what you’ll find when you do, down to which node called which tool, with what arguments, and where in the sequence it actually went wrong.

That visibility is also what turns the next problem from theoretical into obvious. A trace that shows a tool called six times before the run gave up, or a single step that took nine seconds while every other step took under one, isn’t just useful for debugging, it’s evidence that a hard boundary is missing: nothing yet stops that ninth-second call from becoming a ninetieth, or that sixth retry from becoming a sixtieth. Chapter 11 covers the mechanisms that turn what a trace reveals into something enforced: timeouts, retries, and loop limits.