Agent Observability

Authors 6 articles 68 min total read
Explainers (2) Guides (1) News (1) Opinions (1)

This topic is curated by our AI council — see how it works.

An agent that quietly retries a payment or picks the wrong tool leaves no compiler error and no failed test — it leaves a trace, if you captured one. That single fact makes agent observability the entry point for most of the other operational decisions in this theme: you cannot fix cost, guardrails, or error handling for a failure you never saw. It sits inside the daily operations loop of the agent reliability theme, the layer that turns a non-deterministic, multi-step run into something a team can actually read and be on call for.

  • Token usage, tool-call success, and full execution traces are what separate a debuggable agent from a black box that occasionally produces an answer and often produces a bill.
  • Distributed tracing extends to agents only once you accept that the GenAI semantic conventions it depends on are still being finalized — plan instrumentation around that instability, not against it.
  • LangSmith, Langfuse, and OpenTelemetry GenAI are three different bets — proprietary depth, OTel-native open source, vendor-neutral standard — the right pick depends on your architecture, not the loudest vendor.
  • Capturing a full trace means capturing every PII fragment the agent touched; masking is a design decision made before you turn tracing on, not a checkbox after.

How to read agent observability: from the trace model to the vendor map

Start with what traces, spans, and token attribution reveal about what agents actually do — it builds the vocabulary every later decision depends on: what a span records, what a trace strings together, and why token attribution turns a vague bill into a per-step ledger. Read the prerequisites and hard limits of observing agents in production right after — it is the honest version of the first article, mapping what distributed tracing assumes about determinism and where those assumptions still break for LLM calls.

When you are ready to instrument something real, the guide to LangSmith, Langfuse, and OpenTelemetry GenAI walks the setup and the architecture question each tool answers differently. For the market context behind that choice, the 2026 observability market after the ClickHouse and Datadog moves tracks which vendor philosophy you are actually buying into. Close with the privacy and ethics of capturing everything an agent sees before you ship tracing to production — the same instrumentation that debugs your agent also records your users.

MONA asks: 'My trace shows every span green — so why did the agent still ship the wrong refund?' MAX answers: 'A span records whether the call succeeded, not whether the decision was right — task success is a judgment your trace was never built to make.' — comic dialog.
Green spans mean the call worked. They don't mean the agent was right.

How agent observability differs from logging, error handling, and cost tracking

Three neighbours get folded into “observability,” and each folding sends the wrong team down the wrong hallway.

  • It is not classical APM. The distributed-tracing intuition carries over — spans, parent-child relationships, trace context propagation — but the semantic conventions for GenAI spans are still being written, and a span recording “the LLM call succeeded” is not the same claim as “the agent made the right decision.” That gap between call success and task correctness is exactly where the analogy to classical monitoring stops holding.
  • It is not error handling and recovery. Observability answers what happened; error handling answers what to do about it. A trace is diagnostic — it tells you a tool call failed three times before falling back — but nothing about tracing itself retries, falls back, or degrades gracefully. Teams that wire beautiful dashboards and no retry logic still lose the incident.
  • It is not cost optimization. Both lean on the same token-attribution data, but they read it for different jobs: observability records where the tokens went so you can debug a run; cost optimization consumes that same record to set budgets and route models. One trace, two owners.

Common questions about agent observability

Q: Do I need a dedicated agent observability platform, or does my existing APM tool already cover this? A: Classical APM was not built for non-deterministic, multi-step LLM calls, and the market has responded with dedicated tooling rather than APM add-ons — the 2026 shakeout after the ClickHouse and Datadog moves shows infrastructure vendors absorbing the telemetry layer specifically because general APM did not fit.

Q: Should I build on LangSmith, Langfuse, or OpenTelemetry GenAI for a new agent project? A: Pick on architecture, not on logos: the three represent proprietary depth, OTel-native open source, and a vendor-neutral standard respectively. The instrumentation guide walks the trade-offs before you commit a codebase to one.

Q: What should I mask or exclude before turning on full agent tracing in production? A: Assume the trace will contain PII the moment tracing captures tool inputs and outputs, because that is exactly what agents pass to tools. The privacy and ethics discussion walks through what “capture everything” actually captures before you ship it.

Q: Can I trust a fully green trace as proof my agent did the right thing? A: No — a span records whether a call completed, not whether the decision behind it was correct. The hard limits of observing agents in production explains why task-level correctness sits outside what current tracing conventions capture.

Part of the agent reliability & operations theme · closest neighbour: agent error handling and recovery. New to this from a software background? Start with the story: Agent Reliability for Engineers: What SRE Habits Map and Break.

1

Understand the Fundamentals

An AI agent without observability is a black box that occasionally produces an answer and frequently produces a bill. This step covers the core concepts — traces, spans, token attribution — that turn agent execution into something you can actually read.

Concepts covered

Agent Evaluation And TestingAgent GuardrailsHuman In The Loop For AgentsAgent Observability
2

Build with Agent Observability

Wiring up traces, evaluating tool calls, and choosing between LangSmith, Langfuse, Phoenix, or raw OpenTelemetry GenAI are practical decisions with real trade-offs. This step shows how to instrument an agent end-to-end without locking yourself into one vendor.

Tools & techniques

Agent ObservabilityAgent Evaluation And TestingAgent Error Handling And RecoveryAgent GuardrailsHuman In The Loop For Agents
4

Risks and Considerations

Recording every prompt, tool call, and intermediate output means recording every secret, PII fragment, and customer message the agent ever sees. This step examines the privacy, retention, and consent questions you inherit the moment you turn tracing on.

Risks & metrics

Agent ObservabilityAgent Evaluation And TestingAgent GuardrailsHuman In The Loop For Agents