Both trace agents, run experiments on datasets, score production traffic and version prompts. Datadog is the natural pick when your services already report to Datadog and you want agent traces beside them; LangSmith is the pick for teams on LangChain or LangGraph, or anyone who needs a self-hosted deployment.[1][4]
Datadog Agent Observability, renamed from LLM Observability in June 2026, is a module of the Datadog platform that puts agent traces next to your APM and infrastructure data, while LangSmith is a standalone platform from LangChain built for the agent development loop. Datadog bills only on LLM spans and has no self-hosted option. LangSmith bills per seat and traces and can be self-hosted on an Enterprise contract.
| Decision factor | Datadog | LangSmith |
|---|---|---|
| Ownership | Datadog, public company (Nasdaq: DDOG); needs a Datadog account | LangChain, independent; $125M Series B in October 2025 |
| What it sits beside | Datadog APM, infrastructure and RUM data | LangChain, LangGraph, Deep Agents and LangSmith Deployment |
| Code evaluators in production | Run in your application and submitted by API; automatic only in Experiments | Run automatically through automation rules, offline and online |
| Blocking at run time | Not in Agent Observability; needs AI Guard, a separate product in limited availability | Partial: LLM Gateway (beta) caps spend, restricts models and redacts model calls |
| Self-hosting | None; Datadog-hosted only | Enterprise add-on on Kubernetes; BYOC also Enterprise |
| Free tier | 40K LLM spans a month, 15-day retention | Developer: 1 seat, 5k base traces a month, 14-day retention |
| First paid tier | Pro: $160 a month billed annually for 100K LLM spans | Plus: $39 per seat a month, 10k base traces |
| Capability | Datadog | LangSmith | Prefactor |
|---|---|---|---|
| Observe | |||
| LLM call tracing | ✓ | ✓ | ✓ |
| Multi-step agent tracing (tool calls, nested spans) | ✓ | ✓ | ✓ |
| Cost tracking | ✓ | ✓ | ✓ |
| OpenTelemetry support | ✓ | ✓ | ✓ |
| Evaluate | |||
| Offline evals (datasets, experiments) | ✓ | ✓ | ✓ |
| Online evals on production traffic | ✓ | ✓ | ✓ |
| LLM-as-a-judge scoring | ✓ | ✓ | ✓ |
| Code-based evaluators | ✓ | ✓ | ✓ |
| Human review and annotation | ✓ | ✓ | ✓ |
| Prompt versioning and playground | ✓ | ✓ | — |
| Act on findings | |||
| Runtime controls on live agent actions | Partial | Partial | ✓ |
| Gateway or proxy in the request path | — | — | — |
| Licence and deployment | |||
| Open source | Partial | Partial | Partial |
| Self-hosting | — | ✓ | Partial |
This is usually the deciding question. Agent Observability is one product inside the Datadog platform, and its main draw is correlation: an agent trace links to the backend services, hosts and front-end sessions around it. If an agent slows down because a database it calls is struggling, you see both in one place. Datadog renamed the product from LLM Observability to Agent Observability in June 2026; its documentation still sits under the old path.
LangSmith stands alone. It does not try to monitor your wider estate, and it goes further into the agent development loop: datasets built from traces, experiments, annotation queues, a Prompt and Context Hub with commit-based versioning, Engine for drafting fixes from production traces, and LangSmith Deployment for running LangGraph agents. Teams without Datadog lose nothing by choosing it, and teams on LangChain gain the deepest integration available.
The trade-off is reach versus depth. Datadog sees everything around the agent but treats agent evaluation as one module among many, and several of its newer agent features were still in Preview in mid-2026. LangSmith sees only the agent work, but every part of the product is aimed at it.
Both run LLM-as-judge evaluators on sampled production traffic and score experiments against versioned datasets. One difference matters for agents with deterministic checks. In Datadog, code evaluators run automatically in Experiments, but in production you run them in your own application and submit results with the SDK or HTTP API. LangSmith runs code evaluators online through its automation rules, including on multi-turn threads.
Datadog's judges run on your own provider account, such as OpenAI, Anthropic, Vertex AI or Bedrock, so judge cost appears on that bill. Datadog also recognises evaluators from DeepEval, Pydantic Evals and NVIDIA NeMo. Some Datadog features, including Patterns and parts of prompt versioning, were in Preview as of mid-2026.
Instrumentation is broad on both sides. Datadog auto-instruments the OpenAI Agents SDK, LangGraph, CrewAI, the Claude Agent SDK, Google ADK, Pydantic AI and MCP clients, and accepts OTLP directly. Its Java SDK auto-instruments only OpenAI, and its Go SDK is experimental. LangSmith ships SDKs for Python, TypeScript, Go and Java.
Agent Observability monitors and evaluates but does not block. Datadog sells blocking of prompts, responses and tool calls through AI Guard, a separate product billed per token scanned and in limited availability. LangSmith's LLM Gateway, in public beta since 30 July 2026, sits in the model-call path and can cap spend, restrict models and redact personal data, but it does not cover tool calls.
Datadog documents no self-hosted deployment, and the product is not available on Datadog's US government sites. Lapdog, its local trace viewer, is for development only. LangSmith offers cloud regions in the US, EU and APAC, plus self-hosted and BYOC deployments on Enterprise.
Default retention is close: 15 days on Datadog, with 30, 60 or 90 days as paid add-ons, and 14 days for LangSmith base traces, with extended retention up to 180 days.
As of 20 October 2026, Datadog bills only spans that call an LLM provider; tool, agent, retrieval and embedding spans are free. The free plan covers 40K LLM spans a month with 15-day retention. Pro is $160 a month billed annually for the first 100K LLM spans, then $3.50 per 10K, with longer retention as a paid add-on. LangSmith's Developer tier is free for one seat and 5k base traces, and Plus is $39 per seat a month with 10k base traces; the overage rate per trace is not published.
Datadog cost follows model-call volume and ignores team size; LangSmith cost follows seats as well as traces. Check Datadog's pricing and LangSmith's pricing against your own volumes.
We sell the layer this section describes. Read it with that in mind.
Stated plainly, since Prefactor publishes this page: Prefactor is an agent evaluation platform that scores each step of a run and acts on what it finds through runtime policies on tool calls and data access, approval holds for risky actions, and a kill switch for a run, an agent or a team. Datadog keeps blocking in AI Guard, a separate product, and LangSmith's gateway covers model calls only; neither documents approval holds. Prefactor has no prompt playground or prompt registry, is hosted, and does not monitor your wider infrastructure the way Datadog does. See the full comparison of 15 platforms.
Reviewed against public product and documentation pages on October 20, 2026. If a vendor has changed a feature, product name, or positioning since then, send a correction and we will update it. Numbered source links in the page body point to the ordered sources below.
Prefactor scores each step of every agent run and can hold or stop an action when a score says it should not happen. Start on the free tier: 5,000 spans a month, no credit card required.
Prefactor helps teams observe, evaluate, and improve their AI agents in production, across every framework and provider.