Both trace agents, run experiments and score production traffic with LLM judges and code. Datadog fits teams whose services already report to Datadog and who want prompt versioning in the same place; Fiddler fits teams that also monitor classic ML models, need on-premises deployment or want content guardrails from the same vendor.[1][4]
Datadog Agent Observability, formerly LLM Observability, is a hosted module that puts LLM and agent traces beside Datadog's APM and infrastructure data, while Fiddler is a standalone platform that monitors classic ML models, LLM applications and agents and adds real-time content guardrails. Fiddler can be deployed in a VPC or on premises on Enterprise; Datadog is hosted only. Datadog has prompt versioning and a playground; Fiddler has neither.
| Decision factor | Datadog | Fiddler |
|---|---|---|
| Ownership | Datadog, a public company (Nasdaq: DDOG); needs a Datadog account | Independent; $30M Series C in January 2026, $100M raised in total |
| Scope | LLM and agent traces, correlated with APM, infrastructure and RUM | Classic ML models, LLM applications and agents |
| Run-time checks | Separate product: AI Guard, Limited Availability, billed per token scanned | Guardrails score content in real time; inline blocking through a third-party gateway |
| Prompt tooling | Prompt registry with per-environment deployment, A/B tests and a Playground | None |
| Human review | Annotation Queues | Span annotations in public preview; no review queue |
| Self-hosting | None documented | VPC or on premises on Enterprise; also as a SageMaker Partner AI App |
| Free tier | Free: 40K LLM spans a month, 15-day retention | Guardrails only, through a demo request |
| First paid tier | Pro: $160 a month billed annually, 100K LLM spans | Developer: $0.002 per trace, SaaS only |
| Capability | Datadog | Fiddler | Prefactor |
|---|---|---|---|
| Observe | |||
| LLM call tracing | ✓ | ✓ | ✓ |
| Multi-step agent tracing (tool calls, nested spans) | ✓ | ✓ | ✓ |
| Cost tracking | ✓ | Partial | ✓ |
| OpenTelemetry support | ✓ | ✓ | ✓ |
| Evaluate | |||
| Offline evals (datasets, experiments) | ✓ | ✓ | ✓ |
| Online evals on production traffic | ✓ | ✓ | ✓ |
| LLM-as-a-judge scoring | ✓ | ✓ | ✓ |
| Code-based evaluators | ✓ | ✓ | ✓ |
| Human review and annotation | ✓ | Partial | ✓ |
| Prompt versioning and playground | ✓ | — | — |
| Act on findings | |||
| Runtime controls on live agent actions | Partial | Partial | ✓ |
| Gateway or proxy in the request path | — | — | — |
| Licence and deployment | |||
| Open source | Partial | Partial | Partial |
| Self-hosting | — | ✓ | Partial |
Fiddler started in classic machine learning and still covers it: drift, performance, explainability and fairness for traditional models, alongside LLM monitoring and agent observability. If one team is responsible for both a credit model and a customer-facing agent, Fiddler is the one product here that covers both. It is an independent company that raised a $30M Series C in January 2026.
Datadog Agent Observability, renamed from LLM Observability in June 2026, is narrower in model scope and wider in system scope. It covers LLM applications and agents only, but links each trace to the APM, infrastructure and RUM data Datadog already collects. If your services report to Datadog, an agent failure can be followed into the backend calls behind it. Fiddler lists Datadog among its integrations, so the two are not exclusive.
This is the clearest technical difference. Fiddler Guardrails score requests and responses in real time for harmful content, faithfulness, personal and health data, and secrets, and return the result for your application to act on. Blocking or redacting inline, without code changes, works when Fiddler is plugged into a third-party AI gateway: Kong, AgentGateway, LiteLLM or Portkey. The checks are on content; Fiddler documents no approval step or policy on agent tool calls.
Datadog Agent Observability monitors and evaluates but does not block. Blocking prompts, responses and tool calls is the job of AI Guard, a separate Datadog product billed per token scanned and in Limited Availability as of mid-2026. If you need run-time checks today, Fiddler's are part of the same platform; Datadog's need a second product and its availability terms.
Both run offline experiments on datasets and score live traffic. Fiddler's Evaluator Rules run evaluators on new production spans that match filters, with optional backfill and downsampling to limit judge cost. Its judges can run on Fiddler-hosted models, such as a Ministral 3 8B model made generally available in September 2026, or on your own provider keys. Datadog runs LLM-as-a-judge evaluators on sampled production spans using your own provider account; its code evaluators run automatically only in Experiments, and in production you submit their scores from your application.
The tooling around evaluation differs more. Datadog includes a versioned prompt registry with per-environment deployment, prompt A/B tests, a Playground and Annotation Queues for structured human review. Fiddler has no prompt versioning or playground, and its span annotations, in public preview since August 2026, have no review queue. Fiddler's native agent SDKs are Python only (LangGraph, LangChain, Strands and Google ADK); other languages use OpenTelemetry or the REST API.
Datadog is hosted only and not available on its US government sites. Fiddler's Enterprise plan adds VPC and on-premises deployment, and Fiddler can also run inside your AWS account as an Amazon SageMaker Partner AI App. Its self-serve Developer plan is SaaS only.
Cost tracking also differs. Datadog calculates cost from token counts for over 800 text-based models, with custom values for others. Fiddler tracks tokens and stores monetary cost when the trace source emits cost attributes, for example from LiteLLM. Fiddler SDK users should also note that Python 3.10 support ends on 27 October 2026.
As of 20 October 2026, Datadog's Free plan covers 40K LLM spans a month with 15-day retention, and Pro is $160 a month billed annually for the first 100K LLM spans, then $3.50 per 10K; only spans that call an LLM provider are billed. Fiddler's Developer plan is $0.002 per trace, so 100,000 traces a month would be $200. Its free plan covers guardrails only and starts with a demo request, and Enterprise is quote-based. A trace and an LLM span are different units, so compare on your own traffic using Datadog's pricing and Fiddler's pricing.
We sell the layer this section describes. Read it with that in mind.
Stated plainly, since Prefactor publishes this page: Prefactor is an agent evaluation platform that scores each step of a run and acts on what it finds through runtime policies on tool calls and data access, approval holds for risky actions, and a kill switch for a run, an agent or a team. Fiddler's guardrails check content in real time but have no approvals or tool-call policies, and Datadog's blocking sits in AI Guard, a separate product. Prefactor has no prompt playground or prompt registry, does not monitor classic ML models and is hosted, so for those needs Datadog or Fiddler is the better fit. See the full comparison of 15 platforms.
Reviewed against public product and documentation pages on October 20, 2026. If a vendor has changed a feature, product name, or positioning since then, send a correction and we will update it. Numbered source links in the page body point to the ordered sources below.
Prefactor scores each step of every agent run and can hold or stop an action when a score says it should not happen. Start on the free tier: 5,000 spans a month, no credit card required.
Prefactor helps teams observe, evaluate, and improve their AI agents in production, across every framework and provider.