Both trace agents in production and judge what they do. Datadog Agent Observability is the natural choice if your services already report to Datadog, with experiments, judges and prompt versioning in the same place; Raindrop is a standalone, production-first monitor that finds recurring failures and compares changes on live traffic.[1][4]
Datadog Agent Observability, formerly LLM Observability, is a module of the Datadog platform that traces, evaluates and versions prompts for agents next to your APM and infrastructure data, while Raindrop is a standalone hosted monitor that classifies live agent traffic and groups recurring failures into issues. Datadog needs a Datadog account and has no self-hosted option. Raindrop needs no wider platform but has no prompt versioning and no hosted offline eval runner.
| Decision factor | Datadog | Raindrop |
|---|---|---|
| What it is | A module of the Datadog platform; renamed from LLM Observability in June 2026 | A standalone hosted platform for agents in production |
| Offline evals | Yes: experiments on versioned datasets, plus a Playground | Partial: Workshop runs evals locally; Simulations in early access |
| Production scoring | Custom LLM judges and preset evaluators on sampled spans | Signals on every new event; Issue Detection groups recurring failures |
| Prompt tooling | Versioned prompt registry with A/B tests of prompt versions | None documented |
| Run-time checks | Not in Agent Observability; AI Guard is a separate product in limited availability | None documented |
| Self-hosting | None; Datadog-hosted only | Beta: Enterprise VPC deployment for select partners |
| Free tier | 40K LLM spans a month, 15-day retention | Hobby: 1,000 events a month, 14-day retention |
| First paid tier | Pro: $160 a month billed annually for 100K LLM spans | Pro: $299 a month plus $0.003 per event |
| Capability | Datadog | Raindrop | Prefactor |
|---|---|---|---|
| Observe | |||
| LLM call tracing | ✓ | ✓ | ✓ |
| Multi-step agent tracing (tool calls, nested spans) | ✓ | ✓ | ✓ |
| Cost tracking | ✓ | ✓ | ✓ |
| OpenTelemetry support | ✓ | ✓ | ✓ |
| Evaluate | |||
| Offline evals (datasets, experiments) | ✓ | Partial | ✓ |
| Online evals on production traffic | ✓ | ✓ | ✓ |
| LLM-as-a-judge scoring | ✓ | Partial | ✓ |
| Code-based evaluators | ✓ | Partial | ✓ |
| Human review and annotation | ✓ | Partial | ✓ |
| Prompt versioning and playground | ✓ | — | — |
| Act on findings | |||
| Runtime controls on live agent actions | Partial | — | ✓ |
| Gateway or proxy in the request path | — | — | — |
| Licence and deployment | |||
| Open source | Partial | Partial | Partial |
| Self-hosting | — | Partial | Partial |
Datadog Agent Observability is one product inside the Datadog platform. Agent traces sit next to APM, infrastructure and real-user data, so a slow tool call can be traced back to the backend service behind it. That is the main reason to choose it, and it comes with a condition: you need a Datadog account, and there is no self-hosted version. It is also not available on Datadog's US government sites.
Raindrop is a standalone product. It does not need any wider observability platform and takes data from its own SDKs, an HTTP API or OpenTelemetry. If your services do not report to Datadog, adopting Datadog for agent traces alone is a larger decision than adding Raindrop.
Check language and framework coverage against your own code. Datadog auto-instruments common providers and frameworks, including the OpenAI Agents SDK, LangGraph, CrewAI and the Claude Agent SDK, but its Java SDK has a single auto-instrumented integration and its Go SDK is experimental. Raindrop has SDKs for TypeScript, Python and Go, with Rust and Java in beta, and a long list of framework integrations, most of them labelled beta. Both calculate model cost from public rates; Datadog covers text-based models only.
Datadog covers the full loop. Experiments run your application against versioned datasets, which can be built from production traces, and score each record with evaluators. Custom LLM judges run on your own connected provider account, at span, trace or session level. Code evaluators run automatically in experiments; in production you run them in your own code and submit the results. Annotation queues and a prompt registry with A/B testing round it out.
Raindrop covers the production half and covers it differently. You describe a behaviour in plain language and it drafts a classifier, a Signal, that runs on every new event and backfills a sampled 90 days. You cannot bring your own judge prompt or model. Issue Detection groups recurring failures, a Triage Agent investigates them in the web app, Slack or over MCP, and Experiments compare signal rates between cohorts of live traffic. For pre-release work it offers Workshop, an MIT-licensed local tool where a coding agent writes evals, and Simulations on pull requests, in early access.
Both now cluster production behaviour. Datadog announced Patterns in Preview at DASH in June 2026; Raindrop shipped the second version of Issue Detection with Raindrop 2.0 the same month.
Agent Observability monitors and evaluates; it does not block. Datadog sells blocking of prompts, responses and tool calls separately as AI Guard, a product in limited availability billed per token scanned. Raindrop documents no blocking, guardrails or approvals. Its PII Guard add-on redacts data at ingestion, which protects what Raindrop stores rather than controlling what the agent does, and its Simulations can block a merge in CI before anything reaches production.
The billing units differ in a way that matters for agents. As of 20 October 2026, Datadog bills only LLM spans, meaning calls to a model provider; tool, workflow, agent, retrieval and embedding spans are not billed. Free covers 40K LLM spans a month with 15-day retention. Pro is $160 a month billed annually ($200 month to month) for the first 100K LLM spans, then $3.50 per 10K, with longer retention as a paid add-on.
Raindrop bills events. Hobby is free for 1,000 events a month, after which ingestion stops until the next month. Pro is $299 a month plus $0.003 per event up to 1M events, then $0.002, with unlimited retention and a 14-day trial. An agent that makes many tool calls per model call will look cheaper on Datadog's unit; one with long retention needs will weigh Raindrop's unlimited retention. Compare against Datadog's pricing and Raindrop's plans.
We sell the layer this section describes. Read it with that in mind.
Stated plainly, since Prefactor publishes this page: Prefactor is an agent evaluation platform that scores each step of a run and acts on a failing score through runtime policies on tool calls and data access, approval holds for risky actions, and a kill switch for a run, an agent or a team. Datadog offers blocking through AI Guard, a separate product in limited availability; Raindrop documents no run-time controls. Prefactor has no prompt playground or prompt registry and its platform is hosted, and it does not correlate with your APM data the way Datadog does. See the full comparison of 15 platforms.
Reviewed against public product and documentation pages on October 20, 2026. If a vendor has changed a feature, product name, or positioning since then, send a correction and we will update it. Numbered source links in the page body point to the ordered sources below.
Prefactor scores each step of every agent run and can hold or stop an action when a score says it should not happen. Start on the free tier: 5,000 spans a month, no credit card required.
Prefactor helps teams observe, evaluate, and improve their AI agents in production, across every framework and provider.