+ P Evaluate for Claude Agent SDK agents
Capture every Claude Agent SDK tool call through the same hooks the SDK itself exposes for permission control — PreToolUse and PostToolUse — not a separate interception layer.
Prefactor evaluates your Claude Agent SDK agents — score outcome quality against the captured spans, track drift by comparing custom spans across versions and environments, and classify every run's risk on two axes: data sensitivity and action consequence.
How to add evaluate to Claude Agent SDK
Install the SDK
Add prefactor-core to your environment.
Register & create spans
Register the agent instance and record spans around your Claude Agent SDK run.
Spans flow to Prefactor
Structured trace data lands in Prefactor as your agent runs.
# pip install prefactor-core
import os
from prefactor_core import PrefactorCoreClient, PrefactorCoreConfig
from prefactor_http import HttpClientConfig
config = PrefactorCoreConfig(
http_config=HttpClientConfig(
api_url="https://app.prefactorai.com",
api_token=os.environ["PREFACTOR_API_TOKEN"],
)
)
client = PrefactorCoreClient(config)
await client.initialize()
# then instrument your Claude Agent SDK run with spans — see docs.prefactor.aiShown with the Prefactor SDK — a first-class, working integration today.
How the Claude Agent SDK integration actually works
- Each agent turn, tool use, and sub-agent call is captured as a nested span.
- PreToolUse is a native permission-decision hook — it returns allow, deny, ask, or defer before the tool call executes; that's the real, built-in mechanism a Prefactor runtime policy runs through to block or hold a Claude Agent SDK tool call, not a workaround bolted onto the SDK.
- PostToolUse can replace a tool's output before Claude ever sees it (updatedToolOutput) — useful for redacting sensitive data out of a result after the tool ran but before it reaches the model.
- A native Claude Agent SDK package (@prefactor/claude) is available — verify its snippet against docs.prefactor.ai before switching this page to it.
What Prefactor captures for Claude Agent SDK agents
Outcome quality
Score agent outputs against your own criteria, task by task, using the continuous run history as the record to measure against.
Drift detection
Compare custom spans for the same operation across versions and environments — a model update or prompt change shows up as a difference between spans, before it shows up as a quality drop.
Risk classification
Every run scored on data sensitivity and action consequence, weighted into a Low / Medium / High / Critical classification.
Data tagging
Tag PII and other sensitive fields once in the schema; every run carrying that tag becomes searchable — the record enforcement acts on.
Trace anything — not just LLM calls
The SDK captures LLM, tool, and agent spans automatically. With withSpan you wrap any operation in your own span type — an API call, a database query, a quality check, a business action — each with its own payload and schema. It all flows into the same Observe, Evaluate, and cost views.
import { withSpan } from "@prefactor/core";
// Wrap ANY operation in a span you define — an API call, a quality
// check, a business action — with its own spanType, inputs and schema
await withSpan(
{
name: "research competitor",
spanType: "research_competitor",
inputs: { competitor },
},
async () => {
const results = await search(competitor); // your tool / API calls
return summarize(results); // captured as one span
},
);Nest spans to capture business-level actions, and start with permissive schemas you tighten over time. Instrumentation strategy →
An example run, span by span
Illustrative — a single Claude Agent SDK run as nested spans.
Illustrative example.
Manual logging vs DIY vs Prefactor
| Capability | Manual | DIY OpenTelemetry | Prefactor |
|---|---|---|---|
| LLM / tool / agent spans | Hand-rolled | ✓ build it | ✓ via the SDK |
| Token usage captured per call | — | Build it | ✓ |
| Configurable capture & sampling | — | Partial | ✓ |
| Hosted Admin UI (agents, instances) | — | — | ✓ |
| Risk profiles & audit trail | — | — | ✓ |
Claude Agent SDK evaluate — FAQ
Does evaluation reuse the same Claude Agent SDK integration?+
Yes — evaluation runs on the spans the SDK already records for Claude Agent SDK, so there is one integration, not a separate one for evaluation.
What is quality drift and how is it detected for Claude Agent SDK agents?+
Drift is a change in behaviour over time — after a model update, a prompt change, or a data shift. Because a custom span records the same Claude Agent SDK operation every time it runs, this week's spans can be compared directly against last week's, or against a different version or environment.
How is risk scored for Claude Agent SDK runs?+
On two axes — data sensitivity (what kind of data the run touched) and action consequence (what the agent did with it) — combined via configurable weights into a Low/Medium/High/Critical classification per run.
Can I tag and find PII across Claude Agent SDK runs?+
Yes — tag the fields that matter once in your agent's data schema, and every run carrying that tag becomes searchable across your whole Claude Agent SDK deployment, not just the run it was tagged in.
Do I need a dedicated package for Claude Agent SDK?+
Claude Agent SDK has a native package (@prefactor/claude); you can also instrument it today with the framework-agnostic prefactor-core SDK.
What does Prefactor capture from Claude Agent SDK?+
Prefactor records agent turns, tool use, sub-agent calls and LLM calls as structured, timestamped spans — so every Claude Agent SDK run is captured as trace data you can reconstruct, search and export end to end.
Does Prefactor add latency or change how Claude Agent SDK runs?+
No. Observability capture is designed to stay off your agent's critical path, so it doesn't alter your Claude Agent SDK logic or your users' responses. The only part that acts inline is the optional runtime guardrails you enable per agent — by design, so a high-risk or low-confidence action can be held for human approval before it executes.
Can I evaluate agents built with Claude Agent SDK and catch regressions?+
Yes. Once runs are captured, eval suites score quality and groundedness on real traffic, drift detection flags behaviour changes after deployment, and versioned eval history catches regressions before they ship — the observe → evaluate → improve loop applied to your Claude Agent SDK agents.
Keep going
Evaluate for other frameworks
LangChain →CrewAI →LangGraph →OpenAI Agents SDK →Microsoft AutoGen →LlamaIndex →See it on your Claude Agent SDK agents
Book a 15-minute setup and our team gets you evaluate in production.