+ P Observe, evaluate, and improve your Haystack agents
Capture every Haystack Pipeline as the directed graph it actually is — each Component's run as its own span, connected the same way Haystack connects component outputs to inputs.
What Prefactor records from Haystack
Haystack + Prefactor
Observe for Haystack
Prefactor observes your Haystack agents in real time — every LLM call, tool invocation, and custom span captured as stru
Open → EvaluateEvaluate for Haystack
Prefactor evaluates your Haystack agents — score outcome quality against the captured spans, track drift by comparing cu
Open → ObserveAct for Haystack
Prefactor acts on your Haystack agents at runtime — block, throttle, sandbox, or escalate a tool call or data access bef
Open →How the Haystack integration works
- A Haystack Pipeline is a directed multigraph of Components, each connected by which output feeds which input — Prefactor's spans follow that same graph, so a trace reads as the pipeline's real execution order, not a flat list of calls.
- Haystack's own tracing captures component input/output for debugging, disabled by default to avoid sending sensitive data to a backend; Prefactor's span capture is controlled separately via PREFACTOR_CAPTURE_INPUTS/_OUTPUTS, so the two don't need to be configured in lockstep.
- Beyond auto-captured spans, use withSpan to record any custom step you define — an API call, a quality check, a business action.
Haystack integration FAQ
Do I need a dedicated package for Haystack?
You can instrument Haystack today with the framework-agnostic prefactor-core SDK; a dedicated package can be added on request.
What does Prefactor capture from Haystack?
Prefactor records pipeline component runs, retrievals, agent tool calls and LLM calls as structured, timestamped spans — so every Haystack run is captured as trace data you can reconstruct, search and export end to end.
Does Prefactor add latency or change how Haystack runs?
No. Observability capture is designed to stay off your agent's critical path, so it doesn't alter your Haystack logic or your users' responses. The only part that acts inline is the optional runtime guardrails you enable per agent — by design, so a high-risk or low-confidence action can be held for human approval before it executes.
Can I evaluate agents built with Haystack and catch regressions?
Yes. Once runs are captured, eval suites score quality and groundedness on real traffic, drift detection flags behaviour changes after deployment, and versioned eval history catches regressions before they ship — the observe → evaluate → improve loop applied to your Haystack agents.
Related guides
See it on your Haystack agents
Book a 15-minute setup and our team gets you tracing in production.