Agent evaluation for legal teams shipping AI
An agent that drafts a memo or reviews a contract is touching privileged information and citing sources a partner will be held to, and most firms have no record of how it got there.
Every run scored for quality, performance and risk, and checked against the activity schema you define.
Every legal agent run gets watched, evaluated for quality and risk, and checked against the rules you set. One record answers what any agent did, with evidence ready for review.
What's at stake in legal
For the partners, associates, and engineering teams behind a drafting or research agent, the stakes are privileged client information the agent can reach and citations a partner is professionally responsible for.
Agents reach documents under privilege, and the firm needs a record of what each one read.
A fabricated or misapplied citation is real professional exposure, and it looks plausible on the page.
Few firms can show, for a given memo, where each citation actually came from.
One source of truth for your legal agents.
Instrument the agent frameworks you build on, and ingest the systems those agents touch as custom spans. Every run, score and signal lands in one place.
LlamaIndex
Claude Agent SDKSDKCore SDK · TS + PythonOTLPOpenTelemetry · closed toolsObserve. Evaluate. Act.
A record of what the agent did, and a check before it goes further. Each step below closes one of the problems above.
- Observe. Every document read recorded as its own span
- Evaluate. Each citation checked against the case actually retrieved
- Act. Written to an immutable audit trail, queryable per matter
- State bar ethics opinions on AI use
- ABA Model Rules (competence, confidentiality)
- GDPR (international clients)
- SOC 2
Built for everyone who owns legal AI.
The same record, read the way each team needs it.
Instrument the legal agents you already run — no gateway in the request path, no re-architecture.
For developers →Ship legal features without regressions — every run scored before a customer ever sees it.
For product teams →One portfolio view of every legal agent — its owner, cost and risk in a single place.
For heads of AI →Audit-ready evidence for every decision a legal agent makes, ready when a regulator asks.
Security & governance →Two agents legal teams are already shipping.
Hypothetical, but grounded in how legal teams deploy agents today.
Contract review agent
Low risk · 91/100A LlamaIndex agent reviews incoming NDAs and vendor contracts against a firm playbook, extracts key clauses, and flags deviations from standard terms for an associate to review before redlining.
- Each clause extraction and each flagged deviation recorded as its own span
- Validated against an activity schema that checks the extracted clause actually appears in the source document, rather than being paraphrased or invented
- Scored for quality and risk on every run
For any flagged deviation, an associate sees exactly which clause and document it came from, before it reaches a client.
Legal research memo agent
Medium risk · 86/100A Claude Agent SDK agent researches a legal question across a case law database, drafts a memo with citations, and routes it to an associate for review before it reaches a partner.
- Every case retrieved and every citation written into the memo captured as a span
- Each citation checked against its activity schema: whether the case actually exists, and whether the retrieved text supports the point the memo attributes to it
- A configurable risk profile flags memos with a high rate of unverified citations for closer review
The associate sees which citations were verified against the source case and which were not, before it goes to a partner.
Runs stay isolated — terminate one without touching the rest of your fleet via the kill switch → · Prefactor vs. observability tools →
The problems this solves
Agents fail quietly and the first signal is a complaint, not an alert.
How it gets caught →Stuck at POCThe pilot worked; sign-off takes months because risk has no evidence.
How it gets caught →No kill switchWhen an agent misbehaves, nothing can stop it short of stopping everything.
How it gets caught →See agent evaluation on your own legal agents
Book a demo and we'll walk through span-level scoring and audit evidence on a fleet like yours.
Unified performance platform for agents, authentication, and risk management