Problems · in production

The pilot worked in weeks. The sign-off is taking months.

The pilot proved value in six weeks; production approval is measured in months, because risk has nothing to review.

approval-timeline · pilot to production example
Illustrative pilot, before and after it carries its own record
Pilot proved value week 6
Months since, still in review 9
Risk questions outstanding 14
Evidence of what each run did none → per-run record
Time to answer "what did it access?" weeks → minutes
the pilot walks into review carrying its record, mapped to the frameworks risk asks about
§01 / THE SYMPTOM you see: the signals
TL;DR

Pilots stall because risk has no evidence to sign off on. Give every run a record and a verdict, with holds for runs that breach the rules, and the review has something concrete to approve.

The symptom

What stalled looks like

The technology question was answered months ago. The approval question is still open, and it is the one that decides whether the agent ever ships.

01
Approval measured in quarters

Teams describe six months per review round, production sign-off taking up to a year, and structured adoption programmes running 12 to 24 months. The build took six weeks; the queue takes the rest.

02
Pilots die of funding drift

Production is so far away that a budget reshuffle or a reorganisation kills the POC before the sign-off arrives. The agent did not fail; the calendar did.

03
The same questions, every round

What did it access, what happens when it goes wrong, who gets alerted. Each committee asks again, and each time the answers are rebuilt by hand because nothing records them.

04
Evidence-gathering becomes its own project

SOC 2, ISO 27001, the EU AI Act, and sector regimes each want documentation of what the agent does and how it is controlled. Assembling that manually takes longer than the pilot did.

§02 / WHY IT HAPPENS cause: not carelessness
Why it happens

Why sign-off stalls

Risk teams are not being obstructive. They are being asked to approve something that arrives with no evidence and no controls.

01
Risk has nothing to read

A demo shows the agent working once. It says nothing about run 4,000, what the agent will and will not do, or how often it fails. Reviewers cannot sign off on the absence of evidence, so they ask for more time instead.

02
No runtime control to point at

The reviewer asks what stops a bad run before it acts. If the honest answer is nothing, the review cannot close, because approval would mean accepting whatever the agent does next.

03
The pilot was built to be disposable

The POC existed to prove value fast, so it kept no record of its own runs. Everything the review needs, what it accessed, how it failed, who was told, has to be retrofitted after the fact.

04
Frameworks assume software, not agents

Compliance controls were written for deterministic systems and human operators. Mapping an agent's behaviour onto them is translation work nobody owns, so it lands on whichever team wants the approval most.

§03 / HOW YOU CATCH IT loop: observe → evaluate
How you catch it

How the record answers the review

Prefactor watches every run and evaluates the outcome, so the evidence risk keeps asking for accumulates from the first week of the pilot.

Connect

Instrument the pilot itself. Native SDKs for common frameworks, a TypeScript and Python core SDK for anything custom, and OpenTelemetry ingest for closed tools. Evidence starts accumulating in week one, not after the review requests it.

Observe

Every run in one queryable record. What the agent accessed, decided, and changed, per run, tagged to the agent that did it. "What did it access?" becomes a lookup instead of a reconstruction.

Evaluate

Every run gets a verdict. Each run is checked against the agent's job, so failure rates, failure types, and quality per version are numbers a reviewer can read, not assurances they have to take on trust.

Map

Evidence lines up with the frameworks. The record maps to the controls in SOC 2, ISO 27001, the EU AI Act, and sector regimes, so the review reads reports rather than commissioning an evidence project.

§04 / HOW YOU FIX IT loop: act → improve
How you fix it

From demo to sign-off

The record answers what the agent did. The controls answer what happens when it goes wrong, which is the other half of every review.

Act

Holds and escalation exist before the review asks. A run that fails its evaluation can be held or routed to a person before it acts. When the reviewer asks what stops a bad run, the answer is a mechanism you can demonstrate, not a plan to build one.

Improve

Each review round starts from the last one. The questions risk asked become saved queries against the record, so the next committee opens with answers instead of restarting discovery, and each round gets shorter rather than longer.

Prove

The pilot carries its record into review. What it accessed, how often it failed and how each failure was handled, who was alerted and when. Sign-off becomes a review of evidence rather than an act of faith, which is the version risk teams can actually approve.

A pilot proved value in six weeks, then spent two quarters restating the same answers about access and failure handling for each committee, and was nearly cancelled when its sponsor changed roles. Instrumented from the start, the same pilot arrives in review with a per-run record, evaluation results across a few thousand runs, and a hold path the reviewer can watch fire. The committee's questions become lookups, and the open item list shrinks instead of rolling over. Illustrative, but the questions are the standard ones.

§06 / QUESTIONS faq: the common ones
Questions
Why do AI agent pilots never reach production?
Most stall at risk and compliance sign-off, not on the technology. Teams describe six months per approval round and up to a year for production sign-off, and pilots die when funding shifts in the meantime. The underlying cause is usually evidential: reviewers are asked to approve an agent with no record of what it does and no runtime controls to point at.
What evidence does a risk team need to approve an AI agent?
The recurring questions are what the agent accessed, what happens when it goes wrong, and who gets alerted. A queryable per-run record answers the first, evaluation results with failure rates answer the second, and demonstrable holds and escalation answer the third. That set, mapped to the frameworks the team works from, is what a review can close on.
How does Prefactor help with SOC 2, ISO 27001, or the EU AI Act?
It supplies the runtime evidence those frameworks ask about: what each agent did per run, how each run was evaluated, and how failures were handled, mapped to the relevant controls. It does not replace your compliance programme; it removes the part where gathering agent evidence becomes its own project.
Where does Prefactor's data live?
Residency is usually one of the review's first questions. Prefactor's primary infrastructure runs in Australia; for enterprise engagements it deploys where your data needs to live, in your region or your environment.
Can we add this to a pilot that is already running?
Yes. Native SDKs, a TypeScript and Python core SDK, and OpenTelemetry ingest instrument a running pilot without a rebuild. The record accumulates from the day you connect; runs before that stay unrecorded, which is a reason to instrument during the pilot rather than after the review asks.

See it in action on a fleet like yours

Book a demo and we will show what a pilot looks like when every run arrives in review already watched, evaluated, and answerable.

Agent Performance Platform
Unified performance platform for agents, authentication, and risk management
All Systems Operational
3Global Agents
7Instances
5Services
12%Human Intervene
4High Risk
$2,360Monthly Spend
Mission ControlLive agent health with 7-day activity heartbeat
Claims Proc...68
$330/moRed
Claims Proc...65
$160/moRed
Claims Proc...82
$170/moAmber
ChatGPT74
$150/moAmber

See how every agent performs, and make it better

Prefactor helps teams observe, evaluate, and improve their AI agents in production, across every framework and provider.