What continuous evaluation gives you that point-in-time testing does not
Deploying an agent into production and running a benchmark suite before launch are two different problems. Benchmarks tell you how an agent behaves against a fixed set of inputs under controlled conditions. Production tells you how it behaves against the actual distribution of inputs, memory states, tool responses, and injected content your users and attackers send it every day.
According to S&P Global Market Intelligence and McKinsey data from mid-2026, 31% of enterprises run at least one AI agent in production, with banking and insurance leading at 47%. Yet only 47.1% of deployed agents are monitored, and 88% of organizations report confirmed or suspected AI agent security incidents. The gap between deployment rate and monitoring rate is where unexpected behavior lives.
This article covers three evaluation patterns that close that gap: behavior deviation detection, reasoning artifact surfacing, and adversarial prompt detection. Each pattern is designed to run continuously, not just at release time, and to produce signals a human reviewer can act on quickly.
Detecting deviation from expected behavior
Every agent has an implied contract: given a certain class of inputs, it should take actions within a bounded range. When it steps outside that range, you need to know before the action completes or before the trace disappears into a log file nobody reads.
The practical approach is to define that contract explicitly as an activity schema. An activity schema specifies which tools the agent is allowed to call, in what order, with what argument types, and within what latency and cost bounds. Each span the agent produces is then validated against the schema at runtime.
Goldman Sachs is deploying hundreds of autonomous agents, scaling toward thousands, across trade reconciliation, compliance, and client onboarding. At that scale, a single misconfigured agent silently drifting from its intended behavior pattern is not detectable by human spot-check. Schema validation run against every span gives you a machine-speed signal that a human can investigate rather than a haystack to search.
Deviation signals worth tracking include: tool calls the agent was never expected to make, argument values outside the distribution seen in staging, step sequences that skip mandatory intermediate actions, and latency spikes that indicate the agent is retrying or looping. Detecting agent drift through activity schema validation covers the mechanics of building those schemas and wiring them into your evaluation pipeline.
A complementary signal is quality decay over time. An agent that scores 94% on response quality at launch and 78% three weeks later has drifted, even if it has not violated any hard schema rule. Tracking quality scores per agent per cohort of inputs, rather than as a single aggregate, surfaces this kind of decay early. Detecting agent quality decay and production drift monitoring walks through how to structure those cohorts.
Surfacing reasoning artifacts for audit
Human-in-the-loop workflows only work if the human can understand what the agent was doing when it made a decision. That requires capturing more than the final output: it requires capturing the chain of reasoning steps, the tool call arguments, the retrieved context, and the model's internal rationale where the API exposes it.
JPMorgan Chase's COIN system reviews 1.2 million hours of legal work annually, and the bank is planning deployment of agents that operate autonomously for hours at a time. When a contract review surfaces a risk flag, the compliance team needs to see not just the flag but the clause the agent read, the rule it matched against, and any intermediate conclusions it drew. Without that trace, the human reviewer is approving or rejecting a black box output.
Structured span recording makes this tractable. Each span should carry: the input the agent received at that step, the tool or model call it made, the raw response, and a timestamp. When Prefactor instruments an agent, it records these spans and attaches quality scores and risk scores to each one, so a reviewer can filter to spans above a risk threshold rather than reading every trace from start to finish.
The audit trail problem is distinct from the debugging problem. Debugging asks "why did this fail?" Auditing asks "can I prove what happened and in what order?" Auditability and reproducibility in autonomous workflows covers the data retention and chain-of-custody requirements that make traces usable in a regulatory context. For financial services teams specifically, data retention for AI agents in regulated industries addresses the record-keeping obligations that apply when agents touch customer data.
A large financial services firm that moved from 15-day manual reporting cycles to an automated multi-agent pipeline reduced cost per report from $2,200 to $9 and cut error rates from 3 to 0.3 per report. The prerequisite for trusting those outputs was a reviewer workflow where each agent's reasoning was captured and inspectable, not just the final executive summary.
Catching adversarial prompts before they reprogram agent logic
Security researchers demonstrated the InjecMEM attack in August 2026, showing that hidden instructions can be injected into an AI agent's memory store and retrieved later to alter the agent's behavior, without the injection being visible in the agent's immediate context window. The attack works by poisoning memory at one point in time and having the agent act on the poisoned instructions in a later, unrelated session.
This is a qualitatively different threat from a prompt injection in a single turn. The agent does not look compromised when the injection happens. It looks compromised later, when it follows instructions nobody issued.
Detection requires evaluation at two points: at memory write time and at retrieval time. At write time, you can run a classifier against content being written to the agent's memory store to flag text that contains instruction-shaped patterns, role-override language, or references to the agent's own identity and constraints. At retrieval time, you can compare the agent's behavior in sessions that retrieved flagged memory chunks against a baseline of unflagged sessions.
Financial services organizations are actively shifting from fully autonomous agents toward human-in-the-loop models partly in response to exactly this class of risk. The shift does not eliminate the need for automated detection; it changes the stakes. If a human is reviewing agent outputs before they reach customers, an undetected injection means the human is reviewing outputs shaped by adversarial instructions they cannot see.
Securing AI agents beyond traditional access control covers the layered approach that combines access restrictions with behavioral monitoring. For teams working through the threat model systematically, how to analyze multi-agent AI attack surfaces maps out where injection vectors appear across orchestrator and subagent boundaries.
A detection rule that fires on every retrieved memory chunk that contains instruction-like content will produce false positives on legitimate tool documentation and user preferences. Calibrating the classifier against your specific agent's memory schema, rather than a generic prompt injection wordlist, reduces false positive rates to a manageable level. The signal should route to a review queue, not an automatic block, until you have enough labeled data to trust the precision.
Prefactor's risk scoring attaches to individual spans, which means a span that retrieves suspicious memory content gets a risk score independent of whether the agent's final output looks normal. That separation is what makes the detection useful: a skilled injection is designed to produce normal-looking output.
Where to start
Pick one of the three patterns above, instrument one agent for it, and run it for two weeks before adding the next. Schema validation is usually the fastest to implement because it requires only that you articulate what you already expect the agent to do. Measuring what agents actually do versus what you think they do is a good framing exercise before you write your first schema.
Start evaluating your agents and review the instrumentation reference in the docs.