The cost of a routed ticket nobody questioned
A support agent classifies an incoming ticket as `queue = "billing"` with probability 0.91. The type is correct. The field is a valid string from the allowed enum. The schema validator passes. The ticket moves to the billing queue, where it sits for two days before a human notices the customer actually needed a refund reversal, not a payment query.
No error was logged. No alert fired. The agent did exactly what it was designed to do, structurally speaking.
This is the problem Jev solves only partway. Jev guarantees that its output is a valid instance of the type you asked for. That is a parsing guarantee. It tells you the agent produced a well-formed object. It says nothing about whether that object is the right answer for this customer, this context, this moment.
If you are replacing LLM classification steps with Jev, you are buying real value: you eliminate malformed outputs, you get deterministic structure, you reduce the retry logic required to coerce unpredictable text into usable data. Those gains are real and bounded. What Jev does not give you is a correctness signal.
What type safety cannot see
Consider a concrete support-triage agent. A customer emails to say they were charged for a full annual subscription when they only wanted one month. The agent:
- Classifies the intent as `intent = "billing_dispute"`, probability 0.88.
- Routes to `queue = "billing"`, probability 0.91.
- Sets `refund_eligible = true` based on a policy retrieval step.
- Sets `refund_amount = 708.00` rather than `59.00` because it read the annual price from the product catalogue instead of the customer's actual charge.
Every field is correctly typed. The schema is valid. The agent completed without an exception. A customer-support agent of this kind has no structural failure mode to report. The only failure is semantic: the wrong dollar amount, the wrong queue priority, the wrong policy applied.
This exact pattern appeared in a documented case where an AI support agent issued a refund of $708 when the customer had requested $59, the difference between an annual and a monthly subscription. The agent was shut down entirely after the incident. The type was never the problem. The behaviour was.
Air Canada's chatbot case followed the same logic at a different layer: the agent produced confident, well-formed responses describing a bereavement fare discount policy that did not exist. A tribunal held the airline liable for the fabricated policy. The outputs were grammatically and structurally coherent. They were also wrong, and consequentially wrong.
Why the misroute never surfaces
In a typed pipeline, the agent's output passes through three checkpoints: schema validation, type coercion, and downstream consumption. All three pass when the output is well-formed. None of them ask whether the value was appropriate.
The silent misroute is a failure mode that production observability alone does not catch. Watching spans tells you that the agent ran, that it called the right tools in sequence, and that it returned in 340 milliseconds. It does not tell you that the `refund_amount` field was wrong for this customer's billing history, or that the `queue` assignment routed a time-sensitive refund into a two-day backlog.
Observability and evaluation are not the same thing. Observability records what happened. Evaluation asks whether what happened was correct. You need both, and the second one requires ground truth or a proxy for it.
In the support-triage case, the ground truth is available after resolution: did the customer receive the correct refund amount? Was the ticket resolved within the SLA appropriate to its actual priority? Was the policy cited in the agent's response the one actually on record at the time? These are answerable questions. They just require a deliberate evaluation step that a type check cannot substitute for.
What behaviour evals actually measure
A behaviour eval for this agent would score each run against the actual outcome. For a refund ticket, that means:
- Routing accuracy: was the queue assignment consistent with how a human reviewer would have classified this ticket?
- Amount correctness: did the `refund_amount` match the customer's actual charge for the period in dispute?
- Policy grounding: was the policy cited present in the retrieved context, or was it inferred?
- Resolution outcome: was the ticket resolved without escalation or correction?
Evaluating runs against actual outcomes requires attaching a scorer to each span that compares the agent's decision to the ground truth label, whether that label comes from a human reviewer, a downstream system event, or a validated reference dataset.
A deployment handling 58% of refund tickets autonomously, as one IrisAgent deployment reached by week six, is processing a significant volume without human review. At that scale, a systematic classification error does not produce one wrong refund. It produces a pattern of wrong refunds, none of which appear in your error logs.
Quality issues, including accuracy, relevance, and consistency, are the top barrier to production agent deployment for 32% of surveyed professionals, based on a survey of 1,340 respondents in late 2025. That figure suggests the field already knows type safety is not enough. The gap is in what to measure instead.
How Prefactor fits into this
Prefactor records spans for each agent run and attaches scores to them. For a support-triage agent, that means logging the classification decision, the tool calls made during policy retrieval, and the fields written to the output object. It then scores those decisions against configurable criteria: routing accuracy relative to a labelled hold-out set, refund amount within a tolerance of the customer's actual charge, policy citations present in the retrieved context.
Validating behaviour against an activity schema means you can define what a correct run looks like for this agent, not just what a structurally valid run looks like. When the agent routes a refund ticket to the billing queue instead of the refunds queue three times in a row, the schema validator flags a drift in routing behaviour before it becomes a pattern visible in customer complaints.
Detecting that drift early is what separates an agent you can confidently keep in production from one you quietly pull back after the sixth misfired refund.
The audit trail Prefactor keeps also matters for accountability. In regulated industries, knowing what decision the agent made and on what basis is not optional. A typed output tells you what the agent returned. An audit trail tells you why, which tool calls preceded it, which policy was retrieved, and what probability the model assigned.
57.3% of surveyed professionals reported having AI agents in production as of 2026, up from 51% the year before. More agents in production means more silent misroutes accumulating in systems that check types but not behaviour.
Where to start
Run your support-triage agent against a labelled sample of resolved tickets and score each classification decision against the actual resolution outcome. That tells you your routing accuracy baseline. Then set up span-level scoring so new runs are evaluated against the same criteria in production.
Start evaluating your agents or read through the docs to see how to attach scorers to spans for classification agents.