One call, one connected story — not three disconnected logs

The problem

A voice agent has two tiers — one that talks to the customer (read-only, can't do much damage) and one that acts on their behalf in the background (mutating, gated behind approval). Quality metrics are specific to the team — clarifying-question iterations, voice glitches, latency, hallucination — with no formal weighting between them today, cross-referenced manually against error monitoring and customer feedback about once a week. The call, the background agent, and whatever runs after the call ends get checked as three separate things instead of one connected story.

Agent Built on LiveKit + Ultravox + ElevenLabs

Call starts

Conversational layer — read-only, low risk

Background agent acts

Mutating actions, gated by approval

After-call workflow runs

Async, on a separate service

Risk profile applied

Prefactor Watching every step, in real time

Spans captured per layer

Business-logic intent, not raw tool calls

Risk profile diverges

Conversational vs. background agent, tracked over time

Quality scored per call

Iteration count, TTS glitches, latency, and hallucination — native evals against your own definitions

How it works in Prefactor

p95 and p99 are tracked per span type on more than latency — cost and quality too — so a degrading slice of calls shows up as a number before it shows up as a pattern of complaints.

Risk is tracked as a trend, per layer: the conversational agent scores low-risk (read-only); the background agent scores higher (it mutates data) — and both are tracked over time, so drift toward riskier behaviour in either layer shows up before an individual call looks alarming.

Business-logic-level spans capture what the agent was actually trying to do, which matters when the pipeline underneath changes but the intent doesn't.

Your quality definitions — iteration count, TTS glitch rate, latency, hallucination — plug in as native evals: Prefactor captures the run and attaches the result; you bring the scoring logic via API. The same pattern applies to LLM-as-judge.

One phone call, its background agent's actions, and the async after-call workflow — even across separate services — stay connected to the same instance.

Proactive alerts fire on a real pattern, like a spike in negative feedback, rather than a single thumbs-down.