Open source · Apache-2.0

Evaluate your agents.
Without an LLM.

Deterministic checks that catch wrong tools, silent loops, and half-finished runs in your real traces. No judge models, no token cost, no config to start.

Install & run

# install from the repo (not on PyPI yet)
$ pip install "git+https://github.com/prefactordev/prefactor-open-evals.git#subdirectory=packages/python"
$ export PREFACTOR_API_URL=https://your-host PREFACTOR_API_TOKEN=your-admin-token
$ prefactor-evals-no-llm run --agent <your-agent-id> --since 7d

View on GitHub →Get started - free

Python & TypeScript · exits non-zero in CI · scores publish to your Prefactor instance

§01 / THE CHECKSset and go: no config

The checks

Generic checks. Any agent. Zero config.

Every check measures how the agent behaved, not what it does, so the same set works on a support bot, a voice agent, or a coding agent unchanged. Results are pass, fail, or skip, and skip is never counted as a pass.

core.errorsAny step that failed

core.loop_detectionThe agent stuck repeating itself

core.redundant_tool_callsThe same call twice, no progress

core.termination_stateRuns that ended cancelled or failed

core.latency_budgetA single step hanging

core.efficiencyFar more steps than this agent’s normal

core.conversation_lengthFar more turns than usual to finish

Five more optional checks are just as generic but need one value only you can supply, so they stay off until you set it: core.forbidden_actions, core.tool_arg_schema, core.output_schema, core.cost_budget, core.escalation_rule.

§02 / THE CLIusage: install, then run

Spin up the CLI

One command per run. Scorecard, JSON report, CI gate.

Export your PREFACTOR_API_URL and an admin PREFACTOR_API_TOKEN (there is no login step), then point it at any agent. It exits non-zero on failure, so it drops straight into CI.

CLI

# score the last week of runs (zero config)
prefactor-evals-no-llm run --agent support-bot --since 7d

# turn on the optional checks once configured
prefactor-evals-no-llm run --agent support-bot --pack advanced

# write scores back to Prefactor
prefactor-evals-no-llm run --agent support-bot --publish

# build a browsable HTML dashboard and open it
prefactor-evals-no-llm run --agent support-bot --html ./quality --open

scorecardIllustrative

$ prefactor-evals-no-llm run --agent support-bot --since 7d
  ✓ core.errors             pass
  ✓ core.loop_detection     pass
  ✓ core.termination_state  pass
  ! core.efficiency         flag  2.4× median steps
  42 runs · resolution 0.93 · 6 passed · 1 flagged
  report → ./agent-evals-report.json · exit 1

§03 / LIVE MODEgrade a run while it is still running

Watch & stop

Catch a loop on the offending call. Not after.

watch grades a run span by span while it is still going, and can stop one that loops or errors before the damage lands.

✓Terminate is never the default. Run watch with no --terminate and it only warns, touching nothing.
✓You name what can stop a run. Policy is per check: off, warn, or terminate.

watch

$ prefactor-evals-no-llm watch --agent support-bot --terminate loops,errors
  watching active runs · policy: loops=terminate errors=terminate
✓ run 5a1f · ok
■ run 5a20 · loop on tools:search → terminated

§04 / WITH PREFACTORscores that live next to the run

With Prefactor

Add `--publish`. The score lands next to the run.

✓Quality tab. Each finished run’s score writes into its instance, not a lost log.
✓Read-only by default. Publishing is opt-in and writes only the fields it owns. Free dev tier, 25k spans a month.
✓Browsable dashboard too. --html writes a self-contained page per run, no network, safe to share.

Get started - free →View docs

Prefactor · Quality tabIllustrative

0.93

RESOLUTION

RUNS SCORED

FLAGGED

EVAL COST

run 4f2a · complete11 spanspass

run 4f2b · complete9 spanspass

run 4f2c · complete27 spansflag

checks (7 core + 5 optional)

token cost

languages: Python & TS

config to start

§05 / QUESTIONSbefore you clone

Questions

Do I need an LLM or API key?

No. Every check is deterministic: no judge models, no token cost, byte-identical results every time. There is no model client in the package by design. LLM-as-judge evals, if they ship, are a separate package.

What license?

Apache-2.0. The checks are yours to use, fork, and extend, in Python or TypeScript.

Is it on PyPI or npm?

Not yet. Install straight from the repo. Both a Python and a TypeScript implementation live there, held byte-identical by a conformance suite, so you can pick whichever fits your stack.

Does it work without Prefactor?

Yes. All Prefactor coupling lives in one file. Point it at Prefactor for real traces, or swap that file for any trace source matching the published schema/v1 shape.

Get started

See it on your own agents.

Free for your first 25,000 spans a month. Instrument with the SDK, then run the evals against your own traces.

Get started - free →Read the docs

Evaluate your agents.Without an LLM.