What is LLM Evaluation?
How language model quality is measured, with benchmarks, metrics, judges and human review, and why a state-of-the-art model can still be a broken agent.
LLM evaluation is the practice of measuring how well a language model performs a task, its correctness, groundedness, safety and helpfulness, using benchmarks, automatic metrics, LLM-as-a-judge graders and human review. It scores the model in relative isolation, usually at development time. That is necessary but not sufficient once you ship an agent: a state-of-the-art model can still be a broken agent, because the agent is the model plus its tools, retrieval and multi-step actions.
- LLM evaluation measures a language model's quality, correctness, groundedness, safety, helpfulness, with benchmarks, automatic metrics, judge models and human review.
- It scores the model in relative isolation at development time: necessary before you build, not sufficient once you ship.
- You can have a state-of-the-art model and a broken agent: the agent is the model plus tools, retrieval and multi-step actions, and it needs its own evaluation.
- Use benchmarks to shortlist models, your own dataset to decide, and online evaluation to catch the drift offline scores never see.
- Fix conditions, run multiple trials, and report method and trial count: a score without them cannot be compared.
What is LLM evaluation?
LLM evaluation is the practice of measuring how well a large language model performs a task: its correctness, groundedness, safety and helpfulness, scored with benchmarks, automatic metrics, LLM-as-a-judge graders and human review. It answers the question every team asks before building on a model: is this model good enough at the kind of work we need it to do?
The defining property is isolation. LLM evaluation scores the model on its own, one prompt in, one output out, usually at development time, against curated datasets. That is exactly what makes it useful for choosing between models, and exactly what limits it once the model is embedded in a larger system.
The term is used interchangeably with 'LLM evals', and they are the same discipline: an eval is a repeatable, automated test that scores a model's output against an expectation, and LLM evaluation is the practice of running them.
LLM evaluation vs agent evaluation
LLM evaluation scores a model in relative isolation at development time. Agent evaluation scores the whole system in production: the model plus its tools, its retrieval, its memory and the multi-step actions it takes. The two sound similar, but they answer different questions at different layers.
The difference matters because of a fact every team shipping agents eventually runs into: you can have a state-of-the-art model and a broken agent. The model can top every leaderboard while the agent built on it picks the wrong tool, loses the thread three steps in, or grounds a claim in a document it never retrieved. No amount of model evaluation surfaces those failures, because they happen outside the model.
So treat LLM evaluation as necessary but not sufficient: it qualifies the engine, not the vehicle. Once your model is inside an agent, the scores that matter move to the agent layer, and that discipline has its own guide: see What is Agent Evaluation?.
What metrics and benchmarks measure LLM quality?
Public benchmarks cover the broad capabilities: MMLU and its successors for knowledge and reasoning, coding and maths suites for problem-solving, and long-context and instruction-following tests for practical use. They are how models are compared across the industry, and how a shortlist gets made.
Task-level metrics measure quality on your data. For retrieval-augmented systems, faithfulness (is every claim supported by the retrieved context) and answer relevance (did the response address the question) do most of the work. For tasks with a single right answer, exact match and accuracy still apply. For open-ended generation, rubric-based scores, correctness, completeness, tone, replace string matching, because there is no single correct string to match.
The practical rule: benchmarks tell you which model to start with; metrics on your own dataset tell you whether it works for your task. Both are LLM evaluation; neither is a substitute for the other.
Offline benchmarks vs online evaluation
Offline evaluation runs a model against a fixed dataset in a controlled setting: same inputs, known references, comparable scores. It is repeatable and cheap, which is why it dominates development: every prompt tweak and model swap can be scored against the same cases before anything ships.
Online evaluation scores real traffic after you ship: sampled outputs graded continuously, user feedback aggregated, drift tracked over time. It exists because offline scores age: providers ship model updates and user behaviour drifts, so a model that scored well in March can behave differently in August with no change in your code.
Mature teams run both, and the split is not model-versus-agent: it is before-versus-after. Offline evaluation gates what ships; online evaluation tells you what is actually happening, and feeds the failures it finds back into the offline dataset.
What methods are used to evaluate an LLM?
Three method families cover the field. Reference-based metrics compare the output to a known-good answer: exact match, accuracy, and overlap scores like BLEU and ROUGE. They are fast and objective but only work when a reference exists, which rules out most open-ended tasks.
LLM-as-a-judge uses a second model to score output against a rubric: correctness, groundedness, helpfulness, tone. It is how subjective quality gets measured at scale, thousands of outputs scored for the price of API calls, at the cost of needing calibration against human labels and controls for the judge's own biases. It has become the workhorse of modern LLM evaluation: see What is LLM-as-a-Judge?.
Human review is the ground truth the other two approximate. It is slow and expensive, so spend it where it counts: building the calibration set your judges are checked against, spot-checking scores on a schedule, and making the call on high-stakes outputs.
How do you evaluate an LLM?
Start by picking the measures that match your task: public benchmarks for a shortlist, plus a dataset of your own representative cases, real inputs with expected outcomes, for the decision that matters. Twenty real cases beat two hundred synthetic ones; they encode the weirdness of actual usage.
Then fix the conditions and run enough trials. Pin the model version, temperature and prompt, and run each case several times, because LLM output varies between runs and a single pass tells you little. Score the results with the method the task deserves: reference metrics where a right answer exists, a judge for open-ended quality, and human spot-checks to keep the judge honest. Report the method and trial count with the score: a number without them cannot be compared to anything.
Finally, know where this stops. Evaluating the model tells you what to build on. Once the model is wired to tools and retrieval and shipping as an agent, evaluation has to follow it into production, which is the handoff from LLM evaluation to agent evaluation.
What are the best LLM evaluation tools?
Open-source frameworks are the standard starting point: DeepEval brings a pytest-style workflow with a broad metric library, Ragas specialises in RAG metrics like faithfulness and context precision, and OpenAI Evals is a registry-driven harness. All three are strong for offline, developer-run evaluation and free to adopt.
Observability platforms with evals, Langfuse and LangSmith, attach scores to the traces they already capture, which makes them a natural fit once you have live traffic to sample. Eval-first platforms like Braintrust centre the workflow on datasets, experiments and judge calibration, and suit teams running large structured eval programmes before deploy.
One distinction to check before choosing: whether the tool scores model outputs, one prompt, one completion, or full agent sessions with tools and multi-step actions. Most were built for the former. LLM evaluation tells you the engine is good. Prefactor tells you the car gets people where they are going: continuous, judge-based and golden-dataset evals over multi-step agents in production, with per-agent quality and cost analytics.
LLM evaluation vs agent evaluation
| Dimension | LLM evaluation | Agent evaluation |
|---|---|---|
| What it scores | The model in isolation: one prompt, one output | The whole agent: model plus tools, retrieval and multi-step actions |
| When | Development time, before you ship | Continuously, in production |
| Typical signals | Benchmark scores, accuracy, faithfulness, rubric grades | Task completion, tool-call accuracy, groundedness, cost per session |
| Question answered | Is this model good at this task? | Is my agent doing its job right now? |
Evaluate the agent, not just the model, in production
Prefactor helps teams observe, evaluate, and improve their AI agents in production — across every framework and provider.
Book a demo →