How to evaluate LLM applications: test sets, judges, regressions

You evaluate an LLM application by running it against a fixed set of real and difficult inputs, grading each output with the cheapest method that can judge it (code checks first, then an LLM judge, then people), and repeating that run on every change so you can see which cases got better and which broke. The test set matters more than the grading tool, and the judge needs checking just like the application does.

Most teams skip this at first. They change a prompt, try five questions in a playground, and ship.

A small prompt change, an invisible break

Here is an illustration. A support team runs an assistant that drafts replies to customer emails. Someone notices the replies are too long, so they add one line to the prompt: "Keep replies under 120 words." They test it on six recent emails. All six replies are shorter and still polite. It ships on Friday.

On Monday a customer asks how to return a damaged item that was bought with a gift card. The old prompt produced a reply with all four steps of the return process and the line about refunds going back to store credit. The new one drops the store credit line because it ran out of words. Nobody notices for two weeks, until the refund complaints start.

Nothing about that change was careless. The six emails just didn't include a case where length and completeness pulled against each other. A test set of a hundred real emails, with a check for "mentions store credit when the payment method was a gift card", would have caught it before Friday afternoon.

Your own code isn't the only thing that changes, either. Lingjiao Chen, Matei Zaharia and James Zou compared the March 2023 and June 2023 versions of GPT-4 on the same tasks. Its accuracy at identifying prime numbers fell from 84% to 51%, and it made more formatting mistakes in generated code. Behaviour behind the same model name can shift, so anything built on top needs measuring again.

Read outputs before you pick metrics

The temptation is to start with a framework and a list of standard metrics. I'd start with a spreadsheet and fifty real outputs instead.

Hamel Husain, who has written one of the most practical guides on this, puts it bluntly: you must remove all friction from the process of looking at data. Open real inputs and outputs side by side, mark each one good or bad, and write a sentence about why. After an hour you will have a list of failure types specific to your application, which no generic metric would have given you.

There is research behind this habit. Shreya Shankar and colleagues studied people building evaluators for LLM pipelines and named a pattern they called criteria drift: users need criteria to grade outputs, but grading outputs is what helps them define the criteria. A rubric written before anyone looks at data tends to be the wrong rubric.

For the support assistant, an hour of reading might give you criteria like these: the reply answers the actual question, policy details match the policy document, it never promises a refund the policy doesn't allow, and it asks for the order number when one is missing.

Building the test set

A good test set looks like your real traffic, plus the cases that hurt. Anthropic's documentation says to design evals that mirror your real-world task distribution and not to forget edge cases, and lists the kinds worth including: irrelevant or nonexistent input, overly long input, poor or harmful user input, and ambiguous cases where humans would also disagree. OpenAI's guide makes the same split between typical, edge and adversarial cases.

The best source of typical cases is your logs. OpenAI's guide says to log everything so you can mine the logs for eval cases. Real users write messages you would never invent: half in Hindi, pasted with an email signature, asking two things at once.

Edge cases come from your failure list. Adversarial cases come from thinking like someone misusing the system. Our article on prompt injection has examples worth turning into test cases if your application reads email, documents or web pages.

How big should it be? Anthropic's guidance is to prioritise volume over quality: more questions with slightly noisier automated grading beat fewer questions with careful hand grading. In practice I'd start with whatever you can label in a day, often 50 to 150 cases, and grow it from production failures. Tag each case by type (refund, delivery, gift card, angry customer) so you can see results per slice later. An average across everything hides the slice that broke.

For each case, store the input, the context the application would see (retrieved documents, conversation so far), and a description of a good answer. "Must mention store credit; must not promise cash refund" is often more useful than a perfect reference reply, because the model will never match its wording exactly.

Keep the test set in your own repository, versioned like code. OpenAI's guide notes that its hosted Evals platform became read-only for existing users on 31 October 2026 and shuts down on 30 November 2026, a reminder that cases built inside one vendor's tool last only as long as the tool.

Three kinds of grader

Once you know what good looks like, you need something to check each output. There are three options, and the order you reach for them matters.

Use code wherever it can decide. Husain describes LLM unit tests as assertions like you would write in pytest, cheap enough to run on every commit. Is the output valid JSON? Does it contain the order number from the input? Did the agent call the refund tool with an amount at or below the order total? A surprising share of real failures can be caught with string matching and a few lines of logic, so if you can rewrite a criterion for code, do that before reaching for a judge.

Agents need one more layer: grade the outcome (was the right ticket filed with the right fields?) and the path (how many steps, which tools, any errors). Our guide to AI agents in production covers the agent side in more detail. CloudOEngine traces every agent step and tool call in its AI runtime, which gives these path checks something concrete to read: the actual tool calls and their arguments, not the model's description of what it did.

LLM-as-a-judge, and where it lies to you

For criteria that code can't check, such as "the reply is faithful to the returns policy" or "the tone is calm with an angry customer", the usual tool is a second model acting as a judge. It reads the input, the output and a rubric, and returns a verdict.

Lianmin Zheng and colleagues, in the 2023 paper that introduced MT-Bench, found that strong judges such as GPT-4 matched human preferences over 80% of the time, about the same level at which humans agree with each other. That is good enough to make judges worth using.

The same paper also listed the problems: position bias, verbosity bias, self-enhancement bias and limited reasoning ability. In plain terms, judges tend to prefer whichever answer they see first, prefer longer answers, and may favour output that looks like their own.

Position bias is larger than most people expect. Peiyi Wang and colleagues showed that with ChatGPT as the judge, Vicuna-13B could beat ChatGPT on 66 of 80 tested queries simply by changing the order the answers were presented in. Same answers, different winner.

So set the judge up defensively. These are the defaults I'd use, drawn from the OpenAI and Anthropic guides and the papers above:

• Ask for pass or fail on one criterion at a time instead of a 1 to 10 score across everything. OpenAI's guide recommends pairwise comparison or pass/fail for reliability.
• When comparing two versions, run every comparison twice with the order swapped, and count it as a tie if the verdict flips.
• Ask the judge to give its reasoning before the verdict, and keep that reasoning, because it's how you debug a wrong grade.
• Watch for length. If the judge's favourite version is always the longer one, check whether it is actually better.
• Use a different model as the judge from the one that produced the answer, which Anthropic's documentation calls generally best practice.
• Wrap the output being graded in clear tags, and define each criterion precisely, with an example of a pass and a fail.

Who checks the judge?

A judge is a model with a prompt, so treat it as one more component under test.

The method is simple. Take 50 to 100 outputs, have a person who knows the domain label each one pass or fail on a criterion, and run the judge on the same outputs. Then compare. Husain recommends tracking the correlation between model-based and human evaluation, and warns that raw agreement can mislead when classes are imbalanced. If 90% of outputs are fine, a judge that passes everything agrees with people 90% of the time and catches nothing. Look at precision and recall on the failures specifically.

Then read the disagreements. Sometimes the judge's prompt is unclear, sometimes the person was wrong, and sometimes the criterion was vague (criteria drift again). Fix and repeat until you'd act on the judge's verdict.

Keep a person on a small sample every week, because your traffic changes and so does the model behind the judge. The same idea shows up outside text: CloudOEngine's Vision AI counts people with two independent trackers and sends the cases where they disagree to a human review queue, so people spend their time on the cases a machine is least sure about.

Regression checks on every change

Once you have a test set and graders you trust, wire them to every change. That means changes to the prompt, the model or model version, the retrieval setup (chunking, embedding model, number of results), tool definitions, and the source data itself. Our RAG explainer covers why retrieval changes in particular can move answer quality a lot without anyone touching a prompt.

Compare results case by case with the last release, and treat the average as a summary only. As an illustration, an overall pass rate that goes from 91% to 92% can hide five cases that used to pass and now fail, offset by six that improved. Those five are your regressions, and they are often the ones users already depend on. Show them in a list next to every proposed change.

Decide rules before you look at results, so you're not tempted to argue a failure away. A workable set: code-check failures on safety or policy criteria block the release outright; any case that passed before and fails now needs a person to look at it; overall score changes smaller than the normal run-to-run noise are ignored.

The same prompt and input can give different outputs on different runs, so for cases that matter, run them three or more times and record the pass rate per case rather than a single result. A case that passes two times out of three is a different problem from one that always fails.

Husain describes a cadence I'd copy: assertions on every code change, model and human evaluation on a set schedule, and A/B tests only after significant product changes. As OpenAI's guide puts it, evaluation is a continuous process.

If the model sits behind a gateway, a model or version change becomes one configuration change you can test against the full set before switching traffic. CloudOEngine's AI runtime includes an LLM gateway, so applications call models through one place, and that one place is where you point a test run at a candidate model before switching.

The first week

If you have an LLM feature in production with no evals, this is the order I'd go in.

1. Export 100 recent real inputs and outputs from your logs. Read 50 of them and write down every failure type you see.
2. Turn each failure type into a criterion. Rewrite as many as you can into checks code can run.
3. Add 20 edge and adversarial cases on top of the real ones: empty input, very long input, other languages, attempts to change the assistant's instructions.
4. Write a pass/fail judge prompt for the criteria code can't check. Label 50 outputs by hand and compare the judge with your labels, focusing on the failures.
5. Run the full set against the current version and save the results as the baseline.
6. Add the run to your release process so that no prompt, model or retrieval change ships without a per-case comparison with the baseline.
7. Every time a user reports a bad answer, add that input to the test set the same day.

Step 7 is the one that compounds. After three months your test set describes how your application actually fails.

How CloudOEngine helps

CloudOEngine is one platform and console for connecting data (databases, files, camera streams), pipelines, and an AI runtime with an LLM gateway, agents with tool access, and RAG, under one permission model and one audit trail.

For evaluation, the most useful part is tracing. Agents run real tools, for example filing a ticket, and every step and tool call is traced and inspectable. When a user reports a bad answer, you can open that run, see what the agent read and which tools it called with which arguments, and turn it into a test case with the context it actually had.

The LLM gateway gives applications one place to call models, so a model change is one change you can test against your set before moving traffic.

Autopilot runs triggered agents with approval rules: a person approves an action, in the app or through an email approve link, before it runs. That is a safety net while eval coverage is still growing.

On the vision side, CloudOEngine's counting uses two independent trackers, routes disagreements to a human review queue where your staff check them, and ships a finalised count with a published accuracy report that shows every counted crossing.

Governance is included, not upsold: tenant isolation, role-based access, audit logs and per-tenant encrypted secrets. CloudOEngine is built in Bengaluru, India.

Book a demo or see the AI runtime.

Frequently asked questions

How do you evaluate an LLM application?

Build a test set from real inputs plus edge and adversarial cases, decide what a good answer looks like for each, and grade outputs with code checks, an LLM judge and a sample of human review. Run the same set on every change and compare each case against the last release.

How many test cases does an LLM eval need?

There is no fixed number. Anthropic's guidance favours more cases with slightly noisier automated grading over a few hand-graded ones, so start with the real examples you have, often a few dozen, and add every production failure as a new case.

Is LLM-as-a-judge reliable?

It can be useful but it is not neutral. The MT-Bench paper found strong judges agreed with human preferences over 80% of the time, about the level at which humans agree with each other, but also documented position, verbosity and self-enhancement biases. Check your judge against human labels before trusting its scores.

What is position bias in LLM judges?

It is when the judge's verdict changes depending on which answer it sees first. One 2023 study found Vicuna-13B could beat ChatGPT on 66 of 80 queries just by reordering responses when ChatGPT was the judge. Running each comparison in both orders is a simple fix.

What is an LLM regression test?

It is a rerun of a fixed test set after any change to the prompt, model, retrieval or tools, comparing results case by case with the previous version. It catches answers that used to be right and now are wrong, which an average score can hide.

Why do LLM evals need to run continuously?

Because the system changes even when your code does not. A 2023 study measured GPT-4's accuracy on one prime-number task falling from 84% in March to 51% in June, so a model version update alone can break behaviour you relied on.

Sources

1. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)
2. Large Language Models are not Fair Evaluators (Wang et al., 2023)
3. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (Shankar et al., 2024)
4. How is ChatGPT's behavior changing over time? (Chen, Zaharia and Zou, 2023)
5. Create strong empirical evaluations (Anthropic documentation)
6. Evaluation best practices (OpenAI documentation)
7. Your AI Product Needs Evals (Hamel Husain, 2024)

#llm evaluation #llm as a judge #regression testing #llmops #test sets