How to Test and Evaluate an AI (LLM) Feature Before You Launch It

Most AI features look impressive in a demo. The hard part is knowing whether they are good enough for real customers, and whether the change you made yesterday made things better or quietly made them worse. That is what evaluation is for. Here is the practical process we follow.

Why AI features need a different kind of testing

Normal software gives the same output for the same input. A language model can give different wording each time, and small changes to a prompt or model can shift behaviour in ways nobody expected. So instead of checking for one exact answer, you check outputs against criteria: is it correct, is it grounded in the source, did it refuse what it should refuse, is it in the right format.

You still need ordinary tests for the code around the model (integrations, permissions, error handling). Our testing checklist covers those. This guide is about the AI part.

Step 1: define what good looks like

Write down, in plain language, what the feature must do and must never do. For a support assistant that answers from your help centre, it might be:

• Answers only from the help centre and links the source article.
• Says it does not know when the answer is not in the sources.
• Never gives refund decisions or legal advice; hands over to a person instead.
• Replies in the customer's language, in under a set number of words.

These become your scoring criteria. If you cannot write them down, the feature is not ready to evaluate, and probably not ready to build. An AI proof of concept is a good place to work them out.

Step 2: build an evaluation set

An evaluation set is a list of inputs with the answer or behaviour you expect. Use real questions wherever possible: support tickets, search logs, sales emails. Cover four kinds of case:

A few dozen good examples beat hundreds of easy ones. Keep the set in version control next to the code, and add every real failure you find in production.

Step 3: choose how to score

Most teams combine them: rule checks on every case, model-graded scoring for volume, and human review on a sample and on anything the model grader flags.

Step 4: decide what to measure

• Correctness: is the answer right?
• Faithfulness: is it supported by the sources it was given? This matters most for RAG features.
• Refusal: did it decline or hand over when it should?
• Safety: did it resist attempts to make it break its rules? See AI security risks.
• Format and tone: does it follow the required structure and voice?
• Speed and cost: response time and cost per request, measured on the same set.

Step 5: set a pass threshold before launch

Agree the bar before you see the results, so you are not tempted to move it. For example: no failures on the "must never" cases, and a minimum share of correct answers on the common and hard cases. Where mistakes are costly, add a human check before the answer reaches the customer, at least at first. Our guide on reducing AI hallucinations explains the design options.

Step 6: run it on every change

Treat the evaluation set like a test suite. Run it whenever you change the prompt, switch or upgrade the model, change how documents are retrieved, or update the source data. Compare scores with the previous version and look at the specific cases that changed, not just the total. A small drop in a "must never" category matters more than a gain elsewhere.

Step 7: monitor after launch

• Log inputs and outputs (with personal data handled properly) so you can review real behaviour.
• Give users a simple way to flag a bad answer.
• Review a sample of conversations on a regular schedule.
• Track cost, response time and the rate of hand-overs to people.
• Add real failures to the evaluation set so they are tested from then on.

Common mistakes

• Judging the feature from a handful of demo questions.
• Building the evaluation set from easy, made-up questions instead of real ones.
• Changing the prompt and the model at the same time, so you cannot tell which helped.
• Trusting model-graded scores without ever checking them against people.
• Stopping evaluation after launch.

How we approach it

At Creuto, an OpenAI Select Partner, we build an evaluation set alongside the first version of every AI feature, agree pass criteria with the client before launch, and keep it running afterwards. If you are building an AI chatbot, a RAG system or an AI agent, book a free 30-minute call and we can look at how to measure it.