AI Proof of Concept: How to Scope, Run and Judge One Before You Commit to a Full Build

Almost every AI conversation I have with founders and business owners starts the same way. They have seen a demo, they have an idea for their own business, and they want to know whether it will really work before they spend serious money. That is exactly what an AI proof of concept is for.

The trouble is that many proofs of concept end with a nice demo and no decision. The team says "it mostly works", nobody knows whether "mostly" is good enough, and the project drifts. At Creuto we treat a proof of concept as an experiment with a question, a test set and a pass mark agreed before anyone writes code. Here is how to run one that way.

When you need a proof of concept, and when you do not

A proof of concept is worth running when there is real uncertainty about whether AI can do the task. If the uncertainty is about whether users want the product, you need a prototype or an MVP instead. Our guide to MVP vs prototype vs proof of concept explains the difference.

• Run one when: the AI has to read your messy documents, handle your specific language or domain, make judgement calls, or reach an accuracy level you have not seen proven elsewhere.
• Skip it when: the task is well understood, such as summarising meeting notes or drafting simple emails, and the main risk is product fit or integration work.
• Question it when: the task needs an exactly correct answer every time with no human check. Ordinary rule-based software may be the better tool.

Step 1: Scope it to one question

The most common mistake is a proof of concept that tries to prove everything at once. Write a single sentence in this form:

"Can AI [do this task] on [this data] at [this quality level] for [this user]?"

For example: "Can AI pull the supplier name, invoice number, date and total from our scanned purchase invoices correctly often enough that staff only need to check the exceptions?" That sentence tells you the task, the data, the user and roughly how quality will be judged. Anything outside it, such as a dashboard, user accounts or a mobile app, waits for the real build.

Step 2: Set pass marks before you start

Decide what success looks like while you are still neutral. Once people see a demo, it becomes hard to say no. A short criteria table is enough:

The right pass marks depend on the task. Ticket tagging that a person reviews can live with more mistakes than reading medical or financial figures. The point is to write the numbers down first.

Step 3: Build a test set from real data

The test set is the most valuable thing a proof of concept produces. It is a collection of real inputs from your business, each paired with the answer a skilled person would give. Without it, you are judging the AI by feel.

• Use real examples, not clean samples written for the test. Include the blurry scans, the badly worded questions and the odd cases.
• Include the hard cases on purpose. If a fifth of your real inputs are messy, a fifth of your test set should be too.
• Have a domain expert write the correct answers. This is often the slowest part, so start it on day one.
• Remove or mask personal data that the test does not need, and agree in writing which provider will see the data.

Model providers publish guidance on this kind of testing. OpenAI's guide to evaluations describes how to score model outputs against expected answers, and the same idea applies whichever model you use.

Step 4: Try the simplest approach first

Start with the least complex approach that could pass, and add complexity only when the test results tell you to.

1. A well-written prompt to a hosted model. Often enough for drafting, sorting and simple extraction.
2. Retrieval over your documents. When answers must come from your own content and show their source. Our guide on how to reduce AI hallucinations covers why grounding matters.
3. Tool use or multiple steps. When the task needs the AI to look things up in your systems or follow a sequence.
4. Fine-tuning. Rarely the first step. Consider it only when the simpler options fall short and you have many good examples.

Run the full test set after every meaningful change and keep a simple log of the scores. That log shows whether you are improving or just moving errors around.

Step 5: Make the go or no-go decision

At the end, compare the results with the pass marks you set in step 2. There are only three honest outcomes:

• Go: it met the pass marks. Move to a proper build and a small pilot with real users.
• Change course: it came close, and the failures point to a fixable cause such as poor source documents or a narrower task.
• Stop: it missed clearly and the failures look built into the task. You have saved the cost of a full build.

Ask for a short written report, not only a demo. It should cover the question, the test set, the approaches tried, the scores, examples of failures and an estimate of what the real build needs. Our guide to AI app development cost explains what drives that next budget, and the project costs page shows how we price work.

What changes between the proof of concept and the product

A proof of concept skips things on purpose. The real build adds them back, and they are often most of the work:

• Security and permissions, so each user only sees what they are allowed to see.
• Human review screens for low-confidence or high-risk outputs.
• Logging and monitoring, so you can see quality and cost after launch.
• Integration with your existing software, data and sign-in.
• Ongoing testing, because models and your data change over time.

For a structured view of AI risk across this whole lifecycle, the US National Institute of Standards and Technology publishes the AI Risk Management Framework. It is a useful checklist even for small teams.

Red flags when a vendor runs your proof of concept

• No written question or pass marks before work starts.
• Testing only on examples the vendor chose or wrote.
• A demo with no scores and no failure examples.
• The test set and prompts stay with the vendor instead of in your accounts.
• Pressure to sign for a full build before the results are in.

At Creuto, everything we build for a client, including test sets and prompts, lives in the client's own accounts from day one. If you are comparing partners, our guide on how to choose an AI development company lists the questions to ask.

How we run proofs of concept at Creuto

We start with a free 30-minute call to understand the task and look at sample data. If a proof of concept makes sense, you get a clear written proposal within 24 hours of the call with the question, the pass marks and what you need to provide. If the idea passes, we move to a proper build, and the first working version usually takes about 4 weeks. You can read more about our work at creuto.com.