Your AI is only as trustworthy as your 20 test cases

Can we trust what the AI tells us? Only once it has passed a test you wrote. That test is about 20 questions you already know the answers to, with a pass mark agreed before the first run, and it takes about a day of work to set up.

"I can't fully trust the answer AI gives me" comes up in nearly every audit interview I run.

An eval is that test. You give the AI a question you already know the answer to and mark what comes back. Twenty of them, run every time the AI changes, will tell you whether a change made it better or quietly broke something. Anthropic's own guide says to start with 20 to 50 cases drawn from real failures, not hundreds. Since 1 October our standard is a set like this, agreed with the client in week one.

What you need before you write your eval

• One person who knows the right answers: whoever gets asked these questions today.
• The documents the answers live in: contracts, job sheets, price list, staff handbook.
• A few weeks of what people have typed into the tool, or the questions they ask each other on Slack.
• Half an hour with whoever signs off on the tool, to agree the pass mark.

Step 1: Read 20 real conversations before you write anything

Spend 30 minutes reading 20 to 50 real things people have typed into the tool, and write one line on each about what went wrong. Don't write a scoring rubric first. You'll score what you expected to see and miss what actually happens.

Say it's a 40-person building company. Half an hour in, the failures sort into a handful of kinds: it answered about the wrong job, it quoted last year's price list, it made up a number when there was nothing on file. Those kinds become the test.

Step 2: Turn them into 20 AI eval cases with an answer key

A case has three parts: the question, worded the way someone on the team would actually type it; the right answer, or the document it must come from; and anything the AI must not say.

For the building company:

• Everyday questions. About half. "what are our payment terms for a new client?"
• Same thing, different name. The Smith job, Smith Group, the Dandenong site. Ask once each way; it has to land on the same job.
• Traps. "What did we quote Jones for the roof?" when you never quoted Jones. The right answer is "nothing on file". A made-up number is a fail. The other trap: last year's price list sitting next to this year's. The old one is a wrong answer and you write that down.
• Two-place questions. "How many days did the Harbour job run over, and what did it cost?" Dates from the schedule, cost from the invoices.

The answer key comes from the documents and the person, never from asking the AI and copying what it says, because then the test can only confirm what the AI already does. Two people who know the business should give the same pass or fail on every case; if they'd argue about one, rewrite it.

Every case is pass or fail, not a 1 to 5 score. Nobody can say what a 3 means, and a 3 never blocks anything.

Then type each question into normal document search. If the right document comes up first, the question is too easy: you're testing the filing, not the AI.

Step 3: Set the pass mark before the first run

Write the pass mark next to the cases before you see a result. Once results are on the screen, every number looks negotiable.

Set it per kind, not one score across all 20. A single number hides the failures that matter: the AI gets better at easy questions, worse at traps, and the total still goes up. For the building company: everyday questions 8 of 10, different names all 4, traps all 4, two-place both. Traps are all or nothing. If it invents a quote for a client you never quoted, that's a fail whatever the rest says.

The pass mark is a product decision, not a technical one. The client's person signs the answer key and we agree the marks with them. Then two rules for the life of the set. Never lower a mark to make a run go green. And a case you've decided to live with failing stays in, with the reason next to it. If it starts passing, that's a fail too, until someone finds out what changed.

Step 4: Find the cheapest model that passes

Run the same 20 on more than one model, cheapest first, using your real questions and documents rather than a benchmark table. The model that clears every pass mark at the lowest price is the one you build on. A dearer model has to pass something the cheaper one fails. Do this before you build, while changing your mind is free. Pin the exact version you tested, and read the answers yourself rather than trusting the marks alone.

Step 5: Run the AI eval on every change, and let a fail block the change

A test you run once at launch tells you about launch day. After that the model gets updated, someone edits a prompt, documents move. We run the 20 automatically on every change, and a fail blocks the change. Pin the documents to a fixed copy so a fail means the change, not a document someone edited that morning.

Our company brain searches two ways at once, by keyword and by meaning, and merges the lists. We tested a change that looked like a harmless speed-up: use only the first list. The overall score went up. The different-name cases and the traps failed, because those only work with both lists. One overall score would have shipped it. The per-kind marks blocked it before anyone typed a question in.

When a case fails, read the whole answer, not the score.

Step 6: Keep the AI eval honest

Most changes to our AI are now made by AI agents, and an agent that can read the test can pass it without getting better. Three things stop that.

Every bug becomes a case. When a user reports something wrong, the first thing written is a test case that reproduces it, not a fix. The fix can't merge until that case fails on the old version and passes on the new one. Skip the first half and you have a test that proves nothing.

A second AI argues against every change. Before a change merges, a different model from a different company reads it looking for ways it's wrong. It has to end with a plain verdict: clean, or findings. A review it couldn't finish counts as not clean, never as a pass.

A case that must fail. Next to the 20 real cases we run one with a deliberately wrong answer key: we tell it the answer to "what are our payment terms" lives in the holiday policy. That one has to fail every run. If it ever passes, the test has stopped checking anything and everything goes red. It's the test button on a smoke alarm.

How you know the eval is working

• A change gets blocked before a user sees it.
• A case goes stale because the business changed, not the AI. We moved every price onto one page and the test still pointed at the old page. The AI was right, the case was out of date. Fix the case and write down why.
• The set still fails something. If it passes 100% every run for a month, it's too small or too easy. A hard set at 70% tells you more than an easy one at 100%.

Where the eval does not work

• Twenty of the same kind. Twenty document-by-name questions tell you nothing about traps. Twenty spread across the kinds beat sixty of one shape.
• Judgement. The set checks facts: right document, refused when it should. It can't tell you whether the advice was any good.

If you're buying an AI tool, not building one

Write your 20 and agree the pass mark before the demo. Mark the answers against your records, not theirs. Run the same 20 after every update they ship. If it can't sit your exam, it isn't ready for your work.

The golden-set idea came to us from Mark Ajzenstadt at Limestone Digital. The checks we run on every change are the hg-eval-gates skill in hgstack, our open AI stack.