If you are building an AI system that answers from your own documents, you need some way to check whether it found the right ones. That sounds simple until the test itself starts giving you the wrong answer. Here is what happened when I nearly rewrote my retriever over a problem that was partly in my own answer key.
The score that made me want to rewrite the retriever
The evaluation came back and said the retriever had found the right evidence for about 1 question in 16.
That is bad enough that you stop reading and start planning. I spent most of that morning working out what I would replace it with.
Then, because the number was so bad, I went and read the failures instead. Not the score. The actual questions, and the actual passages that had come back for each one. A few in, I hit one where the passage on my screen answered the question. Plainly, completely, better than the page I had written down. The evaluation had scored it as a miss.
It had scored it as a miss because my answer key named one page, and this was a different page.
There were more of them.
This is one offline assistant, running on CPU with no network, answering advice-style questions from a private library of source documents. One system, and I was the only person scoring any of it.
The answer key was the thing that was wrong
An answer key is the list of what counts as correct. Mine was one line per question: the question, and the page in the library where the answer lived. Scoring was an exact match. Did that page come back or not.
That works when a question has one right page. My library is advice, written by different people from different angles, and the same idea turns up in several documents. Several of those pages genuinely answer the question. I had written down one of them and let an exact match settle everything else.
It got that way for a boring reason. One page per question is quick to write and easy to score without anyone reading anything. It looked fine while the questions were easy, and only broke once they got hard.
So I scored the same run again, by hand, asking a different thing of each result: did anything that came back actually answer this question? Same retriever, same library, same index, same questions, and the same retrieved passages sitting in the same file. Two things were different. Which passages counted as correct, and who applied that rule: me, reading each one, instead of an exact page match.
Scored that way, the same keyword search came out at about half.
Nothing about the retriever improved. Not a line of code, not a parameter, not a document. It was exactly as weak at the end of the day as it had been that morning. Only the scoring changed. “About half” is not a corrected final score. It is what the same weak result looks like once you stop grading it against a list of pages I happened to write down.
There are two ways to get a key wrong. Too narrow, and it fails work that was fine. That is what happened here, and it cost me a morning and very nearly a rewrite of a component that was not the problem. Too loose, and it passes work that is not fine, which is the more expensive mistake, because you find that one out later and not on your own.
The narrow key had also been flattering something. I had recently put a newer retrieval method in beside the old keyword search, and against that key it looked like an enormous leap. Scored properly, most of the gap closed, because the narrow key had been punishing the older method hardest. The newer method was still genuinely better, on evidence I trust. It was just not the leap I had been enjoying.
Here is the part I would rather leave out.
I published that number. It is the number that carries the argument in When the small language model isn’t the bottleneck, which went out in July: about 1 question in 16. The corrected reading was already sitting in my own notes when that piece went live, and I never went back to what I had published. There is no good reason. The number had already done its work in that argument and I had moved on. Nobody was going to check it except me, and I did not.
Both numbers are true. About 1 in 16 is what that retriever scores against the key that existed. About half is what the same retriever scores once a person checks whether anything that came back genuinely answered. What was missing was that the second number exists.
Fixing the key did not save the system
None of this rescued anything. Later I built a larger set of questions and ran it with better retrieval than that keyword search. I scored it against the measure that actually decided whether the work continued. Not “did something relevant come back”, but “how much of what a complete answer needs came back”. Different questions, a different retriever, a harder measure. None of it is a continuation of what the keyword search scored.
That measure came in under half, against a bar of 70%.
I tried three different embedding models and all three landed in roughly the same place. That was enough for me to stop treating the embedding model as the main problem.
It failed the gate, so I stopped there. Designing Enterprise Agentic AI Systems covers the wider version of this: the gates and the evidence a grounded system needs before anyone trusts it.
What I changed about the key
First, where the answers come from. Write a key from memory and you are grading the system against your memory, not against the documents. Every entry now quotes a sentence that actually appears in the library, and I check the quote is really there before I score anything with it.
Second, the key now records what kind of question each entry is.
| Kind of question | The right answer |
|---|---|
| The answer is in the library | Find it and answer |
| It should be covered, but no passage covers it | Say the source does not contain enough |
| It was never this library's business | Decline |
My page-match key could not see the last two. That is where the failure I most needed to catch lives: answering confidently when the library has nothing to answer with. A key that records only a right or wrong page gives a correct refusal and an invented answer the same score.
There is a trap right next to that, and I sat in it for a while. A basic top-k vector search can still return the nearest passages even when none of them are good enough. So a question the library cannot answer still produces results. Finding the evidence and declining when there is none are two different tests. I was reading one number as though it covered both.
Who grades the grader
I also had a cheap automated judge scoring the answers. It passed nearly everything, which meant it was not telling me anything. So I stopped letting it decide anything, and went back to reading the transcripts myself. Slow, and still the one I trust. I spot-check the judge against my own reading, and I have not reached the point where I can stop.
The other thing that stopped me taking credit was duller. I ran the stock base model, with no fine-tuning at all, as a control. It turned down questions that were none of this library’s business, about as well as the tuned one did. The refusal discipline I had been quietly attributing to my training run was not mine. Without the control I would have kept investing in something I believed was working.
The part I still find hard
All of this reads as reasonable now. It did not feel reasonable at the time. Once you have seen a bad score, correcting the answer key and moving the goalposts feel identical from the inside. Both of them start with the thought that the measurement was not fair.
I do not have a clean test. I have one question I ask myself:
Would I make the same correction if I did not know which retriever produced this result?
It changed two things. When I rebuilt the scoring for the larger set of questions, I judged each passage without knowing which method had retrieved it, so I could not quietly be generous to the method I wanted to win. And I settled the bar before the scoring, not after. Settling it was itself a correction, because as written the bar had come to demand near-perfect retrieval, and changing it once I had seen the numbers would have been indefensible. A gate you can move after you have seen the score is not a gate, which is the same reason a readiness check has to be a gate and not an average.
I had also written a rule down in advance for exactly this: labels that turn out to be too narrow get corrected and rerun. I still had to sit with it and call the result what it was, which was encouraging, and not a pass.
The best evidence that the correction was not self-serving is what it produced. It raised a number on the questions it touched, and the system still failed the gate it later faced.
Where it stands
It still fails. The score that decided it is still under the bar, and I have not gone back to it.
If you are building an eval set now, budget for two things. Reading failures by hand, on purpose and on a schedule, because the score will not tell you the key is wrong. And being wrong about the key at least once, which is fine as long as you find it yourself rather than having it found for you.
If you want an outside read on whether your agent’s evaluation is measuring what you think it is, that is part of what the Agentic AI Readiness Review looks at. Details at how I can help.
Frequently asked
Quick answers
- What is an answer key, or ground truth, in an AI evaluation set?
- It is the list of what counts as a correct result for each question in your evaluation set. Mine was one line per question: the question, and the single page in the library where the answer lived, scored as an exact match on that page. That works only when a question has exactly one right answer. Mine was not like that, because the same idea turns up in several documents written by different people from different angles.
- My evaluation score went up after I corrected the answer key. Did the system get better?
- Not necessarily, and mine did not. The retriever did not change between my two numbers: not a line of code, not a parameter, not a document. Two things changed, which passages counted as correct and who applied that rule, so the higher number is what the same weak result looks like under a fairer rule. Correcting the key did not rescue the system either. It later failed the gate that decided whether the work carried on.
- How do you score questions an AI assistant is supposed to refuse?
- Put them in the key as their own kinds of question, where the right answer is a refusal. My old key only recorded a right or wrong page, so it gave a correct refusal and an invented answer the same score. Watch for one trap: a basic top-k vector search can still return its nearest passages when none of them are good enough, so a question the library cannot answer still produces results. Finding the evidence and declining when there is none are two different tests.