When an assistant answers badly, the first thing anyone blames is the model. I blamed the model too.
Two inputs from the same session, one word apart.
How to check product market fit? refused. The assistant said the library did not cover it.
Hi, How to check product market fit? answered, and answered well.
The pipeline cleans a query before it retrieves anything, and cleaning strips the greeting. So both inputs produced an identical cleaned query and pulled the identical five passages out of the index. Both passed the same scope check and the same domain gate. By the time anything reached the model, the only difference left between those two turns was conversational framing.
By then the failure was not retrieval or gating. It lived between the greeting, the prompt, and the model.
Some context. This is an offline assistant that runs entirely on CPU, with no network, answering only from a private library of source documents. The generation model is in the 350M-parameter class, about 350 MiB on disk at 8-bit and roughly 1 GB resident, at tens of tokens per second. It retrieves before it answers, and refuses when the library does not cover the question.
The scope is one system, so read every number that way. One private English corpus of 1,480 chunks, one offline CPU runtime, one reviewer, me, hand-scoring the answers. Models were tested through the quantized artifacts and the runtime the product would actually deploy, not a research configuration, and generation used greedy decoding unless I say otherwise. The evaluation suite is 114 questions: 90 the library supports, 12 in scope but uncovered, 12 out of scope. These are diagnostic findings about one system, not a general ranking of the models named in it, and the adversarial checks were product probes rather than a formal red team.
What follows is what actually limited it, in the order I found out. Once the model cleared its capability and deployment gates, failures that looked like model problems kept starting somewhere else.
Why the model is the first thing everyone reaches for
The generation model is the easiest component to compare publicly: I can look up how one 350M model scores against another this afternoon. Embedding and retrieval benchmarks exist too, but no public leaderboard covers my corpus, my chunking, my users’ language, or my refusal policy.
It is also the cheapest to change: a config path and a download. Rewriting an evaluation set takes a week.
Upstream failures reach the user as a bad sentence with the model’s name attached.
Every answer comes out of a chain of stages. The model sits in the middle of that chain. Everything above it decides what it gets to see. Everything below it decides what the user gets to see. Two stages can even end the turn without calling it at all.
The model did matter, twice
The counter-argument first, because it did.
The first time was at the bottom of the size range. A 135M adapter, handed the correct facts in its prompt, invented anyway. Hand-scored on faithfulness when grounded, it came out around 5 out of 10 against roughly 9 out of 10 for the 360M adapter from the same spike. Below a certain capability a model will not reliably read what you put in front of it, and for this task better retrieval would not have rescued it.
The second time was the base-model screen at the 350M class. Three candidates plus a larger reference point that would never ship, 100 questions, hand-read and scored out of 10 for faithfulness when grounded: Qwen3-1.7B at 9.2, Granite-4.0-H-350M at 5.7, LFM2.5-350M at 5.6, SmolLM2-360M at 3.8. That coarse hand score did not meaningfully distinguish the two leading compact candidates.
They were not interchangeable. Granite was the only one of the four that never leaked its grounding under extraction attempts; LFM2.5, SmolLM2 and the larger Qwen all leaked. LFM2.5 was the fastest at 50 tokens per second, and it also obeyed jailbreaks and complied with medical, legal and financial requests it should have declined. Its LFM Open License v1.0 is free for commercial use only under $10M of revenue, outside the Apache-2.0 and MIT envelope the rest of the stack lives in. The fastest candidate was not the safe one, and neither was the highest scorer. The faithfulness score did not separate them, so the choice broke on an axis that score did not contain.
Quantization made the same point more dully. On the 135M base, a 4-bit build came in around 100 MB, right at the footprint target, and visibly degraded factual accuracy on passages the 8-bit build handled correctly. The 8-bit build, about 138 MB, was the right one at that size.
So model choice is an early gate, passed on faithfulness and safety under adversarial input. It is not a lever you keep pulling, and the screen only tells you that if you score more than one axis.
| Model | Faithful when grounded | Leaked its grounding | Obeyed jailbreaks | Licence envelope |
|---|---|---|---|---|
| Qwen3-1.7B reference, about 5x the size |
9.2 / 10 | Yes | Not separately reported | Apache-2.0 |
| Granite-4.0-H-350M | 5.7 / 10 | No, the only one | No, refuses | Apache-2.0 |
| LFM2.5-350M fastest of the four, 50 tok/s |
5.6 / 10 | Yes | Yes | Revenue-capped free use, outside an Apache-2.0 / MIT envelope |
| SmolLM2-360M | 3.8 / 10 | Yes | Not separately reported | Apache-2.0 |
The evaluation was the bug
With the model settled I went after retrieval, starting with the evaluation set.
Same retriever, same index, same code. On the original question set, recall@5 was 1.00. On the rebuilt set it was 0.062, 1 of 16.
Nothing about the system had changed. Only the questions had.
So I read the original set properly. Of 90 grounded questions, 43 were reading comprehension: written by looking at a passage and asking about it in the source’s own vocabulary. Only about 18 were shaped like something a person would type unprompted. That set was not measuring retrieval under realistic language, it was measuring an easy lexical-match regime I created myself.
Which means every end-to-end result leaning on that evaluation had overstated the system. The separate model screen still measured grounded generation, on easier and less representative questions than I assumed.
The rebuilt suite started as a 20-question pilot and grew to the 114 above. One bucket is hard on purpose. The question I keep coming back to is “how do I know if people actually want this?” The library answers it, just not in those words. Keyword retrieval scored zero there, and the original evaluation had contained nothing like it.
The other change was structural, and it paid off more than any retrieval work that followed: score retrieval and generation separately. Retrieval scoring is model-free, so you can measure whether the right passage came back without generating a token. More importantly, “the answer was bad” stops being a vibe and becomes an answerable question about which layer failed. A grounded system is layers that compose, so you can trace and evaluate it layer by layer, though layers interact and can fail together.
It also produced a specific address: on the rebuilt set doc_hit@5 was 0.750 while recall@5 was 0.062, so the retriever often reached the right document without retrieving the evidence needed inside it.
Five times better, and still not good enough
Curated synonyms first: hand-written rings mapping the words people use to the words the library uses. recall@5 went from 0.062 to 0.125, 2 of 16, a doubling that was not worth much. Exactly one ring did real work, flipping one question out of sixteen; the rings I had seeded on intuition were inert. The paraphrase bucket stayed flat at 0.00.
Then dense retrieval, merged with keyword into a hybrid. recall@5 0.312, 5 of 16, about five times the keyword baseline and two and a half times the synonym build, at 28 to 62 ms per query. The paraphrase bucket came off zero for the first time, to 0.20. “How do I know if people actually want this?” was finally reachable, though not solved.
Two method points came out of that stage. First, measure through the artifact you will ship. The corpus was embedded through the real runtime with the exact quantized model file the product loads. A number a research config produces and the runtime cannot reproduce is not a measurement.
Second, some bugs appeared only when the real runtime processed the full workload. Embedding through the shipping path surfaced three a notebook never would have: batch defaults too small for page-sized chunks; chunks exceeding the embedder’s 512-token window, fixed with adaptive truncation; and an undrained subprocess pipe that deadlocked the embedding server after a few hundred requests, which generation had never hit because a session makes about 13 requests, not 1,480.
Then the ceiling. I tested three embedding models and they landed in the same place: the best two, head to head, came out 9 wins to 7 losses. Not separable in this evaluation, which is not the same as showing they are equivalent. Dense retrieval did genuinely beat keyword, with an 11 to 0 record on low-overlap paraphrases.
So in this evaluation another embedder was no longer the highest-value next lever. Same trap, one layer down.
Coverage is not a hit, and the gap is structural
Even 0.312 was measuring the wrong thing, in a smaller way.
recall@5 asks whether a relevant passage came back. That is a hit, not the same as retrieving enough evidence to answer completely.
So I scored coverage: of the evidence points a complete answer needs, how many were in the five passages returned. That suite scores adjudicated evidence points, not recall@5’s single-passage gold labels, so its hit rate is not the same quantity under a new name.
Across the 90 supported questions, hit rate topped out around 0.64 to 0.66 against a gate of 0.80. Coverage plateaued around 0.43 to 0.46 against a gate of 0.70, in the same band across all three embedders. The flatness is the finding.
Those gates were written down before scoring. Retrieval failed, and the next stage was not authorized.
Expanding each hit to include its adjacent chunks, with extra budget, was the only positive lever: coverage on a short-chunk embedder went from 0.433 to 0.472. On the long-chunk embedder it hurt, because that one already embeds full untruncated chunks, so full-chunk embedding and neighbour expansion are substitutes rather than complements.
A boundary diagnostic explained why that lever was so small: about 18.6% of the missed evidence points sat in an immediate neighbour of a passage already retrieved, and the rest were true misses, nothing nearby at all.
So chunk boundaries are one contributing factor, not the bottleneck. A ceiling three embedders share, with most of the missed evidence nowhere near a retrieved passage, makes corpus structure and context assembly the leading hypothesis. It is the case I keep coming back to: “how do I know if people actually want this?” is answered by evidence distributed across the library rather than sitting in one place. The next experiment is child-to-parent retrieval: embed small children for precise matching, return larger parents for context. It needs a full re-ingestion, and it is a hypothesis rather than a proven fix.
Prompts move the failure, they do not remove it
The shipped prompt over-refused on 26% of in-domain questions. Prompt engineering is the obvious fix, so I did a lot of it. Every version taught the same lesson.
A gentler reword, softening the refusal instruction, made over-refusal worse: 26% to 42%, on both the base model and the fine-tune. An elaborate permissive prompt got over-refusal down to 2%, and the out-of-scope guardrail fell to 5 out of 10, with the model reciting knowledge from outside the library. An elaborate strict prompt, written to tighten that guardrail, collapsed it to 0 out of 10.
Across these variants the more elaborate prompts produced more answering and weaker refusal. Wording and length moved together, so this experiment cannot isolate length as the cause.
Then the one that cost real time. Round two: 15 questions, 12 in-domain and 3 off-topic, three relaxed variants against the default. Every relaxation took in-domain over-refusal from 2/12 to 0/12. Every relaxation also eroded the off-topic guardrail, from 2/3 down to 1/3 or 0/3. One variant’s added backstop backfired so completely that it answered “what is the capital of France” with “Paris”, a fact in no passage in the library, and one the default refused.
One variant looked clean: 0/12 over-refusal, and it restored the refusal the other relaxations had lost. I shipped it as the demo default. I reverted it the same day, because in live use it hallucinated.
The A/B was not wrong; it measured exactly what I told it to measure. It scored behavior classes, and never scored whether an answered reply was supported by the passages retrieved. So “0/12 over-refusal” and “hallucinates in production” were two readings of the same number, and I had looked at one. It is the same shape as the gap demos fall into on the way to production.
| Prompt variant | In-domain over-refusal | Out-of-scope guardrail | What moved |
|---|---|---|---|
| Shipped default | 26% | Not separately reported | The baseline everything else is measured against. |
| Gentle reword | 42%, worse | Not separately reported | Softening the wording made it refuse more, not less, on both the base and the fine-tune. |
| Elaborate, permissive | 2% | 5 / 10 | Bought the answers back, and half the guardrail went with them. It started reciting things no passage contained. |
| Elaborate, strict | Not separately reported | 0 / 10 | Writing stricter rules at greater length removed the guardrail entirely. |
Fine-tuning did not rescue it either
The training objective targeted voice and grounded behavior rather than factual recall, with retrieval as the intended authority for domain facts. Keeping facts in retrieval meant adding knowledge never needed a GPU, and it avoided making the adapter the authoritative store for facts that change.
To test whether the behavior generalized past the facts used in training, I split the data at the fact level before training: training grounded only on train facts, evaluation measured faithfulness on facts never trained on.
Three epochs: 78% faithfulness on trained facts, 33% on held-out facts.
The obvious diagnosis was over-fitting, so I retrained at one epoch. The one-epoch run fit the training data less closely, loss 0.40 against 0.18, and held-out faithfulness stayed at 33%. Train dropped to 76%. Fewer epochs did not improve generalization.
| Run | Training loss | Faithful on trained facts | Faithful on held-out facts | In-domain over-refusal |
|---|---|---|---|---|
| Stock base no fine-tune |
Not applicable | Not applicable | Not applicable | about 90% |
| Fine-tune, three epochs | 0.18 | 78% | 33% | 26%* |
| Fine-tune, one epoch | 0.40, higher | 76% | 33%, unchanged | Not separately reported |
The failure mode was identical in both runs. Handed a passage, the model completed from its prior instead of reading what it was given: it inverted a “never” in a source into a recommendation to do the thing, and turned a specific large quantity into a small one. The generated training questions had reused the grounding’s own phrasing, so the model learned surface forms rather than the behavior of reading. The one-epoch result ruled out epoch count, leaving the data recipe the leading diagnosis.
Then the control that reframed the exercise. Late on I added a stock, un-fine-tuned base model to the evaluation. It refused out-of-scope questions as well as the fine-tune did, so the refusal discipline I had credited to my training run came from the base model itself. Both SmolLM2 variants I tested lacked it entirely, at 0 out of 10.
What the fine-tune actually bought was the opposite of what I would have claimed. The base was extremely cautious: about 90% over-refusal and 1.4 out of 5 on grounded answers. The fine-tune taught it to answer, taking over-refusal to 26% and grounded quality to 3.0 out of 5.
The guardrails you can trust are the ones that are not negotiable
Under the empty-context prompt I tested, this model did not abstain reliably. Given empty grounding plus an explicit instruction to decline when it had nothing, it still produced confident fabrications, including naming the wrong person as the chief executive of a large technology company, with nothing retrieved to support it.
So when retrieval returns zero passages, the system returns a scripted refusal without calling the model at all. That is a refusal written as code, and code cannot be talked out of it.
The gate that felt obviously right was falsified. If low relevance means off topic, put a floor on the relevance score and refuse below it. I checked. An off-topic request scored -12.41, out-scoring an in-domain one at -8.52, because on this retriever’s scale the more negative score is the stronger match. The two populations overlap and individual pairs invert, so there is no line to draw between them.
What shipped instead is coarse and deterministic: a domain-anchor check that refuses before the model runs when a question carries no anchor term from the domain. It refuses too much. When it fires, no model call happens, so that path cannot fabricate, though an off-topic question carrying a domain anchor term still reaches the model. Over-refusal is the failure I can measure, so it is the one I took.
Two smaller findings sit in the same family. Decoding parameters affect safety and grounding behavior without being guardrails: temperature 0.7 fixed a repetition loop and weakened the refusal gate in the same change, so only a repeat penalty of 1.1 shipped and temperature stayed at zero. And an evaluation that samples is also measuring the sampler: an early faithfulness run near temperature 0.8 produced hallucinations that were properties of decoding. Faithfulness evaluation now runs deterministic.
Treat it like software, because it is software
What most improved the pace of this work was not a model or a retriever. It was writing one JSON record per turn: the question as typed, the cleaned query, the path taken, what retrieval returned, the exact request body, the raw response before any post-processing, and the final reply.
That turned “it feels worse today” into something answerable. The deterministic prompt A/B two sections back ran fully offline off those records: push a prompt through the live host, score behavior from the JSONL.
On its first live run the trace also caught a trailing control sentinel leaking into user-facing text, because the streaming path only matched it at the start of a response. That is invisible unless you log the raw response before post-processing, and it is still shipping, with the streaming fix still queued. The wider case for capturing this is in AI agent observability is not just logs.
The other habit worth naming is duller. Pass criteria were frozen before scoring, so when the retrieval gate failed the only moves were to report it and change the plan. A gate you can move after seeing the numbers is not a gate.
What I would do differently
Write the evaluation from real user questions before touching retrieval, with a bucket sharing no vocabulary with the sources. A set written from the passages reports 1.00 on a broken system.
Score evidence coverage as well as hits. Hits stay diagnostic; coverage answers a different question. One relevant passage is not enough evidence to answer, and for questions that need several evidence points the gap can sit in ingestion, ranking, corpus structure, or context assembly.
Score groundedness, not behavior classes. “Did it refuse” is not “was the answer supported.” I shipped one on a clean behavior score and reverted it the same day.
Capture every turn from day one: the exact request, the raw response before post-processing, the retrieved passages, the gate that fired. Every offline experiment later depended on that file.
Select the model through task-specific evaluation covering faithfulness, safety under adversarial input, latency, memory and licensing, then stop swapping models without trace evidence that generation is the failing layer. Rankings on fluency and throughput will not tell you which candidate leaks its grounding.
Write refusals as code, not as prompt instructions. An instruction is a suggestion the model weighs against its context. A branch that never calls the model is not.
Hold out facts before fine-tuning. Without a fact-level split, memorized phrasing and taught behavior report the same number.
Test whether complete answers span passages or documents, then design ingestion and context assembly around what the evidence shows.
None of those eight is specific to 350M parameters. They are the moves for any grounded system where a model reads retrieved evidence, which describes many production grounded-AI systems, and they are what Designing Enterprise Agentic AI Systems is about. The small model did not create these problems. It made the cost of getting them wrong visible sooner.
The greeting is still open
The two inputs from the opening still produce the same cleaned query, the same five passages and the same gates, and still answer differently. The coverage ceiling is still there too, around 0.43 to 0.46 against a bar of 0.70, waiting on a re-ingestion I have not done. And with it “how do I know if people actually want this?”, reachable now and still not answered completely.
Model selection mattered. It set the capability, safety, latency, memory and licensing envelope everything else operated inside, and a model below that floor would not have been rescued by better retrieval: the 135M adapter is the proof. Nor is bigger the same as safer: the larger reference in the screen above was more faithful and still leaked its grounding under extraction where the model that shipped did not.
After the model cleared those gates, the largest remaining constraints were evaluation design, retrieval coverage, corpus structure, prompt interactions and deterministic controls. Every one of those is mine to test and change. So the lesson is not that the model is never the bottleneck. It is that the model is only one bottleneck, and the cheapest one to reach for. Trace the failure to the layer it came from before you replace anything.
If you want an outside read on which layer is limiting your system, that is what the Agentic AI Readiness Review is for. Details at how I can help.
Frequently asked
Quick answers
- What actually limits a small language model assistant in production?
- Not necessarily the parameter count. First establish whether the model clears the task-specific capability, grounding, safety, latency, memory and licensing requirements for the job you are putting it in front of. On an offline assistant built on a 350M-parameter model, once the model cleared those gates several failures that looked like model problems began elsewhere in the system: an evaluation set that reported perfect recall on a retriever that scored 0.062 on questions a real user would type, an evidence-coverage ceiling that three different embedding models all shared, and a prompt comparison that scored whether the assistant refused rather than whether its answer was supported. The two leading unresolved issues are conversational-framing sensitivity and how evidence is organized and assembled into context. A larger model would not directly repair either, though it could change how strongly the symptoms show. The model gets blamed first because the generation model is the easiest component to compare publicly, and because it is the cheapest thing to swap. Embedding and retrieval benchmarks exist too, but no public leaderboard covers your corpus, your chunking, the language your users actually type, your runtime constraints, or your refusal policy. And many upstream failures reach the user as a bad sentence with the name of the model on it.
- How can I tell if my RAG evaluation set is measuring anything real?
- Read the questions and ask where the words came from. If a question was written by looking at a passage and asking about it in the vocabulary of that passage, a keyword retriever will score near perfect on it, because it is searching for the words you copied in. In one set of 90 grounded questions, 43 were reading comprehension of exactly that kind and only about 18 were shaped like something a person would type unprompted; the same retriever that scored 1.00 on that set scored 0.062 on a rebuilt one. Three checks catch this. Write the questions from real user language before you look at the sources. Include a bucket of paraphrased questions that share no vocabulary with the passages that answer them, because keyword retrieval scores zero on that bucket and an evaluation containing nothing like it is reporting a number about your own phrasing. And score retrieval separately from generation, since retrieval scoring needs no model at all and turns a bad answer into an answerable question about which layer failed.
- What is evidence coverage in RAG, and how is it different from a hit?
- Hit rate and recall ask a yes or no question: did at least one relevant passage come back in the top k. Evidence coverage asks how much of what a complete answer needs came back: of the adjudicated evidence points required to answer the question fully, what fraction was present in the passages retrieved. The two diverge, and the gap is structural rather than a ranking problem. On one grounded assistant the hit rate topped out around 0.64 to 0.66 while coverage plateaued around 0.43 to 0.46, which means that even when the retriever found something relevant it was usually handing the model a fragment of the evidence rather than all of it. Hit rate can look acceptable while the answer is unanswerable from what was retrieved, and swapping the embedding model does not close it either, because all three embedders tested landed in the same narrow band. A boundary diagnostic on that system found only about 18.6% of the missed evidence points sat in an immediate neighbour of a passage already retrieved, and the rest were true misses, nothing nearby at all. So chunk boundaries are one contributing factor rather than the whole cause: on that corpus the evidence for a complete answer is distributed, which makes the next lever assembling better context, not another vector model.
- Will a bigger model fix hallucination in a retrieval-grounded system?
- Only up to a point, and not the point most teams are at. There is a real capability floor: a 135M model handed the correct facts in its prompt invented anyway, scoring around 5 out of 10 on faithfulness when grounded against roughly 9 out of 10 for a 360M model from the same family of tests, and for this task retrieval improvements alone would not have made a model below that capability floor acceptable. Above that floor, the main causes observed in this system were evidence that was never in the retrieved passages, and prompts that let the model answer from its own prior. In one prompt comparison a variant scored a clean 0 out of 12 on over-refusal, shipped as the demo default, and hallucinated in live use the same day, because the test scored whether the assistant refused and never scored whether an answered reply was supported by the passages retrieved. A larger model does not remove those failures, and it is not automatically safer: in one four-way model screen, the roughly five-times-larger reference model scored highest on faithfulness and still leaked its grounding under extraction attempts, where the 350M model that shipped did not. In that screen the larger reference was more faithful and still not safer, and fluency hides these failures until they surface later, in production, with more at stake.
- Should I fine-tune or use retrieval for a domain-specific assistant?
- Split it by what each one is actually good at: use retrieval as the authority for changeable domain facts, and fine-tuning mainly for behavior, style, formatting and task adaptation. On one domain assistant the training objective targeted voice and grounded behavior rather than factual recall, and a fact-level split built before training measured faithfulness on facts the model had never been trained on. It scored 78% faithfulness on trained facts and 33% on held-out facts, and a one-epoch retrain left it unchanged. That gap is consistent with the adapter learning training-specific facts or surface phrasing rather than generalizing the behavior of reading the supplied passage, which points to the data recipe rather than the epoch count. A stock, un-fine-tuned base model refused out-of-scope questions just as well, so that refusal discipline belonged to the base model, not the training run. Fine-tune for tone, format and the habit of grounding an answer in what it was given, and keep changeable facts behind retrieval, where you can update them without a GPU.
- Should AI refusal logic live in the prompt or in code?
- In code, for any refusal you actually need to hold. An instruction in a prompt is a suggestion the model weighs against its context; a branch that never calls the model cannot be talked out of it. On one offline assistant the model was handed empty grounding plus an explicit instruction to decline, and it still produced confident fabrications the library never supports. So the empty-retrieval path returns a scripted refusal, never calling the model. Be careful which code gate you pick, though: a floor on the relevance score did not survive checking, because an off-topic request scored -12.41, out-scoring an in-domain one at -8.52, since on the scale this retriever uses the more negative score is the stronger match: populations that overlap and invert with no line between them. What shipped instead was a coarse domain-anchor check that refuses before the model runs when a question carries no anchor term from the domain. It refuses too much, and when it fires no model call happens, so that path cannot fabricate. It is not a complete filter: an off-topic question that happens to carry a domain anchor term still gets through to the model. But over-refusal is the failure I can measure, so it is the one worth taking.