Discussion on "The 21-case test that decides whether a RAG chatbot ships"

Kalpak Kale · 2026-10-20T06:07:06.993Z

Every retrieval-augmented chatbot demo looks the same: paste in a folder of PDFs, ask a question, get a plausible answer, applause. Then it meets real users, who ask real questions, and the plausible

Three weight-3 scope cases breaking after a prompt made the bot friendlier made me wince. Tone tweaks are where I've seen refusal behavior slip without anyone noticing. The criterion I'd copy first is the bot having to say "the document does not say" at least once per round. When the client's team tags an answer as a document issue, does it ever turn into a new case in the 21?

Thanks Kartik. Tone changes were the single biggest source of regressions for us, which is why rule 2 (a regression blocks until explained) exists at all.

On your question: a document-issue tag by itself does not become a case, because the suite tests the bot, not the corpus. Those go to the content owner as a governance item. It becomes a case when the tag exposes a behaviour gap: for example the bot answered confidently from outdated material when the correct output was “the document does not say”. That failure is the bot’s, so it gets a case and a weight. Roughly, document issues feed the content backlog, and the bot’s reaction to bad documents feeds the suite.

The session-level cache is an interesting interaction with your short-query expansion. If an answer depends on the previous message, a key containing only the normalised current question can return an answer from an earlier conversational context.

I would add a paired case to the 21-case suite: ask the same short question after two different product references in one session and require the answer to follow the new reference. The key can then include the resolved standalone query and relevant corpus version. That checks whether consistency comes from repeatable grounding rather than simply replaying the first answer.

Good catch, and you are right. Keying on the normalised current question alone can replay an answer from an earlier context when the short query depended on the previous message. The fix is exactly what you describe: key on the resolved standalone query (after the expansion step) plus the corpus version, so a changed document invalidates the entry as well.

Your paired case is a better test than the one we had. Same short question after two different product references in one session, answer must follow the new one. That is going in as case 22. Thanks for the improvement.

Case 22 will be stronger if it asserts the second response as well as a cache miss. The cache might miss correctly while the expansion step still resolves the short question to the first product. Recording the resolved standalone query makes those two failures distinguishable.

I would also run the sequence in both product orders. That catches an accidental fixed-product answer without needing to make the test depend on a particular wording of the generated response.

Search Hashnode

The 21-case test that decides whether a RAG chatbot ships

Responses5