The Tester's Guide to RAG Evaluation, Part 2: Building the Evals
In Part 1 you met the RAG system: retrieval finds passages in a corpus, generation writes an answer from them, and each half fails in its own way. You spiked the system to see how it really behaves, a
Responses
Markdown supported · ⌘↵ to post
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
Assigning the first upstream failure is useful for prioritizing fixes, but I would keep secondary observed symptoms in the trace rather than discarding them from the taxonomy record. A retrieval miss and a generator willing to invent an answer can coexist; repairing retrieval alone may hide the second defect until the next miss.
For a small diagnostic subset, replaying the same question with the correct evidence supplied would help distinguish a cascade from an independent generation failure. The primary-cause counts can remain exclusive while those counterfactual runs reveal which downstream protections still need work.