Comment by Sina Rezaei on "RAG With Claude in the Enterprise: Five Retrieval Patterns That Held Up"

Sina Rezaei

One case I'd pay particular attention to is when retrieval returns documents that are individually relevant but shouldn't be combined. Imagine a bank's compliance assistant retrieving an older AML procedure, a newer policy amendment, and an internal exception memo that only applies to a specific business unit. A reranker might rank all three highly, but that doesn't mean they're jointly valid for the question being asked.

I'd make document version, effective date, jurisdiction, and access scope part of retrieval itself, rather than relying on Claude to resolve those conflicts after the chunks enter its context. I'd also preserve the source version and chunk IDs through generation so every citation can be traced back to the exact evidence used.

The interesting test here isn't just whether the right document gets retrieved. It's whether the system can avoid producing a confident answer from a combination of sources that were never valid together. Have you tested your retrieval patterns against conflicting versions and permission-scoped documents?

Darius Tran

Share thoughts and experience with other developers and product makers

4d ago

Great point. We spend a lot of time evaluating retrieval quality, but not enough time testing whether retrieved documents should coexist in the same context window. Relevance and compatibility aren't always the same thing.

There’s another edge case here: what if the documents needed to resolve a conflict exist, but the user isn’t allowed to retrieve them?

Take a fraud investigation where an analyst can access the standard escalation procedure, but a case-specific exception is restricted to the compliance team. The retrieved procedure may look authoritative, and the answer may be perfectly grounded in what the analyst can see, yet still lead to the wrong action because a material exception is missing.

I’d test this by running the same question under different permission scopes and checking whether the system can detect insufficient evidence without exposing the restricted content. That’s a different failure from a retrieval miss, and reranking won’t fix it. How do you distinguish a genuine conflict from a conflict the current user simply can’t see?

Search Hashnode