Building Search for BrainZen: From Embeddings to Hybrid Retrieval

Wow...nice advice especially with tagging and last modified time. I think I understood what you are trying to say.

For the embedding race, am I understanding it correctly that the problem only appears when the re-embedding jobs are allowed to run "independently"? For example, I am generating embedding for content saved by the user. For instance, CV2(content version 2) queues a job for EV2(embedding version 2), then CV3 queues another job for EV3 before EV2 finishes. Since the CV2 job already captured the old content, it doesn't automatically know that the document has since become CV3. So if EV3 finishes first and then EV2 finishes later, EV2 could overwrite the newer embedding unless we associate each job with a version and reject stale results. Is that the idea?

And on filtering, I'm curious about the tradeoff. I completely agree that authorization filters like userId should happen before retrieval so we never search/return another user's data. But for ordinary relevance filters like tags/type, I'm not sure pre-filtering every retrieval path is always better.

For example, if I retrieve 100 candidates from a large corpus and then filter them down to 20, I may lose relevant documents that never made it into those 100 candidates. If I pre-filter first and then retrieve from the eligible 50k documents, it seems like retrieval gets a better candidate pool within the actual search space. On the other hand, over-fetching could also solve the post-filter problem.

So I'm wondering: would you distinguish authorization/data-isolation filters from ordinary metadata/relevance filters here? And in a hybrid system, when would you prefer pre-filtering each retrieval path versus over-fetching and filtering later?

I'm still learning the practical tradeoffs here, so I'd really like to understand how you'd think about it in a production system.

Yeah, that's exactly it. The failure only shows up because the two jobs are unordered relative to each other, EV2 captured CV2's content but has no way of knowing CV3 exists by the time it writes. Tagging each job with the content version it embedded and rejecting a write once a newer version has already landed closes it without needing the jobs to run in order.

On filtering, I'd draw the line at what happens if the filter is wrong rather than at what kind of filter it is. Getting authorization wrong leaks another user's data, so that has to happen before retrieval regardless of cost. Getting a relevance filter wrong just means a worse ranking, so it's fine to let it happen after retrieval when pre-filtering would shrink your candidate pool too much, which is exactly the top-K problem you described. I'd over-fetch and filter after for anything where the eligible set after the filter is still a meaningful fraction of the corpus, and only move the filter into the retrieval path itself once it's selective enough that post-filtering would regularly starve you below K. If you don't know that ratio yet, over-fetch by two or three times K and see how often the post-filter step still leaves you short, that tells you whether the filter needs to move earlier.

Thanks Man... I'll keep that in mind. BTW I implemented versioning....You can check it as its always good to have second opinion, Thanks