I'd also carry those constraints through to the output, not stop at retrieval. An agent may be allowed to read two sources separately but not combine them into a summary for a recipient who can only access one of them. So the policy needs to cover both source access and destination constraints, including derived content. Otherwise narrow retrieval can still result in a broader disclosure.