Adding questions about what the agent can do, next to where the data lives, fills a real hole in most vendor reviews. The question about whether approval is enforced in code or by the model is the one I would ask first, since a prompt instruction cannot be audited the way a check in the execution path can.
The scoring rule is the strongest part. A 0 on a weight 3 question stopping the rollout regardless of the total keeps a vendor from averaging their way past a single disqualifying gap, which is exactly what weighted spreadsheets tend to allow.
Asking for evidence rather than answers also changes the conversation. A demo of an admin revoking access in minutes says more than a paragraph about it, and the markdown image exfiltration test is cheap to run during a pilot.
For smaller buyers without a security team, would you cut the list to the weight 3 questions only, or is that where people get into trouble?
Adding questions about what the agent can do, next to where the data lives, fills a real hole in most vendor reviews. The question about whether approval is enforced in code or by the model is the one I would ask first, since a prompt instruction cannot be audited the way a check in the execution path can.
The scoring rule is the strongest part. A 0 on a weight 3 question stopping the rollout regardless of the total keeps a vendor from averaging their way past a single disqualifying gap, which is exactly what weighted spreadsheets tend to allow.
Asking for evidence rather than answers also changes the conversation. A demo of an admin revoking access in minutes says more than a paragraph about it, and the markdown image exfiltration test is cheap to run during a pilot.
For smaller buyers without a security team, would you cut the list to the weight 3 questions only, or is that where people get into trouble?