Guardian Lens: Black-Box Identification of Visual-Conditional Decision Policies in Vision-Language Models
Mostafa Bdeir
Guardian Lens is a black-box auditing framework for identifying whether a vision-language model follows neutral, visual-cue-bound, or generalized decision policies from behavior alone. We use matched visual counterfactuals, controlled allocation trade-offs, repeated sampling, and a frozen blinded classifier to test when these policies are distinguishable—and when they become observationally indistinguishable. In our primary Gemini evaluation, the auditor achieved 91.7% held-out accuracy, with all errors occurring in distractor scenes where cue-bound and neutral behavior produced the same observable signature. We additionally replicated the full experimental protocol on a second vision-language model to test cross-model robustness.
Really solid methodological hygiene here, and I mean that as praise. Pre-specified hypotheses, a frozen nearest-centroid auditor, A/B/C blinding before the mapping is revealed, pixel-level validation of the overlay region. Most sprint projects don't come close to that. My main worry is that the identification problem you built is a good deal easier than the one you actually care about. The policies are induced by system prompt and the model simply complies, so allocations pile up at 0, 50 and 100, and several of your CIs literally read [100.00, 100.00]. At that point 91.7% accuracy isn't very informative, and the three Cue-bound errors on distractor scenes follow directly from the instruction rather than being a discovered boundary. I'd add a difficulty dial: induce partial priorities (say a 65/35 lean), or withhold the trigger from the auditor so it has to find it. The Qwen cue-specificity gap was your most interesting result, honestly. Give it more room.
Cite this work
@misc {
title={
(HckPrj) Guardian Lens: Black-Box Identification of Visual-Conditional Decision Policies in Vision-Language Models
},
author={
Mostafa Bdeir
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


