Out of Sight, Out of Mind: Image Preferences in Vision-Language Models

This was a well thought out project. I really liked the direction they went in for image preferences and context compression. This was also unusually well written and communicated! finding that context compression changes preferences completely. Measuring the difference between revealed choice and stated preference is also rigorous. I think the research problem was done well but the implications for the field are missing. Why does it matter whether models appear to prefer one image over another? If we don't believe this finding impacts model welfare which is a big leap and not proven here the work does not have much implications. I think this person has clear skills in AI research and should continue but research taste could use some development.

This submission asks whether vision-language models have preferences about what they look at, measuring stated ratings and repeated choices over the same small image set in four models from four labs, and reporting both that self-report tracks choice and that removing a model's own prior turns from its context collapses exploration into repetition. The process around the result is the strongest part of the work: predictions were registered before any data was collected, several discarded designs are reported together with the measurements that killed them, and the call-level record is public. The redaction condition would repay one more cell, because it currently changes three things at once, removing the assistant turns while also adding a paragraph to the system prompt describing the redaction study and a parenthetical to every user turn restating that the reasoning is missing, so a condition that removes only the turns and leaves the prompt wording identical to the full-history arm would separate memory loss from a response to being told about the manipulation. The correlation between stated rating and revealed choice also needs its per-model values, its unit of analysis and an interval reported alongside it, since at ten items the low end of the reported range does not exclude zero and can be produced by the two degenerate stimuli sitting last on both measures while the photographs are ranked independently.