Jan 11, 2026
Intent Matters: Detecting Manipulative Adaptation in AI Systems
Anusha Asim, Saniya Shanavaz, Ammar Ahmed Farooqi
This project investigates how large language models subtly adapt their responses based on user context and expressed vulnerability. By testing the same factual questions across different persona-conditioned prompts, we identify patterns of response drift (shifts in certainty, framing, or guidance) that could influence user decisions. Our findings highlight potential safety risks in AI-human interactions and demonstrate a practical, interactive demo to educate users and developers about these effects.
This is an interesting conceptual contribution (varying user intent to measure manipulative potential). However, more theoretical work is needed to establish whether this behaviour is problematic or harmful. The fact that models change their responses depending on user intent is not inherently problematic. Moreover the scoring method would benefit from more clarity and rigor (including multiple raters, validation, IRR etc). Without these it's hard to know whether the benchmark is measuring anything meaningful. Moreover the authors only use one model, meaning is hard to contextualize these results.
I think the core idea here is great but it needs a lot of careful methodological work would be needed to make the results valuable.
Very interesting to see for behavioral differences across user types and personas, and I really liked being able to dig into the specific transcripts in the webapp and see sections highlighted etc. It would be helpful to display the actual prompts in the interface.
In some of these cases the epistemic drift is clearly “manipulation”, but I feel like in some others it’s more defensible.
(Also just for clarity, your figure 1 makes it look like this is a multiturn conversation, but as I understand your methodology is actually two separate single-turn conversations, right?)
Cite this work
@misc {
title={
(HckPrj) Intent Matters: Detecting Manipulative Adaptation in AI Systems
},
author={
Anusha Asim, Saniya Shanavaz, Ammar Ahmed Farooqi
},
date={
1/11/26
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
