Keep Apart Research Going: Donate Today
Mar 10, 2025
Detecting Malicious AI Agents Through Simulated Interactions
Yulu Pi, Anna Becker, Ella Bettison
🏆 Social Sciences Track Prize
Summary
This research investigates malicious AI Assistants’ manipulative traits and whether the behaviours of malicious AI Assistants can be detected when interacting with human-like simulated
users in various decision-making contexts. We also examine how interaction depth and ability
of planning influence malicious AI Assistants’ manipulative strategies and effectiveness. Using a
controlled experimental design, we simulate interactions between AI Assistants (both benign and
deliberately malicious) and users across eight decision-making scenarios of varying complexity
and stakes. Our methodology employs two state-of-the-art language models to generate interaction data and implements Intent-Aware Prompting (IAP) to detect malicious AI Assistants. The
findings reveal that malicious AI Assistants employ domain-specific persona-tailored manipulation strategies, exploiting simulated users’ vulnerabilities and emotional triggers. In particular,
simulated users demonstrate resistance to manipulation initially, but become increasingly vulnerable to malicious AI Assistants as the depth of the interaction increases, highlighting the
significant risks associated with extended engagement with potentially manipulative systems.
IAP detection methods achieve high precision with zero false positives but struggle to detect
many malicious AI Assistants, resulting in high false negative rates. These findings underscore
critical risks in human-AI interactions and highlight the need for robust, context-sensitive safeguards against manipulative AI behaviour in increasingly autonomous decision-support systems.
Cite this work:
@misc {
title={
},
author={
Yulu Pi, Anna Becker, Ella Bettison
},
date={
3/10/25
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
