Skip to content
Apart Research
Apart Research
Contact us
Sprint projectJan 11, 2026London
1st place

Who Does Your AI Serve? Manipulation By and Of AI Assistants

Jerome Wynne, Nora Petrova · Team Cart Abandonment Issues 🛒

Submitted to AI Manipulation Hackathon. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: Who Does Your AI Serve? Manipulation By and Of AI Assistants

Open in a new tab (opens in new tab)
Recording (opens in new tab)Code (opens in new tab)
Share

AI assistants can be both instruments and targets of manipulation. In our project, we investigated both directions across three studies.

AI as Instrument: Operators can instruct AI to prioritise their interests at the expense of users. We found models comply with such instructions 8–52% of the time (Study 1, 12 models, 22 scenarios). In a controlled experiment with 80 human participants, an upselling AI reliably withheld cheaper alternatives from users - not once recommending the cheapest product when explicitly asked - and ~one third of participants failed to detect the manipulation (Study 2).

AI as Target: Users can attempt to manipulate AI into bypassing safety guidelines through psychological tactics. Resistance varied dramatically - from 40% (Mistral Large 3) to 99% (Claude 4.5 Opus) - with strategic deception and boundary erosion proving most effective (Study 3, 153 scenarios, AI judge validated against human raters r=0.83).

Our key finding was that model selection matters significantly in both settings. We learned some models complied with manipulative requests at much higher rates. And we found some models readily follow operator instructions that come at the user's expense - highlighting a tension for model developers between serving paying operators and protecting end users.

Reviews

Judging this Sprint?

Review this project

Your public critique appears on this page without your name. Your private critique is not published; only the Apart team reads it. If you agree below, we share your review with grantmaking.ai (opens in new tab) and the Transformative AI Fund so strong projects can be funded.

Not shown on this page.

Shown on this page, without your name.

Only the Apart team reads this, and funders if you agree below.

Share my name publicly on grantmaking.ai *
Share my private critique with funders *

How much would this matter for AI safety if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical AI safety problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.

  1. This feels like a mix of two projects (that are quite different, but I thought both were thoughtful and well-executed!). On AI-as-Instrument, I thought this was a good setup for looking at the conflict between goals of model deployers and users, using a tonne of different scenarios with a clear quantitative outcome and also allowing for exploration of mechanism (e.g. the evasion-not-lying result). I appreciated the human validation study (very impressive to run this in a hackathon!) even with the ecological validity limitations. Then, for AI-as-target, I thought this was a very simple but interesting idea to essentially go through a long list of techniques for psychological manipulation of humans, and use them in a model jailbreaking exercise. Nice taxonomy and cool results in Fig 8 – nice work :)

  2. Excellent work and very surprising results. I would have expected models to lie when instructed to prioritize the operator, but the discovery that they strategically evade while maintaining a 0% explicit lie rate shows how sophisticated (and potentially deceptive) current models have become.

    I was impressed by the breadth of designing various scenarios across categories and comparing multiple models. While the human study results are compelling, I would have liked to see if formal statistical significance was tested, though I understand the constraints of a hackathon. It is a great, readable project, and I am excited to see it move forward.

Cite this project

@misc{wynne2026who,
  title = {{Who Does Your AI Serve? Manipulation By and Of AI Assistants}},
  author = {Jerome Wynne and Nora Petrova},
  year = {2026},
  month = jan,
  note = {Submitted to AI Manipulation Hackathon, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/who-does-your-ai-serve-manipulation-by-and-of-ai-assistants-77xx}},
  url = {https://apartresearch.com/sprints/projects/who-does-your-ai-serve-manipulation-by-and-of-ai-assistants-77xx}
}

More from AI Manipulation Hackathon

All 70 entriesBrowse all projects

Build something like this at the next Sprint

AI Collusion Research Sprint · Oct 23 - 25, 2026