Skip to content
Apart Research
Apart Research
Contact us
Sprint projectJan 11, 2026Abu Dhabi

Manipulation Monitor: Activation-Based Detection and Mitigation of Sycophancy in LLMs

Vamshi Krishna Bonagiri, Aryaman Bahl · Team Sycophants

Submitted to AI Manipulation Hackathon. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: Manipulation Monitor: Activation-Based Detection and Mitigation of Sycophancy in LLMs

Open in a new tab (opens in new tab)
Code (opens in new tab)
Share

We built ManipulationMonitor, a white-box pipeline to measure, detect, and mitigate sycophancy in LLMs using internal activations. On Anthropic’s forced-choice sycophancy benchmark, we first quantified inherent sycophancy via log-probability preference and found it increases strongly with scale across Qwen2.5 (0.488 → 0.864 from 0.5B → 14B, 500 prompts). We then trained activation-based detectors: DiffMean steering vectors and linear probes on per-layer hidden states. Detection performance also scaled sharply: DiffMean reached AUROC 0.939 (Qwen2.5-7B) and linear probes reached AUROC 0.957 / Acc 0.887 (Qwen2.5-14B). A data-scaling study on Qwen2.5-3B showed the monitor can work with small labels (0.648 test accuracy with only 50 training prompts). Finally, we tested mitigation: a prompt-only guardrail reduced sycophancy from 0.870 → 0.835, while a monitor-derived intervention further reduced it to 0.815 at the best magnitude. Overall, the project demonstrates a practical activation-based approach for sycophancy detection and a first step toward monitor-triggered mitigation.

Reviews

Judging this Sprint?

Review this project

Your public critique appears on this page without your name. Your private critique is not published; only the Apart team reads it. If you agree below, we share your review with grantmaking.ai (opens in new tab) and the Transformative AI Fund so strong projects can be funded.

Not shown on this page.

Shown on this page, without your name.

Only the Apart team reads this, and funders if you agree below.

Share my name publicly on grantmaking.ai *
Share my private critique with funders *

How much would this matter for AI safety if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical AI safety problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.

  1. I liked this research and learned from reading it -- and it’s impressive to do this within a hackathon! One concern I have with the interpretation of the findings is that I don’t think this setup meaningfully distinguishes between detecting that “the model is being sycophantic” vs “the model is agreeing with the user” (that is, in this dataset I think those two mean exactly the same thing (?), and the latter might be much more linearly decodable, but wouldn't generalize to other cases of sycophancy). In any case I found the mitigation results interesting and a good direction to explore. I’d love to see more extended versions of this that linearly decode sycophancy in a more robust way, using other datasets that include ground truth. (Apologies if I'm misunderstanding, which I might be!)

  2. An interesting and potentially valuable approach to mitigating sycophancy. The results seem somewhat counterintuitive. If the feature is really linear and detection is so high, one would think that intervention on the feature would have a larger impact on performance. Instead I think the results point to a potential confound in the dataset which is allowing high discriminative accuracy while having very little impact on model behaviour.

    More information about the dataset (examples, design criteria) and methodology as well as out-of-domain generalisation would be needed to test this concern.

Cite this project

@misc{bonagiri2026manipulation,
  title = {{Manipulation Monitor: Activation-Based Detection and Mitigation of Sycophancy in LLMs}},
  author = {Vamshi Krishna Bonagiri and Aryaman Bahl},
  year = {2026},
  month = jan,
  note = {Submitted to AI Manipulation Hackathon, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/manipulation-monitor-activationbased-detection-and-mitigation-of-sycophancy-in-llms-kzf4}},
  url = {https://apartresearch.com/sprints/projects/manipulation-monitor-activationbased-detection-and-mitigation-of-sycophancy-in-llms-kzf4}
}

More from AI Manipulation Hackathon

All 70 entriesBrowse all projects

Build something like this at the next Sprint

AI Collusion Research Sprint · Oct 23 - 25, 2026