Nov 23, 2025
Comparative LLM methods for Social Media Bot Detection
Matthew Pagett, Andreas Raaskov
This project examines the potential of LLMs to detect social media
bots in (near) real-time, and the potential of using less-advanced LLMs to detect more advanced LLMs. It contributes to the cognitive defense toolset for protecting information ecosystems.
Report is well written and the research question has clear value. The lack of a strong ground-truth dataset is the biggest concern. It is hard to evaluate whether this approach will work against the actual threat without real examples of advanced LLM-generated bot accounts. It may be worth a separate project focused on proactively building that dataset. In addition, the inconsistent classifications depending on prompt phrasing make the current prototype unreliable.
Things for the team to consider:
(1) A robust dataset is essential. What are the practical paths to collecting or constructing a vetted set of LLM-generated accounts for evaluation?
(3) It may help to define the risk spectrum. Some bots are annoying while others are genuinely dangerous / harmful. A medium-accuracy classifier becomes more compelling if it focuses on accounts that are both likely AI-generated and capable of causing harm.
(3) Developers will increasingly design bots to evade text-based filters. Are there non-content signals (behavioral patterns, metadata, network structure) that could improve detection reliability over time?
Cite this work
@misc {
title={
(HckPrj) Comparative LLM methods for Social Media Bot Detection
},
author={
Matthew Pagett, Andreas Raaskov
},
date={
11/23/25
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


