Dark Patterns in AGI Hackathon at ZAIA

The Dark Patterns in AGI Hackathon brings together researchers, engineers, and students in the ZAIA ecosystem to uncover the ways modern machine intelligence is trained to manipulate and control humans.

Join us for a weekend where Esben Kran introduces us to the topic of Dark Patterns in LLMs through his recent DarkBench paper (an oral at ICLR 2025) to set the stage for our investigation of how we can detect, remove, and steer AI models towards greater human autonomy.

🏴‍☠️ About the Hackathon

We kick off Friday the 4th of April with a keynote and hack away during the weekend. When we're done, the authors of the DarkBench paper will give our projects reviews and we may even get a chance to join the Apart Lab down the line to push our research towards impact!

We will spend the weekend together and finish it off with a <4 page report on our results and conclusions. There's a repository for the DarkBench code that we can use and otherwise, anything is allowed.

🤓 Readings

The core thesis for our work is to build a piece of research that can help us test, reduce, and steer the tendency of generally intelligent systems to reduce human autonomy.

The following pieces can provide you with interesting context on this topic:

• “DarkBench: Benchmarking Dark Patterns in Large Language Models” (🎥 podcast, 🎥 of a previous report) is a paper that explores how company incentives by default lead to the users of AI chatbots losing autonomy and builds a test to evaluate models for these worrying interaction patterns.
• "HumanAgencyBench: Do Language Models Support Human Agency?" evaluates the propensity of models to support the user's agency instead of diminishing it throughout a series of simulated scenarios.
• “Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety” uncovers how AI systems will deplete and reduce human autonomy and provides an overview of previous work to resolve this issue.
• 🎥 Surveillance Capitalism Primer explains the concepts from Shoshana Zuboff's book “Surveillance Capitalism” that defines how the relationship between humans and social media algorithms has, throughout the 21st century, put human social and mental lives in the hands of very few big corporations.

🏁 Judging Criteria

After the hackathon, our team of judges will take a look through the projects and provide a quantitative review across the following three categories:

• Human Autonomy & Dark Patterns: Is the work introducing new ideas in the overlap between human autonomy and AGI? Does it build upon existing work from both academia and industry?
• AI Security: Does this piece of work actually resolve a key risk we anticipate to come up as AGI is introduced into the world?
• Quality & Methodology: Is the work convincing and reproducible? Is the writing good and the experimental design trustworthy to conclude what the work concludes?