Deception Detection Hackathon: Preventing AI deception

Are you fascinated by the incredible advancements in AI? As AI becomes smarter and more powerful, it's essential that we make sure it always tells the truth and doesn't trick people. Imagine if an AI could manipulate narratives or try to scheme—that could lead to serious problems!

That's where you come in. We're inviting you to join the Deception Detection Hackathon, an exciting event where you'll team up with researchers, programmers, and AI safety experts to create amazing pilot experiments to spot when an AI is being deceptive (and potentially reducing deceptive tendencies!). Over one thrilling weekend, you'll put your skills to the test and develop cutting-edge techniques to keep AI honest and trustworthy.

Why deception detection matters

Deception in AI, a concept severely under-explored, occurs when an AI system is capable of deceiving a user, either designed for it by a malicious actor or due to misaligned goals. Such systems may appear to be aligned with users and humans values during training and evaluation but pursue malign objectives when deployed, potentially causing harm or undermining trust in AI.

Examples of such work can be found in:

• Situational Awareness Benchmark: An evaluation of how well models can identify what state they are in (hackathon project, talk)
• LLM Identification Benchmark: Can an LLM identify whether a human or an LLM wrote a text? Might be important for self-coordination across chat instances (similar hackathon project)
• Strategic Deception Report: An example of strategic deception presented by Apollo Research at the AI Safety Summit 2023 (explainer)

To mitigate these risks, we must develop robust deception detection methods that can identify instances of strategic deception, make headway on understanding AI capabilities for deception, and prevent AI systems from misleading humans. By participating in this hackathon, you'll contribute to the critical task of ensuring that AI remains transparent, accountable, and aligned with human values.

Contribute to AGI deception research

During the hackathon, you'll have the opportunity to:

• Learn from experts in AI safety, deceptive alignment, and strategic deception
• Collaborate with a diverse group of participants to ideate and develop deception detection techniques
• Create benchmarks and evaluation methods to assess the effectiveness of deception detection approaches
• Compete for prizes and recognition for the most innovative and impactful solutions
• Network with like-minded individuals passionate about ensuring the safety and trustworthiness of AI

Whether you're an AI researcher, developer, or enthusiast, this hackathon provides a unique platform to apply your skills and knowledge to address one of the most pressing challenges in AI safety.

Join us in late June for a weekend of collaboration, innovation, and problem-solving as we work together to prevent AI from deceiving humans. Stay tuned for more details on the exact dates, format, and registration process.

Don't miss this opportunity to contribute to the development of trustworthy AI systems and help shape a future where AI and humans can work together safely and transparently. Let's hack for a deception-free AI future!

Prizes, evaluation, and submission

You will join in teams to submit a PDF about your research according to the submission template shared in the submission tab! Depending on the judge's reviews, you'll have the chance to win from the $2,000 prize pool! Find the review criteria on the submission tab.

• 🥇 $1,000 for the top team
• 🥈 $600 for the second prize
• 🥉 $300 for the third prize
• 🏅 $100 for the fourth prize

What is a research hackathon?

The AGI Deception Detection Hackathon is a weekend-long event where you participate in teams (1-5) to create interesting, fun, and impactful research. You submit a PDF report that summarizes and discusses your findings in the context of AI safety. These reports will be judged by our panel and you can win up to $1,000!

It runs from 28th June to 1st July and we're excited to welcome you for a weekend of engaging research. You will hear fascinating talks about real-world projects tackling these types of questions, get the opportunity to discuss your ideas with experienced mentors, and you will get reviews from top-tier researchers in the field of AI safety to further your exploration.

Everyone can participate and we encourage you to join especially if you’re considering AI safety from another career. We give you code templates and ideas to kickstart your projects and you’ll be surprised what you can accomplish in just a weekend – especially with your new-found community!

Read more about what you can expect, the schedule, and what previous participants have said about being part of the hackathon below.

Why should I join?

There’s loads of reasons to join! Here are just a few:

• See how fun and interesting AI safety can be
• Get to know new people who are into the overlap of empirical ML safety and AI governance
• Win up to $1,000, helping you towards your first H100 GPU
• Get practical experience with LLM evaluations and AI safety research
• Show the AI safety labs what you are able to do to increase your chances at some amazing jobs
• Get a certificate at the end!
• Get proof that your work is awesome so you can get that grant to pursue the AI safety research that you always wanted to pursue
• The best teams are offered to participate in the Apart Lab program, which supports teams in their journey towards publishing groundbreaking AI safety and security research
• And many many more… Come along!

Do I need experience in AI safety to join?

Please join! This can be your first foray into AI and ML safety and maybe you’ll realize that there are exciting low-hanging fruit that your specific skillset is adapted to. Even if you normally don’t find it particularly interesting, this time you might see it in a new light!

There’s a lot of pressure from AI safety to perform at a top level and this seems to drive some people out of the field. We’d love it if you consider joining with a mindset of fun exploration and get a positive experience out of the weekend.

What are previous experiences from the research hackathon?

Yoann Poupart, BlockLoads CTO: "This Hackathon was a perfect blend of learning, testing, and collaboration on cutting-edge AI Safety research. I really feel that I gained practical knowledge that cannot be learned only by reading articles.”

Lucie Philippon, France Pacific Territories Economic Committee: "It was great meeting such cool people to work with over the weekend! I did not know any of the other people in my group at first, and now I'm looking forward to working with them again on research projects! The organizers were also super helpful and contributed a lot to the success of our project.”

Akash Kundu, now an Apart Lab fellow: "It was an amazing experience working with people I didn't even know before the hackathon. All three of my teammates were extremely spread out, while I am from India, my teammates were from New York and Taiwan. It was amazing how we pulled this off in 48 hours in spite of the time difference. Moreover, the mentors were extremely encouraging and supportive which helped us gain clarity whenever we got stuck and helped us create an interesting project in the end.”

Nora Petrova, ML Engineer at Prolific: “The hackathon really helped me to be embedded in a community where everyone was working on the same topic. There was a lot of curiosity and interest in the community. Getting feedback from others was interesting as well and I could see how other researchers perceived my project. It was also really interesting to see all the other projects and it was positive to see other's work on it.”

Chris Mathwin, MATS Scholar: "The Interpretability Hackathon exceeded my expectations, it was incredibly well organized with an intelligently curated list of very helpful resources. I had a lot of fun participating and genuinely feel I was able to learn significantly more than I would have, had I spent my time elsewhere. I highly recommend these events to anyone who is interested in this sort of work!”

What if my research seems too risky to share?

Besides emphasizing the introduction of concrete mitigation ideas for the risks presented, we are aware that projects emerging from this hackathon might pose a risk if disseminated irresponsibly.

For all of Apart's research events and dissemination, we follow our Responsible Disclosure Policy.

We have collected a few exciting resources about deception, both to get you started with research in this area but also to dive even deeper. Required reading:

• Check out the Deception Detection Hackbook to train your own Sleeper Agent and get inspired
• Examples of deception in AI systems written by Park et al. 2023 (pages 2 to 10)

Optional research articles on concepts related to deception detection:

• Research attempting to detect deception in AI systems
  • The Situational Awareness Benchmark (Laine et al., 2024) takes steps towards understanding when language models know which situation they're in, a property important to be able to execute strategic deception, such as sandbagging
  • The GDM frontier evals paper (Phuong et al., 2024) includes a section on autonomy and includes good tasks to test applied situational awareness
  • Burns et al. (2024) use a form of PCA on neural network activation to identify whether a language model is lying or not‍
  • Zou et al. (2023) use neural population modeling to develop "representation engineering", a method to understand a neural network's representation of high-level concepts, such as "honesty", "morality", and "emotions"
  • Azaria & Mitchell (2023) classify whether a statement is true or false based on activations in a language model, leading to an understanding of when a model might be lying
  • Pacchiardi et al. (2023) use a black-box method (where we don't have access to model internals) to ask unrelated follow-up questions and use the model's answers to understand whether it's lying or not
  • Kuhn et al. (2023) measure the uncertainty of a language model's responses
  • Anthropic (2024) uses "linear probes" (simple statistics on neural network activation) to catch sleeper agents (see the definition in the paper further down)
• Research introducing methods for reducing deception and trustworthy design‍
  • Quirke et al. (2024) introduce an example mitigation where we "install" verified circuits into neural networks to trust a specific sub-process in a network‍
  • Li et al. (2023) use activation steering to increase honesty and truthfulness in a Llama model after deployment (inference-time)
  • In Hubinger et al. (2024), Anthropic shows that so-called "Sleeper agents" (LLMs trained to be deceptive) are very difficult to inspect and remove harmfulness from (w/ involvement from Apart)‍
• Related research work exploring the frontier of capabilities that are potentially required for deception or is useful for your work during the weekend‍
  • Kinniment et al. (2024) describes methods to use challenges to evaluate models for the ability for autonomy and autonomous research and development - you might find inspiration for developing agent architectures to test for deception in this (and possibly find this METR research useful as well)‍
  • AISI published their fourth progress report with evaluation metrics on various models for 1) ability to program and do cyber operations, 2) show knowledge of chemistry and biology, 3) ability to work autonomously, and 4) security against malicious attacks.
  • AISI also published inspect, a framework for large language model (LLM) evaluations, that might be useful for your work‍
  • Woodside et al. (2023) published an updated list of examples where AI is used to improve AI systems, showing an example of a collaborative literature review project

Optional reading about the potential for deception in superintelligent systems:

• In this work by Hubinger et al. (2021), they explore the theoretical tendency of neural networks to have internal goals that might not be aligned with what the creator wants
• Carlsmith (2023) explores the high possibility that machine learning actually incentivizes deception / scheming
• Weij et al. (2024) introduces the concept of "sandbagging" in AI systems, the potential for models to strategically underperform during evaluation to fool an auditor or engineer
• Scheurer (2024) is an Apollo Research project presented at the AI Safety Summit in UK, showing a preliminary demonstration of an LLM strategically deceiving humans

Additionally, we have previously hosted hackathons where related work was submitted. You can find a few examples here to find inspiration for where you might be able to take your project during the weekend. You can find examples on the Sprints page.

The schedule runs from 4 PM UTC Friday to 3 AM Monday. We start with an introductory talk and end the event during the following week with an awards ceremony. Join the public ICal here.

You will also find Explorer events such as collaborative brainstorming and team match-making before the hackathon begins on Discord and in the calendar.