Dear Apart Community,

Welcome to our newsletter - Apart News!

Happy New Year, we hope you all had a restful break. At Apart Research there is so much brilliant research, great events, and countless community updates to share.

In this week's Apart News we share with you Apart Research's incredible ​2024​.

Apart in 2024

On a warm evening during our work offsite, our co-Director Esben stood up after dinner and told the team about just how far we’d come.

"It is pretty clear that Apart Research's 2024 was the biggest year of our existence. We tripled the size of our core team. Raised $700,000. Wrote a dozen AI safety papers and had several published at major conferences. Facilitated hundreds of AI safety experiments for budding researchers. ‍

We also doubled the amount of Apart Fellows researching in our Lab. Created a whole new Apart Lab Studio program. And added Hackathon Jam Sites from Bangalore to Singapore."

"And not just the numbers look great - we are also beginning to see our impact pay off on real AI safety with labs such as OpenAI, Anthropic, and the UK AI Safety Institute using our work and Apart researchers launching and publishing impactful work."

Apart Fellow Growth

The Apart Lab fellowship has become a crucial pathway for transitioning talented professionals into AI safety research. We provide fully-remote, guided 3-6 month program to accelerate research careers remote-first and part-time, with support for compute and conference participation.

Our structured four-stage process guides fellows from project planning to conference-ready drafts, with supervision from our research lead, program manager, and research assistants.

The program particularly excels at supporting mid-career professionals from technical domains such as cybersecurity and software engineering, as well as academics from related fields.

Recent survey data from 25 program participants reveals concrete impact: 10 respondents now work primarily in AI safety. Among those yet to secure positions, all except one actively pursue AI safety careers through job searching, continued study, or upskilling.

But don't just take our word for it! Below are testimonials from some of our Fellows.

Cristian Curaba: “The fellowship gave me a structured research plan to follow, which helped me stay focused and achieve milestones quicker.”

Denis D’Ambrosi: “Apart definitely helped me improve as a researcher in AI safety, especially by exposing me to relevant discussions and giving me a chance to collaborate.”

Zainab Ali Majid: “I’ve worked in cyber security for the past five years, and in early 2024, I developed a deep interest in exploring the intersection between this and AI safety. The Apart Fellowship provided me with a unique opportunity to conduct impactful research at this nexus. "

"Before the fellowship, I found it difficult to envision how I'd gain the necessary skills to transition into this emerging field. However, through collaboration with other researchers and learning from experts in the field, the experience has significantly accelerated my progress, culminating in an accepted conference paper.”

Alex Foote: “I’ve now written two papers in interpretability thanks to Apart’s fellowship program. The Apart team guided me through the initial steps of developing these research ideas, which I’m continuing in my graduate studies.”

Apart Lab Studio

Our co-Director Jason was very happy to announce our new AI safety research support program: Apart Lab Studio. Our new program is designed to bridge the gap between weekend hackathon projects and a fully-fledged AI Safety research career. Since its inception, we already have over 15 participants!

Our Apart Lab Studio helps participants to fully realize the impact of their hackathon work for their own career as well as for the wider world. Our new 8-week program offers promising researchers worldwide the opportunity to develop their ideas and test their research fit - regardless of their current location or employment status. It is especially for those who want to explore their ideas and potential fit for research. Have a look ​here​!

An AI Safety Lab

Apart Research's publication record demonstrates our growing influence in AI safety research. Since 2023, our work has produced 13 peer-reviewed papers at top venues including NeurIPS, ICLR, ACL, and EMNLP, with six main conference papers and nine workshop acceptances.

Notable publications in 2024 include "Interpreting Learned Feedback Patterns in Large Language Models" (NeurIPS 2024), "Large Language Models Relearn Removed Concepts" (ACL 2024, Best Paper Award at TAIS 2024), and "Understanding Addition in Transformers" (ICLR 2024).

At NeurIPS 2024 alone in December of this year we presented:

• 'CryptoFormalEval: Integrating LLMs and Formal Verification for Automated Cryptographic Protocol Vulnerability Detection' - Here is the full write-up on our ​blog​ of the CryptoFormalEval paper. Arxiv paper ​here​. Paper authored by Cristian Curaba, Denis D’Ambrosi, Alessandro Minisini, & Natalia Pérez-Campanero Antolín.
• 'Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models' - What if an AI hides these dangerous capabilities during testing? That's 'sandbagging' - when an AI deliberately underperforms to avoid detection. Our paper develops a technique to detect this behavior! When it comes to sandbagging, it's not just about the AI doing it on their own accord, it's also about an AI being told or trained to underperform by the model developer. Think of the VW Diesel scandal. Paper ​here​. Authors include Philipp Alexander Kreer, Cameron Tice, Fedor Ryzhenkov, Felix Hofstätter, Jacob Haimes, Prithviraj Singh Shahani and Teun van der Weij.
• 'Deceptive agents can collude to hide dangerous features in Sparse Auto Encoders' - This ​paper​ explores how AI agents in Sparse Autoencoders can collude to hide harmful features - using deceptive labeling & covert communication.Authors ​Simon Lermen​, ​Mateusz Dziemian​, and our Research Manager ​Natalia Pérez-Campanero Antolín​ wrote this one.
• 'Rethinking CyberSecEval: An LLM-Aided Approach to Evaluation Critique' - ****This paper critiques existing approaches to evaluating AI models' cybersecurity capabilities. Finding limitations that undermine its effectiveness real security risks in AI-generated code. Apart Blog writeup ​here​. Arxiv paper here. Authored by ​Suhas Hariharan​, ​Zainab Ali Majid​, ​Jaime Raldua​ and ​Jacob Haimes​.

Hackers to the Core

Our hundreds of remote-first Hackathons but with physical HQs meant budding researchers got the chance to trial their AI safety pilot experiments. The most promising are invited to our more advanced programs.

Apart's hackathon program has expanded significantly, organizing over 20 global AI safety research hackathons engaging more than 1,800 participants. These events have established local hackathon sites at over 50 locations worldwide, emphasizing our commitment to geographical diversity with active participation from traditionally underserved regions including Africa, South-East Asia, South America, and Eastern Europe.

This year we added the following locations to our portfolio of global jam sites: Bangalore, Singapore, CEELAR, and more. We also worked with METR, Goodfire AI, Johns Hopkins, and more.

Communicating to the World

We started our brand new Apart News newsletter. At Apart Research there is so much brilliant research, great events, and countless community updates to share. So we created ​Apart News​ to do just that.

We also started our new ​Researcher Spotlight​ series, which highlights the global community at the heart of Apart Research. We heard from Fellows from around the world about their journey into AI safety technical research.

We want to make sure our work is ​accessible​ to those new to the field. We take the effective dissemination of our research very seriously. We think it is in important skill of a researcher and as such our programs are filled with chances for fellows and participants to improve their public communications skills. We write the research blogs on the methodology and findings of our technical papers. Our fellows also created websites to showcase their findings, like the 3cb team's ​website​ highlighted below.

Looking Towards 2025

With a proven track record and a validated approach to AI safety research, Apart Research is positioned to address the most pressing challenges in the field. Our work bridges the gap between theoretical safety concerns and actionable technical solutions, mobilizing a global network of researchers to tackle AI risks collaboratively.

As we look toward 2025, continued support will enable us to scale our impact and ensure that AI systems are developed and deployed safely and beneficially. Soon Apart's 2025 essay will lay out the next 12 months for Apart Research and how we see the field of AI safety developing.

AI Safety & Assurance Startup Hackathon

AI Safety requires ambition. We are facing critical technological problems in AGI deployment during the next three years; alignment, multi-agent risk, compute security, exfiltration, among many others. Each of these questions deserves a competent team to scale science-informed solutions for.

​​Join us​​ on 17th-20th January where we will join you to kick off an ambitious journey into AI safety and security with other aligned and talented individuals from both science and business. Sign up ​here​.

Opportunities

• Never miss a Hackathon by keeping up to date ​here​!
• OpenAI opens up '​Early access safety testing​' for their new as-of-yet unreleased o3 model.

Have a great week and let’s keep working towards safe and beneficial AI.