AI Safety x Physics Grand Challenge

This Challenge has been concluded

Over the July 25th-27th weekend intensive, participants tackled critical challenges by applying physics methodologies—from thermodynamics and statistical mechanics to computational mechanics—to understand and control AI systems. The event featured both Project and Exploration tracks, encouraging participants to bridge theoretical physics insights with practical AI safety solutions through collaborative innovation.
Following are the prize winners:

1. First Prize: Constrained Belief Updates in Transformer Representations:
   Brianna Grado-White
   This work extended computational mechanics theory to analyze how transformers implement optimal prediction under constraints. It showed belief state geometry across layers and that attention patterns decay exponentially with distance. Despite complex challenges, it bridged physics-inspired theory and practical interpretability, helping understand AI reasoning.
2. Second Prize: Momentum-Point-Perplexity Mechanics in Large Language Models
   by Lorenzo Tomaz, Thomas Jones
   This work found that energy remains approximately conserved across 20 different models. Their theoretical insights led to Jacobian steering—a control method that improves output quality while respecting the underlying physics, offering both fundamental understanding and practical applications for AI safety.
3. Third Prize: Local Learning Coefficients in Reinforcement Learning:
   by Jeremias Ferrao, Ilija Lichkovski
   This project applies Singular Learning Theory to reinforcement learning, demonstrating how Local Learning Coefficients (LLCs) detect training phase transitions. By tracking reward components directly rather than derived metrics, LLC spikes correlate with capability milestones. This scalable approach enhances the understanding of how AI systems learn complex behaviors, with significant safety implications.
4. AI agentic system epidemiology
   by Valentina Schastlivaia, Aray Karjauv
   Using Physics-Informed Neural Networks to solve the governing equations, they created a framework for monitoring AI system health and predicting intervention effectiveness—providing a novel lens for understanding systemic AI risks.

Please find the link to the presentation here: https://docs.google.com/presentation/d/1IWKW9gMtc2qjfQAdWlRcT0EnJzeuJX3h6XOZC2oPOT4/edit?usp=sharing

Uniting Physics to Solve AI Safety

Join us for this AI Safety x Physics Grand Challenge to use your valuable skillset on chosen problem spaces for physicists!

You'll work alongside peers to create an original research project during this two day sprint with the guidance of expert mentors in the unique intersection of physics and AI safety. Be ready for excitement, progress, and, last but not least, prizes!

Physics and AI Safety?

The complexity of neural networks, the topology of their latent space, and the dynamics of the systems that deploy AGI today is an ideal topic of study for any physicist. We are at the absolute frontier of understanding learning dynamics through fundamental physics and during this event, you will become an integral part of this field's future!

Research Tracks

Employ your ML / AI experience from interpretability, benchmarking, or training to develop an original project employing the novel tools that physicists have introduced to the field in the past years.

Examples include the projects we've defined, such as operationalizing the natural abstraction hypothesis for empirical tests of compression or predictability of learned abstractions.

Propose novel research agendas to tackle the most pressing AI challenges or explore a potential bridge between an AI safety problem and a physics concept.

An example could be a proposal, outlining a novel physics-based solution to a key AI safety problem, with supporting material drawn from prior work.

Join

And experience…

• Expert mentorship from leaders in the field of physics and AI safety
• Structured problem areas with starter materials and concrete research directions
• Global community of highly talented participants who are ambitiously solving top challenges
• Follow-up support for exceptional projects through the Apart Research programs
• Optional pre-event orientation and brainstorming for those new to AI safety

Shape the Future!

Join us in applying your academic training to one of the most important technical challenges of our time. Connect with a growing community of researchers safeguarding AI systems for the benefit of humanity.

FAQ

What Background Knowledge Is Required?

This challenge welcomes participants with physics backgrounds who are interested in applying rigorous methodologies to AI safety. While familiarity with machine learning is helpful, we encourage participation from:

• Physics PhD students, postdocs, and faculty across all subfields
• Theoretical physicists interested in complex systems and statistical mechanics
• Experimental physicists with experience in data analysis and modeling
• Computational physicists familiar with simulation and numerical methods
• Physicists working at the intersection of physics and computer science
• Anyone passionate about applying physics thinking to beneficial AI development

How Are Teams Formed?

Teams will self-organize during the first day through structured activities and interest-matching sessions. You can also form teams in advance with other registered participants via our Discord channel: https://ais.pub/disc

Teams should ideally combine:

• Different physics backgrounds (theory, experiment, computation)
• Varied experience levels with AI/ML
• Complementary skills in mathematics, programming, and research

What Are The Expected Outputs?

Teams should produce research outputs and reports that demonstrate progress toward applying physics methodologies to AI safety challenges. The most promising projects may evolve into:

• Academic papers or technical reports
• Continued research collaboration with physics-AI safety researchers
• Integration into ongoing research programs at leading institutions
• Follow-up funding opportunities for extended research
• Theoretical frameworks for future empirical validation

Will Compute Resources Be Provided?

Yes! Lambda is providing $400 in computing credits to each participating team, giving you access to powerful cloud instances (including A100s) to bring your ideas to life.

What Challenge Track Should I Choose?

You must align your project with one of our two tracks:

Project Track: This is a typical hackathon submission based on the starter materials or an original idea. We expect most of these to provide incremental theoretical progress and/or empirical evidence for (or against) an existing idea.

Relevant for higher context participants (i.e. those with experience running experiments on NNs or with an existing physics-AI agenda they want to accelerate progress on).

Exploration Track: Part of the idea for this hackathon is to open up and test the limits of a ‘physics for AI safety’ opportunity space. We have thought of some, but likely many more that would be exciting. Instead of traditional ‘hacking’, this will be a written submission, for example:

1. A proposal, outlining a novel physics-based solution to a key AI safety problem, with supporting material drawn from prior work.
2. A distillation or literature review bridging a CS/ML/AI idea with one from physics, with clear reference to an AI safety problem area.

We think this track has the potential to brainstorm some truly innovative ideas, while also allowing lower-context participants to get involved. For example, a ‘papers’ participant or group could partner with a ‘project’ group, resulting in a small research gain and a better understanding of the bigger picture.

What Problem Area Should I Focus On?

Your project should address one or more of our five problem areas:

Problem Area 1: Theory/Practice Gap for AI Interpretability - Apply physics methodologies (renormalization, computational mechanics, statistical mechanics) to build theoretical foundations for mechanistic interpretability

Problem Area 2: Learning, Inductive Biases, and Generalization - We want to understand how architecture, initialization, data distribution, and training procedures give rise to inductive biases—and in turn, how these biases control what is learned, when generalization occurs, and why models may fail.

Problem Area 3: Mathematical Models of Data Structure - AI alignment faces the challenge of understanding an AI's internal world model to prevent harmful actions. Physics methods, like coarse-graining and renormalization, could provide mathematical models to describe AI's learned representations and improve our understanding of AI's abstraction, generalization, and world modeling.

Problem Area 4: Quantitative Bounds on AI Behavior - This challenge focuses on using physics-inspired uncertainty quantification to establish quantitative bounds on AI behavior, ensuring alignment even when systems develop unanticipated strategies. Your work is crucial for guaranteeing ethical and reliable AI.

Problem Area 5: Scaling Laws, Phase Transitions, and Emergence - Uncover the physical foundations of AI performance, universal limits, and how smooth scaling reconciles with emergent capabilities.

Core AI Safety x Physics Literature

See a full overview of our Five Problem Areas in this post: https://apartresearch.com/news/problem-areas-in-physics-and-ai-safety

Natural Abstractions Research

• The Natural Abstraction Hypothesis by John Wentworth: Explores whether there exist abstractions that diverse cognitive systems naturally converge on using, with major implications for AI interpretability and alignment.
• AXRP Episode on Natural Abstractions: Comprehensive interview covering how natural abstractions relate to agency, value learning, and AI safety. Essential for understanding the theoretical foundations.

Singular Learning Theory

• Distilling Singular Learning Theory by Liam Carroll: Accessible introduction to how Sumio Watanabe's mathematical framework applies to understanding phase transitions and generalization in neural networks.
• Developmental Interpretability: Jesse Hoogland's work on using SLT to predict when dangerous capabilities emerge during training, crucial for AI safety applications.

Physics-Informed Machine Learning

• Physics-Informed Neural Networks (PINNs): Revolutionary approach embedding physical laws into neural network training, offering interpretable architectures and principled integration of domain knowledge.
• PINN Survey Paper: Comprehensive review of techniques, applications, and challenges in physics-informed deep learning.

Scaling Laws and Emergence

• Sharma & Kaplan (2022): Fundamental work connecting scaling laws to data manifold dimension
• Bahri et al. (2024): Latest developments in understanding scaling behavior from geometric perspectives

Interpretability Bottlenecks/Gaps in our Understanding:

• (Decomposing Dark Matter of SAEs) https://arxiv.org/abs/2410.14670
• (Sparse Autoencoders Do Not Find Canonical Units) https://arxiv.org/abs/2502.04878
• (Stochastic Parameter Decomposition) https://arxiv.org/html/2506.20790v1 ]
• (Matryoshka SAEs as new interp tool) Bussman et al. , (MP-SAEs)Costa et al.

Misc. interp refs:

• Mech interp is not pre-paradigmatic — LessWrong
• TAIS RFP: Research Areas | Open Philanthropy
• [2501.16496] Open Problems in Mechanistic Interpretability

Complex Systems perspective

• [2506.14797] Bound by semanticity: universal laws governing the generalization-identification tradeoff
• [2504.13727] High-dimensional dynamics in low-dimensional networks

Learning:

• Alternating Gradient Flows
• Critical Feature Learning

Field theory and Renormalization:

• Separation of scales and a thermodynamic description of feature learning in some CNNs
• Grokking as a First Order Phase Transition
• Koch-Janusz and Ringel
• https://arxiv.org/pdf/2502.03210
• Renormalizing Diffusion Models
• Bayesian Renormalization
• Ncoder
• https://iopscience.iop.org/article/10.1088/2632-2153/adc8fc/pdf
• Bayesian RG flow in neural network field theories
• [2008.08601] Neural Networks and Quantum Field Theory
• [2106.10165] The Principles of Deep Learning Theory
• Laplacian Renormalization for Heterogeneous Networks paper, Nature Physics, 2023
• Higher-Order Laplacian RG paper, arXiv:2401.11298

Hierarchical Data structure (sub of renormalization):

• Representation Learning on a Random Lattice
• Probing the Latent Hierarchical Structure of Data via Diffusion Models
• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
• [2402.09090] Software in the natural world: A computational approach to hierarchical emergence
• [2502.01954] Constrained belief updates explain geometric structures in transformer representations

Submission Guidelines

Your submission must include:

1. A completed project report using the provided template (max 6 pages, unlimited appendix)
   • Make your own copy of the Project Track template on Google Docs
   • Make your own copy of the Exploration Track template on Google Docs
2. Latex Templates
   1. Project Template: https://www.overleaf.com/read/phhpngpwchmy#fa7ac8
   2. Exploration Template: https://www.overleaf.com/read/ybgpwcpwxwjv#f13843
3. Link to a public GitHub repository with your code (optional but recommended for the Experimental Track)
4. A brief (2-3 minute) video demonstration/slide presentation (optional but recommended)

How Will Projects Be Evaluated?

Projects will be assessed on three main dimensions:

1. Technical Rigor & Feasibility - How rigorous is your physics methodology and how feasible is your approach? Is your theoretical framework sound and your empirical work well-designed? Can your proposed methods be implemented and validated?
2. AI Safety Relevance - How clearly does your work address important AI safety challenges? What is the potential impact on ensuring beneficial AI development? Does your approach offer meaningful insights for AI alignment research?
3. Innovation & Originality - How novel and creative is your approach to bridging physics and AI safety? Do you introduce new theoretical connections or methodological innovations? What makes your work distinct from existing research?

AI Safety x Physics Grand Challenge - FAQs

Project Scope and Requirements

Q: Do my ideas need to fit exactly into the problem areas presented?
A: Not at all. The problem areas are meant to provide inspiration and guidelines, but your project doesn't need to fit strictly into any particular category. The goal is to solve important problems using physics insights, regardless of how you categorize the approach.

Q: How do I ensure my project can be completed in a weekend?
A: Focus on the smallest scale version of your project that can still yield meaningful results. Consider what's the most tractable aspect you can tackle within the time constraints, especially for challenging areas like rare event analysis.

Q: Is it okay if my project is an extension of prior work rather than completely novel?
A: Absolutely. Extending prior work to new settings or validating existing approaches in more complex scenarios provides valuable information. Even if the results seem similar to previous work, validation often reveals more complex phenomena than initially expected.

Mentorship and Guidance

Q: When is the best time to reach out to mentors?
A: Email is most likely to get timely responses before the event. During the hackathon, Discord becomes more active, and there are scheduled office hours for guaranteed mentor access.

Q: What if I don't have much AI experience but have a physics background?
A: The event welcomes participants with varying levels of AI experience. The exploration/papers track is particularly suitable for those who want to contribute physics insights without necessarily having deep AI implementation experience.

Computing Resources and Technical Support

Q: Can I use the credits after the hackathon?
A: Yes! The credits remain valid after the hackathon, so you can continue working on your project and take it forward beyond the weekend.

Q: What if I need API credits for language models instead of GPU compute?
A: OpenAI API access is available. Contact the organizers during the hackathon and they'll provide API keys for your project needs.

Q: Is the compute available for everyone?
A: Where possible! However, current sanctions on A100 access means Apart will not and cannot provide compute to Russia and China, among others.

Team Formation and Collaboration

Q: How do I find teammates?
A: Use the Discord projects/teams channel to post your ideas and find collaborators. Add relevant tags to help others discover your project and identify potential team members with complementary skills.

Q: Should I form teams before the hackathon starts?
A: Yes, it's highly recommended. Getting your team together before the weekend allows you to focus on the actual project work rather than spending valuable time on team formation during the event.

Event Structure and Tracks

Q: What's the difference between the Project Track and Exploration Track?
A: Project Track: Traditional hackathon projects with concrete implementations and results, ideal for those with AI/ML experience
Exploration Track: Written submissions like research proposals or literature reviews bridging physics and AI safety concepts, perfect for those who want to contribute novel ideas or have less hands-on AI experience

Q: What kind of deliverables are expected?
A: For projects: concrete results with follow-up proposals and clear AI safety relevance. For the exploration track, consider research proposals or literature reviews that could point to future research directions, opportunities, or challenges.

Physics Background and Relevance

Q: What if my physics background doesn't seem directly applicable?
A: Many physics skills transfer well to AI safety. The event particularly values the ability to shift between different methodologies, balance theoretical rigor with empirical work, identify universal patterns in complex systems, perform multi-scale analysis, build toy models, and handle uncertainty.

Q: Are there any physics approaches that might not be suitable?
A: Be cautious about direct analogies. Terms meaningful in physics, like universality, causality, and locality, may not cleanly map onto AI systems. Avoid conflating analogies with universal laws.

Research Areas and Focus

Q: Has work on physics + AI safety been done before?
A: Some foundational work exists, but there's much more to explore. Areas like renormalization applied to AI systems are active research areas with significant untapped potential.

Q: What if I'm interested in interpretability but it's not directly safety-focused?
A: Basic understanding of AI systems contributes to safety, but be mindful of the distinction between interpretability for safety versus interpretability for capabilities enhancement.

Practical Considerations

Q: What's the target participant level?
A: The event targets participants who have at least started a PhD, generally aiming for mid-senior level talent with physics backgrounds and some AI/ML experience.

Q: How do I submit my final project?
A: Start by posting your ideas in the Discord projects/teams channel. For submission you need to submit it on "Submit Your Project" Button on top of this page

Q: What happens to good projects after the hackathon?
A: Top projects receive follow-up research support, including project management and compute resources through Apart Research and collaborating researchers, with potential pathways to publications or continued research.

AI Safety × Physics Grand Challenge - Event Schedule