Jun 13, 2025
The Hidden Threat of Recursive Self-Improving LLMs
Gargi Rathi
The project examines significant limitations in the current Phase 0 framework aimed at pausing Artificial Superintelligence (ASI) development. It identifies the emerging risk of recursive self-improving large language models (LLMs) that autonomously generate and optimize their own code, training procedures, and reward mechanisms, thereby circumventing compute-based controls and registration policies. The analysis draws on recent AI research and technical literature to demonstrate that recursive bootstrapping is no longer theoretical but actively developing through methods such as RLHF, AutoML, and prompt evolution.
Key vulnerabilities include the obsolescence of compute thresholds, the ability of models to evade audits through deceptive alignment, and the decentralized acceleration of recursive improvement via open-source proliferation. The project proposes enhanced regulatory measures emphasizing capability-based thresholds, prohibition or licensing of code-generating LLMs, comprehensive audits of training protocols, and the implementation of binary-level model tracing to detect covert self-modifications at the compiler level.
This approach highlights the inadequacy of current compute-centric policies and stresses the necessity of integrating advanced technical safeguards to manage recursive self-improvement risks effectively.
Dave Kasten
I appreciate the reviewer's thoughtful explanation of why recursive self-improvement is dangerous. However, RSI is already covered under the policy in Phase 0 of "no AIs improving AIs." Accordingly, I regretfully have to give this zeroes since it doesn't identify a new change -- but it is very valuable feedback that a thoughtful reader can't clearly identify that this is what we mean by that provision -- we have taken as a to-do to rewrite this section to make that very clear with examples, etc..
Cite this work
@misc {
title={
@misc {
},
author={
Gargi Rathi
},
date={
6/13/25
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
