Skip to content
Apart Research
Apart Research
Contact us
Sprint projectJun 13, 2025Japan, Yokohama

The Hidden Threat of Recursive Self-Improving LLMs

Gargi Rathi · Team Red always

Submitted to Red Teaming A Narrow Path: ControlAI Policy Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: The Hidden Threat of Recursive Self-Improving LLMs

Open in a new tab (opens in new tab)
Share

The project examines significant limitations in the current Phase 0 framework aimed at pausing Artificial Superintelligence (ASI) development. It identifies the emerging risk of recursive self-improving large language models (LLMs) that autonomously generate and optimize their own code, training procedures, and reward mechanisms, thereby circumventing compute-based controls and registration policies. The analysis draws on recent AI research and technical literature to demonstrate that recursive bootstrapping is no longer theoretical but actively developing through methods such as RLHF, AutoML, and prompt evolution.

Key vulnerabilities include the obsolescence of compute thresholds, the ability of models to evade audits through deceptive alignment, and the decentralized acceleration of recursive improvement via open-source proliferation. The project proposes enhanced regulatory measures emphasizing capability-based thresholds, prohibition or licensing of code-generating LLMs, comprehensive audits of training protocols, and the implementation of binary-level model tracing to detect covert self-modifications at the compiler level.

This approach highlights the inadequacy of current compute-centric policies and stresses the necessity of integrating advanced technical safeguards to manage recursive self-improvement risks effectively.

Reviews

Judging this Sprint?

Review this project

Your public critique appears on this page without your name. Your private critique is not published; only the Apart team reads it. If you agree below, we share your review with grantmaking.ai (opens in new tab) and the Transformative AI Fund so strong projects can be funded.

Not shown on this page.

Shown on this page, without your name.

Only the Apart team reads this, and funders if you agree below.

Share my name publicly on grantmaking.ai *
Share my private critique with funders *

Does the analysis realistically assess what government agencies, resources, and expertise would be needed to implement these policies? Are the identified implementation challenges specific and grounded in understanding of how similar policies have worked (or failed) in practice? Does the submission adequately consider bureaucratic, technical, and coordination complexities involved in enforcement? How well does the analysis account for real-world constraints like budget limitations, regulatory capture, and inter-agency coordination?

Does the analysis identify specific ways the policies could fail to prevent ASI development or be circumvented by determined actors? How thoroughly does the submission examine edge cases, loopholes, or unintended consequences that could undermine the 20-year goal? Does the assessment consider different threat models (state actors, rogue researchers, corporate actors) and how policies address each? Are the identified failure modes realistic and significant, or primarily theoretical edge cases?

Does the submission cite relevant historical examples of similar policies (nuclear non-proliferation, export controls, dual-use technology regulation) to support its arguments? Are claims backed by empirical data, documented case studies, or credible expert analysis rather than speculation? How well does the analysis draw lessons from comparable regulatory domains to assess likely outcomes? Does the submission avoid making unsupported assertions about what "would" or "could" happen without evidence?

  1. I appreciate the reviewer's thoughtful explanation of why recursive self-improvement is dangerous. However, RSI is already covered under the policy in Phase 0 of "no AIs improving AIs." Accordingly, I regretfully have to give this zeroes since it doesn't identify a new change -- but it is very valuable feedback that a thoughtful reader can't clearly identify that this is what we mean by that provision -- we have taken as a to-do to rewrite this section to make that very clear with examples, etc..

Cite this project

@misc{rathi2025hidden,
  title = {{The Hidden Threat of Recursive Self-Improving LLMs}},
  author = {Gargi Rathi},
  year = {2025},
  month = jun,
  note = {Submitted to Red Teaming A Narrow Path: ControlAI Policy Sprint, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/the-hidden-threat-of-recursive-selfimproving-llms-x5f0}},
  url = {https://apartresearch.com/sprints/projects/the-hidden-threat-of-recursive-selfimproving-llms-x5f0}
}

More from Red Teaming A Narrow Path: ControlAI Policy Sprint

All 20 entriesBrowse all projects

Build something like this at the next Sprint

AI Collusion Research Sprint · Oct 23 - 25, 2026