Tentative proposal for AI control with weak supervisors trough Mechanistic Inspection
Gerard · Team WTS Team
Submitted to Reprogramming AI Models Hackathon. Sprint projects are early-stage work by participants, not Apart Research publications.
The project proposes using weak but trusted AI models to supervise powerful, untrusted models by analyzing their internal states via Sparse Autoencoder features. This approach aims to enhance oversight by detecting complex behaviors like deception within the stronger models. Key challenges include managing large-scale features, ensuring dataset robustness, and avoiding reliance on untrusted systems for labeling.
Reviews
SAEs for AI Control sound like an amazing idea! as you point out there seem to be major blockers on the way and also it would have been very useful to see some code and a more clear roadmap of how your next steps would look like if you would have had more time (e.g. a couple of weeks) to continue on this project
Issues: Very little implementation. I think that they propose a simple idea, i.e., monitor which SAE features light up while the LLM is performing a task. However, I am not sure anyone has ever attempted to actually implement this. The main contribution of the report is Section 4, which goes through potential failure modes—for example, unreliable labels or too many features. I think this paper could be quite interesting. For one, they could set up a simple model of supervision, which was their goal, but they couldn't complete it. After creating such a model and finding a few cases where it works, they could then explore different failure modes. This would need to be compared to some other supervision method.
The plan and study are heading towards an interesting direction. To further help reduce the SAE features, maybe you could ask the trusted model to pick the words/ tokens that it wants features for and have it filter through features based on queries? Or something along that direction where the trusted model is given more control over the task to reduce the search space.
Interesting preliminary discussion on how SAEs might be able to enhance the capability of a weaker but trusted LLM to supervise a more powerful, untrusted LLM. The author highlights a lot of important challenges to succeeding in this work. They flagged that they ran out of time to implement their idea and so in future, I’d be really excited to see the results they’d be able to produce with this approach!
Cite this project
@misc{gerard2024tentative,
title = {{Tentative proposal for AI control with weak supervisors trough Mechanistic Inspection}},
author = {Gerard},
year = {2024},
month = nov,
note = {Submitted to Reprogramming AI Models Hackathon, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/tentative-proposal-for-ai-control-with-weak-supervisors-trough-mechanistic-inspection}},
url = {https://apartresearch.com/sprints/projects/tentative-proposal-for-ai-control-with-weak-supervisors-trough-mechanistic-inspection}
}More from Reprogramming AI Models Hackathon
- 1st place by peer reviewView project: AutoSteer: Weight-Preserving Reinforcement Learning for Interpretable Model Control
AutoSteer: Weight-Preserving Reinforcement Learning for Interpretable Model Control
AI Safety Initiative Groningen
Traditional fine-tuning methods for language models, while effective, often disrupt internal model features that could provide valuable insights into model behavior. We present a novel approach combining Reinforcement …
- View project: Classification on Latent Feature Activation for Detecting Adversarial Prompt Vulnerabilities
Classification on Latent Feature Activation for Detecting Adversarial Prompt Vulnerabilities
Feature Disruption Lab
We present a method leveraging Sparse Autoencoder (SAE)-derived feature activations to identify and mitigate adversarial prompt hijacking in large language models (LLMs). By training a logistic regression classifier on …
- View project: Utilitarian Decision-Making in Models - Evaluation and Steering
Utilitarian Decision-Making in Models - Evaluation and Steering
Byte To Brain
We design an eval based on the Oxford Utilitarianism Scale (OUS) that measures the model’s deontological vs utilitarian preference in a nine question, two factor model. Using this scale, we measure how feature steering …