Interesting preliminary discussion on how SAEs might be able to enhance the capability of a weaker but trusted LLM to supervise a more powerful, untrusted LLM. The author highlights a lot of important challenges to succeeding in this work. They flagged that they ran out of time to implement their idea and so in future, I’d be really excited to see the results they’d be able to produce with this approach!

The plan and study are heading towards an interesting direction. To further help reduce the SAE features, maybe you could ask the trusted model to pick the words/ tokens that it wants features for and have it filter through features based on queries? Or something along that direction where the trusted model is given more control over the task to reduce the search space.

Issues: Very little implementation. I think that they propose a simple idea, i.e., monitor which SAE features light up while the LLM is performing a task. However, I am not sure anyone has ever attempted to actually implement this. The main contribution of the report is Section 4, which goes through potential failure modes—for example, unreliable labels or too many features. I think this paper could be quite interesting. For one, they could set up a simple model of supervision, which was their goal, but they couldn't complete it. After creating such a model and finding a few cases where it works, they could then explore different failure modes. This would need to be compared to some other supervision method.

SAEs for AI Control sound like an amazing idea! as you point out there seem to be major blockers on the way and also it would have been very useful to see some code and a more clear roadmap of how your next steps would look like if you would have had more time (e.g. a couple of weeks) to continue on this project