Modelling the oversight of automated interpretability against deceptive agents on sparse autoencoders

The work suggests a novel way that SAE feature labelling may fail, due to coordinated deception between the explainer model and simulation model in the SAE feature labelling process. This is a novel and clever attack, and pointing it out is a worthwhile contribution on its own. The work is also especially well-presented for a hackathon project (though an additional few sentences on the Table 1 caption would make interpreting the results easier - I’m a bit confused about what the columns mean). Future work could experiment with the discussed mitigation strategies.

This is a clear potential deception issue in future automated interpretability work and proves a key limitation to the approach of using LLMs without enough human oversight in our neuron explanations. I especially liked the figure 6 encoded message ;) A great and in-depth project for a weekend's work and revealing a clear vulnerability. I wonder about its applicability to SoTA methods in interpretability since LLM-based explanation and prediction models might not be feasible with compute limitations on larger models. It would also be nice with more reporting of results though the conceptual work is clearly robust. Great work and definitely worth a write-up for the AlignmentForum.