Unveiling Latent Beliefs Using Sparse Autoencoders
Carlos Cortez, Eivind Otto Hjelle, Sanchit Kalhan · Team Serbot
Submitted to Reprogramming AI Models Hackathon. Sprint projects are early-stage work by participants, not Apart Research publications.
Language models (LMs) often generate outputs that are linguistically plausible yet factually incorrect, raising questions about their internal representations of truth and belief. This paper explores the use of sparse autoencoders (SAEs) to identify and manipulate features that encode the model’s confidence or belief in the truth of its answers. Using GoodFire AI’s powerful API tools of semantic and contrastive search methods, we uncover latent fea- tures associated with correctness and accuracy in model responses. Experiments reveal that certain features can distinguish between true and false statements, while others serve as controls to validate our approach. By steering these belief-associated features, we demonstrate the ability to influence model behavior in a targeted manner, improving or degrading factual accuracy. These findings have implications for interpretability, model alignment, and enhancing the reliability of AI systems.
Reviews
These findings are cool and somewhat surprising - I didn't realise we can nudge models towards being wrong so easily! I'm having trouble parsing figure 1, however - surely with nudge strength set to zero all features should provide the same outputs, but we see an almost 20% range in percentage correctness between features.
Should I conclude that some features can in fact steer the model substantially towards correct answers? If so then that's interesting and I'd highlight it more.
Very promising results! on the point 4 it would have been better to emphasize the contribution of your work instead of talking about next steps only
I really like the direction of applying SAEs to identify hallucinations or uncertainty in model responses. The dataset was well-chosen and the methodology was sound.
The plot for figure 1 could be improved by plotting the most impactful features (perhaps with the entire figure, with a figure legend in the appendix). If I'm reading the figure correctly, it appears that nudging the feature positively causes more incorrect answers, even for features that seem related to correctness. It'd be interesting to see a qualitative analysis into what might be happening there!
In the introduction, they talk about anecdotal evidence for a quite specific behavior, which I have never heard of: "Anecdotal evidence suggests that LLMs are more likely to correct information in common domains of knowledge that are well represented in the training data, whereas they are more likely to apologize for being wrong when it comes to more niche domains of knowledge or specific problems." I think models clearly understand the concept of a degree of belief, but it might be more accurate to simply ask them about it directly. I conducted a small experiment asking ChatGPT which seemed more likely: the moon landings being fake or the Earth being flat. It decided that "the moon landings being fake" is more likely, which I believe is true. They also conducted an experiment steering a model while it solved arithmetic questions to make it produce false results. They experimented with a few features and included a control feature; however, it appears they never returned to this and didn't show results for the control feature. The main plot shows a straight line for one of the features, which could be the private feature, but there is no labeling or legend for the different features. In general, the title seems aspirational—it makes sense that they could only explore the idea superficially as part of this hackathon. For the most part, it appears they just identified features related to truthfulness, manipulated them in both directions, and evaluated the results on a dataset. It would be interesting to identify more specific features for belief, such as changing how certain the model is in certain areas, rather than simply whether its outputs are correct or false. They also haven't yet uncovered any hidden beliefs in the model, as the title suggests. This could be fascinating—for example, does the model give medical advice while being uncertain? Could belief steering be used to make the model doubt itself more or be more cautious in high-stakes areas?
Read full reviewShow less
Cite this project
@misc{cortez2024unveiling,
title = {{Unveiling Latent Beliefs Using Sparse Autoencoders}},
author = {Carlos Cortez and Eivind Otto Hjelle and Sanchit Kalhan},
year = {2024},
month = nov,
note = {Submitted to Reprogramming AI Models Hackathon, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/unveiling-latent-beliefs-using-sparse-autoencoders}},
url = {https://apartresearch.com/sprints/projects/unveiling-latent-beliefs-using-sparse-autoencoders}
}More from Reprogramming AI Models Hackathon
- 1st place by peer reviewView project: AutoSteer: Weight-Preserving Reinforcement Learning for Interpretable Model Control
AutoSteer: Weight-Preserving Reinforcement Learning for Interpretable Model Control
AI Safety Initiative Groningen
Traditional fine-tuning methods for language models, while effective, often disrupt internal model features that could provide valuable insights into model behavior. We present a novel approach combining Reinforcement …
- View project: Classification on Latent Feature Activation for Detecting Adversarial Prompt Vulnerabilities
Classification on Latent Feature Activation for Detecting Adversarial Prompt Vulnerabilities
Feature Disruption Lab
We present a method leveraging Sparse Autoencoder (SAE)-derived feature activations to identify and mitigate adversarial prompt hijacking in large language models (LLMs). By training a logistic regression classifier on …
- View project: Utilitarian Decision-Making in Models - Evaluation and Steering
Utilitarian Decision-Making in Models - Evaluation and Steering
Byte To Brain
We design an eval based on the Oxford Utilitarianism Scale (OUS) that measures the model’s deontological vs utilitarian preference in a nine question, two factor model. Using this scale, we measure how feature steering …