Utilitarian Decision-Making in Models - Evaluation and Steering

Nov 25, 2024

Utilitarian Decision-Making in Models - Evaluation and Steering

Adam Newgas, Sinem Erisken , Pandelis Mouratoglou

Read Project

Details

Summary

We design an eval based on the Oxford Utilitarianism Scale (OUS) that measures the model’s deontological vs utilitarian preference in a nine question, two factor model. Using this scale, we measure how feature steering can alter this preference, and the ability of the Llama 70B model to impersonate other people’s opinions. Results are validated against a dataset of 10,000 human responses. We find that (1) Llama does not accurately capture human moral values, (2) OUS offers better interpretations than current feature labels, and (3) Llama fails to predict the demographics of human values.

See Code

Download PDF

View Presentation

Cite this work:

@misc {

title={

Utilitarian Decision-Making in Models - Evaluation and Steering

author={

Adam Newgas, Sinem Erisken , Pandelis Mouratoglou

date={

11/25/24

organization={Apart Research},

note={Research submission to the research sprint hosted by Apart.},

howpublished={https://apartresearch.com}

}

Review

Reviewer Name

Constructive critique

Criteria 1:

Advancing interpretability (Possible questions to answer in this criteria: Does the project contribute to the field of mechanistic interpretability? Does it provide new insights into understanding or steering AI model behavior? How well does it move us towards reprogramming AI models? How original and innovative is the approach?)

2.5

Criteria 2:

Relevance for AI safety (Possible questions to answer in this criteria: How important is the contribution to advancing the field of AI safety? Do we expect the results to generalize beyond the specific case(s) presented in the submission? Does the approach introduce new safety mechanisms or enhance existing ones in innovative ways?))

2.5

Criteria 3:

Methodology & presentation (Possible questions to answer in this criteria: How well is the project executed from a technical standpoint?, Is the code well-structured, documented, and reproducible?, How effectively does it utilize Goodfire's SDK/API and other provided resources?, How clearly and effectively is the research presented in the paper?, Quality of visualizations and demos (if applicable), Clarity of methodology explanation and results interpretation))

2.5

Reviewer's Comments

Esben Kran

This is a great project! Very well done. I think it's good it has a lower IH than humans and that this IH is less variable as well. It's very interesting to see these underlying patterns and the comparison to human baselines is very interesting. It seems like there are clear pathways forward to studying the moral impact of irrelevant features, such as the elephant features you mention. You may also run the model at a higher temperature and get the density function for each model to compare with human values. There is of course an implicit assumption that utilitarianism is opposed to deontology in the methodological design and it would be great to check if these patterns shift under e.g. legalistic scenarios vs. moral dilemmas (https://paperswithcode.com/dataset/ethics-1).

C1:

4.3

C2:

4.8

C3:

Liv Gorton

I found this project really interesting! It is surprising how poorly the LLMs seemed to model human moral intuitions, even when steered. This is well-written and well-presented!

C1:

C2:

C3:

Dhruba Patra

This is pretty cool and well thought out!

C1:

C2:

C3:

Tom McGrath

This is very well executed and presented research. The comparison of model values vs the human response KDE is interesting, but my favourite plot is figure 2 - it's very surprising how different features have remarkably different trajectories through the moral landscape. It's surprising that most features actually appear to avoid the modal human, and only a single feature actually steers the model in that direction. It's unfortunate that the OUS has so few questions and is so sensitive (e.g. the difference between models being entirely accounted for by question IH2).

C1:

C2:

C3:

4.3

Recent Projects

View All

Mar 24, 2025

Attention Pattern Based Information Flow Visualization Tool

Understanding information flow in transformer-based language models is crucial for mechanistic interpretability. We introduce a visualization tool that extracts and represents attention patterns across model components, revealing how tokens influence each other during processing. Our tool automatically identifies and color-codes functional attention head types based on established taxonomies from recent research on indirect object identification (Wang et al., 2022), factual recall (Chughtai et al., 2024), and factual association retrieval (Geva et al., 2023). This interactive approach enables researchers to trace information propagation through transformer architectures, providing deeper insights into how these models implement reasoning and knowledge retrieval capabilities.

Mar 24, 2025

jaime project Title

bbb

Mar 25, 2025

Safe ai

The rapid adoption of AI in critical industries like healthcare and legal services has highlighted the urgent need for robust risk mitigation mechanisms. While domain-specific AI agents offer efficiency, they often lack transparency and accountability, raising concerns about safety, reliability, and compliance. The stakes are high, as AI failures in these sectors can lead to catastrophic outcomes, including loss of life, legal repercussions, and significant financial and reputational damage. Current solutions, such as regulatory frameworks and quality assurance protocols, provide only partial protection against the multifaceted risks associated with AI deployment. This situation underscores the necessity for an innovative approach that combines comprehensive risk assessment with financial safeguards to ensure the responsible and secure implementation of AI technologies across high-stakes industries.

Mar 24, 2025