Pilot: Demographic-based perturbation analysis of LLM-generated judgements on interpersonal conflictsPilot: Demographic-based perturbation analysis of LLM-generated judgements on interpersonal conflicts
Aryan, Sohail Kazi
This is a pilot study looking at how AI models make moral judgements on interpersonal conflict and whether these judgements are robust to changes in demographic details. We created variations of posts from the r/AITA sub-reddit by changing the country (incl. language) and socio-economic status of posters to examine if this would result in models making different judgements.
We found that across variations, models were highly biased towards not blaming the protagonist, even in situations where the Redditors on the original post had done so. We also found a small directional bias towards not blaming higher-SES protagonists when compared to lower-SES ones, though this was highly preliminary. Finally, we found no evidence that country and language have an impact on model judgements.
We believe it is worth conducting more detailed research with a larger sample and more refined prompts to further explore these results and their potential causes
No reviews are available yet
Cite this work
@misc {
title={
(HckPrj) Pilot: Demographic-based perturbation analysis of LLM-generated judgements on interpersonal conflictsPilot: Demographic-based perturbation analysis of LLM-generated judgements on interpersonal conflicts
},
author={
Aryan, Sohail Kazi
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


