May 6, 2024
Assessing Algorithmic Bias in Large Language Models' Predictions of Public Opinion Across Demographics
Khai Tran,Sev Geraskin,Doroteya Stoyanova,Jord Nguyen
The rise of large language models (LLMs) has opened up new possibilities for gauging public opinion on societal issues through survey simulations. However, the potential for algorithmic bias in these models raises concerns about their ability to accurately represent diverse viewpoints, especially those of minority and marginalized groups.
This project examines the threat posed by LLMs exhibiting demographic biases when predicting individuals' beliefs, emotions, and policy preferences on important issues. We focus specifically on how well state-of-the-art LLMs like GPT-3.5 and GPT-4 capture the nuances in public opinion across demographics in two distinct regions of Canada - British Columbia and Quebec.
Data for GPT 3.5 looks strange
The problem analysis seems super on point. The automation of key institutional features requires significantly super-human implementation to avoid creating distrust and thereby fragmentation. I would like to see more attempts at solving the representativeness problem.
A really cool question to study empirically with lots of potential for relevant insight.
You’ve taken a very interesting approach of polling several LLMs as if they were humans and comparing that with real-world polling data. This is a very bold claim, and I think that the experimental design is lacking in a some aspects to back it up. For example, the was very little prompt engineering, for GPT the demographic data was presented with no preamble; there was little justification to look only at the “strong” response classification; every eval was done only once; different demographic slices were weighed the same (e.g. 86-95 non-binary MSc from Quebec has the same weight as everyone else).
Cite this work
@misc {
title={
Assessing Algorithmic Bias in Large Language Models' Predictions of Public Opinion Across Demographics
},
author={
Khai Tran,Sev Geraskin,Doroteya Stoyanova,Jord Nguyen
},
date={
5/6/24
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
